Nov 04 02:31:53.533002 ip-10-0-134-125 systemd[1]: Starting Kubernetes Kubelet... Nov 04 02:31:53.915956 ip-10-0-134-125 kubenswrapper[2579]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Nov 04 02:31:53.915956 ip-10-0-134-125 kubenswrapper[2579]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Nov 04 02:31:53.915956 ip-10-0-134-125 kubenswrapper[2579]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Nov 04 02:31:53.915956 ip-10-0-134-125 kubenswrapper[2579]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Nov 04 02:31:53.919943 ip-10-0-134-125 kubenswrapper[2579]: W1104 02:31:53.919851 2579 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 04 02:31:53.919943 ip-10-0-134-125 kubenswrapper[2579]: W1104 02:31:53.919939 2579 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 04 02:31:53.919943 ip-10-0-134-125 kubenswrapper[2579]: W1104 02:31:53.919945 2579 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 04 02:31:53.920168 ip-10-0-134-125 kubenswrapper[2579]: W1104 02:31:53.919952 2579 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 04 02:31:53.920168 ip-10-0-134-125 kubenswrapper[2579]: W1104 02:31:53.919959 2579 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 04 02:31:53.920168 ip-10-0-134-125 kubenswrapper[2579]: W1104 02:31:53.919963 2579 feature_gate.go:386] unrecognized feature gate: Example2 Nov 04 02:31:53.920168 ip-10-0-134-125 kubenswrapper[2579]: W1104 02:31:53.919968 2579 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 04 02:31:53.920168 ip-10-0-134-125 kubenswrapper[2579]: W1104 02:31:53.919972 2579 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 04 02:31:53.920168 ip-10-0-134-125 kubenswrapper[2579]: W1104 02:31:53.919976 2579 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 04 02:31:53.920168 ip-10-0-134-125 kubenswrapper[2579]: W1104 02:31:53.919979 2579 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 04 02:31:53.920168 ip-10-0-134-125 kubenswrapper[2579]: W1104 02:31:53.919983 2579 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 04 02:31:53.920168 ip-10-0-134-125 kubenswrapper[2579]: W1104 02:31:53.919987 2579 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 04 02:31:53.920168 ip-10-0-134-125 kubenswrapper[2579]: W1104 02:31:53.919991 2579 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 04 02:31:53.920168 ip-10-0-134-125 kubenswrapper[2579]: W1104 02:31:53.919996 2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 04 02:31:53.920168 ip-10-0-134-125 kubenswrapper[2579]: W1104 02:31:53.920000 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 04 02:31:53.920168 ip-10-0-134-125 kubenswrapper[2579]: W1104 02:31:53.920004 2579 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 04 02:31:53.920168 ip-10-0-134-125 kubenswrapper[2579]: W1104 02:31:53.920008 2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 04 02:31:53.920168 ip-10-0-134-125 kubenswrapper[2579]: W1104 02:31:53.920013 2579 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 04 02:31:53.920168 ip-10-0-134-125 kubenswrapper[2579]: W1104 02:31:53.920017 2579 feature_gate.go:386] unrecognized feature gate: Example Nov 04 02:31:53.920168 ip-10-0-134-125 kubenswrapper[2579]: W1104 02:31:53.920022 2579 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 04 02:31:53.920168 ip-10-0-134-125 kubenswrapper[2579]: W1104 02:31:53.920026 2579 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 04 02:31:53.920168 ip-10-0-134-125 kubenswrapper[2579]: W1104 02:31:53.920030 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 04 02:31:53.920168 ip-10-0-134-125 kubenswrapper[2579]: W1104 02:31:53.920035 2579 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 04 02:31:53.921033 ip-10-0-134-125 kubenswrapper[2579]: W1104 02:31:53.920055 2579 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 04 02:31:53.921033 ip-10-0-134-125 kubenswrapper[2579]: W1104 02:31:53.920059 2579 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 04 02:31:53.921033 ip-10-0-134-125 kubenswrapper[2579]: W1104 02:31:53.920063 2579 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 04 02:31:53.921033 ip-10-0-134-125 kubenswrapper[2579]: W1104 02:31:53.920067 2579 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 04 02:31:53.921033 ip-10-0-134-125 kubenswrapper[2579]: W1104 02:31:53.920072 2579 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 04 02:31:53.921033 ip-10-0-134-125 kubenswrapper[2579]: W1104 02:31:53.920076 2579 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 04 02:31:53.921033 ip-10-0-134-125 kubenswrapper[2579]: W1104 02:31:53.920080 2579 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 04 02:31:53.921033 ip-10-0-134-125 kubenswrapper[2579]: W1104 02:31:53.920085 2579 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 04 02:31:53.921033 ip-10-0-134-125 kubenswrapper[2579]: W1104 02:31:53.920089 2579 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 04 02:31:53.921033 ip-10-0-134-125 kubenswrapper[2579]: W1104 02:31:53.920093 2579 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 04 02:31:53.921033 ip-10-0-134-125 kubenswrapper[2579]: W1104 02:31:53.920098 2579 feature_gate.go:386] unrecognized feature gate: EVPN Nov 04 02:31:53.921033 ip-10-0-134-125 kubenswrapper[2579]: W1104 02:31:53.920103 2579 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 04 02:31:53.921033 ip-10-0-134-125 kubenswrapper[2579]: W1104 02:31:53.920108 2579 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 04 02:31:53.921033 ip-10-0-134-125 kubenswrapper[2579]: W1104 02:31:53.920112 2579 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 04 02:31:53.921033 ip-10-0-134-125 kubenswrapper[2579]: W1104 02:31:53.920116 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 04 02:31:53.921033 ip-10-0-134-125 kubenswrapper[2579]: W1104 02:31:53.920121 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 04 02:31:53.921033 ip-10-0-134-125 kubenswrapper[2579]: W1104 02:31:53.920125 2579 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 04 02:31:53.921033 ip-10-0-134-125 kubenswrapper[2579]: W1104 02:31:53.920130 2579 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 04 02:31:53.921033 ip-10-0-134-125 kubenswrapper[2579]: W1104 02:31:53.920135 2579 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 04 02:31:53.921033 ip-10-0-134-125 kubenswrapper[2579]: W1104 02:31:53.920139 2579 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 04 02:31:53.921969 ip-10-0-134-125 kubenswrapper[2579]: W1104 02:31:53.920144 2579 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 04 02:31:53.921969 ip-10-0-134-125 kubenswrapper[2579]: W1104 02:31:53.920149 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 04 02:31:53.921969 ip-10-0-134-125 kubenswrapper[2579]: W1104 02:31:53.920154 2579 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 04 02:31:53.921969 ip-10-0-134-125 kubenswrapper[2579]: W1104 02:31:53.920158 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 04 02:31:53.921969 ip-10-0-134-125 kubenswrapper[2579]: W1104 02:31:53.920165 2579 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 04 02:31:53.921969 ip-10-0-134-125 kubenswrapper[2579]: W1104 02:31:53.920170 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 04 02:31:53.921969 ip-10-0-134-125 kubenswrapper[2579]: W1104 02:31:53.920176 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 04 02:31:53.921969 ip-10-0-134-125 kubenswrapper[2579]: W1104 02:31:53.920181 2579 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 04 02:31:53.921969 ip-10-0-134-125 kubenswrapper[2579]: W1104 02:31:53.920186 2579 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 04 02:31:53.921969 ip-10-0-134-125 kubenswrapper[2579]: W1104 02:31:53.920190 2579 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 04 02:31:53.921969 ip-10-0-134-125 kubenswrapper[2579]: W1104 02:31:53.920194 2579 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 04 02:31:53.921969 ip-10-0-134-125 kubenswrapper[2579]: W1104 02:31:53.920199 2579 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 04 02:31:53.921969 ip-10-0-134-125 kubenswrapper[2579]: W1104 02:31:53.920204 2579 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 04 02:31:53.921969 ip-10-0-134-125 kubenswrapper[2579]: W1104 02:31:53.920210 2579 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 04 02:31:53.921969 ip-10-0-134-125 kubenswrapper[2579]: W1104 02:31:53.920214 2579 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 04 02:31:53.921969 ip-10-0-134-125 kubenswrapper[2579]: W1104 02:31:53.920219 2579 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 04 02:31:53.921969 ip-10-0-134-125 kubenswrapper[2579]: W1104 02:31:53.920223 2579 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 04 02:31:53.921969 ip-10-0-134-125 kubenswrapper[2579]: W1104 02:31:53.920228 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 04 02:31:53.922776 ip-10-0-134-125 kubenswrapper[2579]: W1104 02:31:53.920232 2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 04 02:31:53.922776 ip-10-0-134-125 kubenswrapper[2579]: W1104 02:31:53.920237 2579 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 04 02:31:53.922776 ip-10-0-134-125 kubenswrapper[2579]: W1104 02:31:53.920241 2579 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 04 02:31:53.922776 ip-10-0-134-125 kubenswrapper[2579]: W1104 02:31:53.920246 2579 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 04 02:31:53.922776 ip-10-0-134-125 kubenswrapper[2579]: W1104 02:31:53.920251 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 04 02:31:53.922776 ip-10-0-134-125 kubenswrapper[2579]: W1104 02:31:53.920256 2579 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 04 02:31:53.922776 ip-10-0-134-125 kubenswrapper[2579]: W1104 02:31:53.920260 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 04 02:31:53.922776 ip-10-0-134-125 kubenswrapper[2579]: W1104 02:31:53.920264 2579 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 04 02:31:53.922776 ip-10-0-134-125 kubenswrapper[2579]: W1104 02:31:53.920269 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 04 02:31:53.922776 ip-10-0-134-125 kubenswrapper[2579]: W1104 02:31:53.920273 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 04 02:31:53.922776 ip-10-0-134-125 kubenswrapper[2579]: W1104 02:31:53.920277 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 04 02:31:53.922776 ip-10-0-134-125 kubenswrapper[2579]: W1104 02:31:53.920282 2579 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 04 02:31:53.922776 ip-10-0-134-125 kubenswrapper[2579]: W1104 02:31:53.920286 2579 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 04 02:31:53.922776 ip-10-0-134-125 kubenswrapper[2579]: W1104 02:31:53.920291 2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 04 02:31:53.922776 ip-10-0-134-125 kubenswrapper[2579]: W1104 02:31:53.920295 2579 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 04 02:31:53.922776 ip-10-0-134-125 kubenswrapper[2579]: W1104 02:31:53.920300 2579 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 04 02:31:53.922776 ip-10-0-134-125 kubenswrapper[2579]: W1104 02:31:53.920304 2579 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 04 02:31:53.922776 ip-10-0-134-125 kubenswrapper[2579]: W1104 02:31:53.920308 2579 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 04 02:31:53.922776 ip-10-0-134-125 kubenswrapper[2579]: W1104 02:31:53.920312 2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 04 02:31:53.923607 ip-10-0-134-125 kubenswrapper[2579]: W1104 02:31:53.920318 2579 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 04 02:31:53.923607 ip-10-0-134-125 kubenswrapper[2579]: W1104 02:31:53.920323 2579 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 04 02:31:53.923607 ip-10-0-134-125 kubenswrapper[2579]: W1104 02:31:53.920327 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 04 02:31:53.923607 ip-10-0-134-125 kubenswrapper[2579]: W1104 02:31:53.920339 2579 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 04 02:31:53.923607 ip-10-0-134-125 kubenswrapper[2579]: W1104 02:31:53.920343 2579 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 04 02:31:53.923607 ip-10-0-134-125 kubenswrapper[2579]: W1104 02:31:53.920348 2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 04 02:31:53.923607 ip-10-0-134-125 kubenswrapper[2579]: W1104 02:31:53.920352 2579 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 04 02:31:53.923607 ip-10-0-134-125 kubenswrapper[2579]: W1104 02:31:53.920356 2579 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 04 02:31:53.923607 ip-10-0-134-125 kubenswrapper[2579]: W1104 02:31:53.920365 2579 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 04 02:31:53.923607 ip-10-0-134-125 kubenswrapper[2579]: W1104 02:31:53.920370 2579 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 04 02:31:53.923607 ip-10-0-134-125 kubenswrapper[2579]: W1104 02:31:53.920374 2579 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 04 02:31:53.923607 ip-10-0-134-125 kubenswrapper[2579]: W1104 02:31:53.920378 2579 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 04 02:31:53.923607 ip-10-0-134-125 kubenswrapper[2579]: W1104 02:31:53.920383 2579 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 04 02:31:53.923607 ip-10-0-134-125 kubenswrapper[2579]: W1104 02:31:53.920387 2579 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 04 02:31:53.923607 ip-10-0-134-125 kubenswrapper[2579]: W1104 02:31:53.920392 2579 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 04 02:31:53.923607 ip-10-0-134-125 kubenswrapper[2579]: W1104 02:31:53.920399 2579 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 04 02:31:53.923607 ip-10-0-134-125 kubenswrapper[2579]: W1104 02:31:53.920404 2579 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 04 02:31:53.923607 ip-10-0-134-125 kubenswrapper[2579]: W1104 02:31:53.920408 2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 04 02:31:53.923607 ip-10-0-134-125 kubenswrapper[2579]: W1104 02:31:53.920412 2579 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 04 02:31:53.923607 ip-10-0-134-125 kubenswrapper[2579]: W1104 02:31:53.920417 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 04 02:31:53.924232 ip-10-0-134-125 kubenswrapper[2579]: W1104 02:31:53.920421 2579 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 04 02:31:53.924232 ip-10-0-134-125 kubenswrapper[2579]: W1104 02:31:53.920425 2579 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 04 02:31:53.924232 ip-10-0-134-125 kubenswrapper[2579]: W1104 02:31:53.920430 2579 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 04 02:31:53.924232 ip-10-0-134-125 kubenswrapper[2579]: W1104 02:31:53.920434 2579 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 04 02:31:53.924232 ip-10-0-134-125 kubenswrapper[2579]: W1104 02:31:53.920438 2579 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 04 02:31:53.924232 ip-10-0-134-125 kubenswrapper[2579]: W1104 02:31:53.920443 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 04 02:31:53.924232 ip-10-0-134-125 kubenswrapper[2579]: W1104 02:31:53.921805 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 04 02:31:53.924232 ip-10-0-134-125 kubenswrapper[2579]: W1104 02:31:53.921817 2579 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 04 02:31:53.924232 ip-10-0-134-125 kubenswrapper[2579]: W1104 02:31:53.921822 2579 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 04 02:31:53.924232 ip-10-0-134-125 kubenswrapper[2579]: W1104 02:31:53.921829 2579 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 04 02:31:53.924232 ip-10-0-134-125 kubenswrapper[2579]: W1104 02:31:53.921835 2579 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 04 02:31:53.924232 ip-10-0-134-125 kubenswrapper[2579]: W1104 02:31:53.921840 2579 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 04 02:31:53.924232 ip-10-0-134-125 kubenswrapper[2579]: W1104 02:31:53.921845 2579 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 04 02:31:53.924232 ip-10-0-134-125 kubenswrapper[2579]: W1104 02:31:53.921851 2579 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 04 02:31:53.924232 ip-10-0-134-125 kubenswrapper[2579]: W1104 02:31:53.921855 2579 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 04 02:31:53.924232 ip-10-0-134-125 kubenswrapper[2579]: W1104 02:31:53.921860 2579 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 04 02:31:53.924232 ip-10-0-134-125 kubenswrapper[2579]: W1104 02:31:53.921865 2579 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 04 02:31:53.924232 ip-10-0-134-125 kubenswrapper[2579]: W1104 02:31:53.921869 2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 04 02:31:53.924232 ip-10-0-134-125 kubenswrapper[2579]: W1104 02:31:53.921873 2579 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 04 02:31:53.924775 ip-10-0-134-125 kubenswrapper[2579]: W1104 02:31:53.921877 2579 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 04 02:31:53.924775 ip-10-0-134-125 kubenswrapper[2579]: W1104 02:31:53.921884 2579 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 04 02:31:53.924775 ip-10-0-134-125 kubenswrapper[2579]: W1104 02:31:53.921889 2579 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 04 02:31:53.924775 ip-10-0-134-125 kubenswrapper[2579]: W1104 02:31:53.921893 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 04 02:31:53.924775 ip-10-0-134-125 kubenswrapper[2579]: W1104 02:31:53.921898 2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 04 02:31:53.924775 ip-10-0-134-125 kubenswrapper[2579]: W1104 02:31:53.921902 2579 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 04 02:31:53.924775 ip-10-0-134-125 kubenswrapper[2579]: W1104 02:31:53.921906 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 04 02:31:53.924775 ip-10-0-134-125 kubenswrapper[2579]: W1104 02:31:53.921911 2579 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 04 02:31:53.924775 ip-10-0-134-125 kubenswrapper[2579]: W1104 02:31:53.921915 2579 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 04 02:31:53.924775 ip-10-0-134-125 kubenswrapper[2579]: W1104 02:31:53.921920 2579 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 04 02:31:53.924775 ip-10-0-134-125 kubenswrapper[2579]: W1104 02:31:53.921924 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 04 02:31:53.924775 ip-10-0-134-125 kubenswrapper[2579]: W1104 02:31:53.921928 2579 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 04 02:31:53.924775 ip-10-0-134-125 kubenswrapper[2579]: W1104 02:31:53.921932 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 04 02:31:53.924775 ip-10-0-134-125 kubenswrapper[2579]: W1104 02:31:53.921937 2579 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 04 02:31:53.924775 ip-10-0-134-125 kubenswrapper[2579]: W1104 02:31:53.921941 2579 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 04 02:31:53.924775 ip-10-0-134-125 kubenswrapper[2579]: W1104 02:31:53.921945 2579 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 04 02:31:53.924775 ip-10-0-134-125 kubenswrapper[2579]: W1104 02:31:53.921950 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 04 02:31:53.924775 ip-10-0-134-125 kubenswrapper[2579]: W1104 02:31:53.921955 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 04 02:31:53.924775 ip-10-0-134-125 kubenswrapper[2579]: W1104 02:31:53.921960 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 04 02:31:53.925614 ip-10-0-134-125 kubenswrapper[2579]: W1104 02:31:53.921964 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 04 02:31:53.925614 ip-10-0-134-125 kubenswrapper[2579]: W1104 02:31:53.921968 2579 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 04 02:31:53.925614 ip-10-0-134-125 kubenswrapper[2579]: W1104 02:31:53.921974 2579 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 04 02:31:53.925614 ip-10-0-134-125 kubenswrapper[2579]: W1104 02:31:53.921981 2579 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 04 02:31:53.925614 ip-10-0-134-125 kubenswrapper[2579]: W1104 02:31:53.921985 2579 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 04 02:31:53.925614 ip-10-0-134-125 kubenswrapper[2579]: W1104 02:31:53.921990 2579 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 04 02:31:53.925614 ip-10-0-134-125 kubenswrapper[2579]: W1104 02:31:53.921995 2579 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 04 02:31:53.925614 ip-10-0-134-125 kubenswrapper[2579]: W1104 02:31:53.921999 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 04 02:31:53.925614 ip-10-0-134-125 kubenswrapper[2579]: W1104 02:31:53.922006 2579 feature_gate.go:386] unrecognized feature gate: EVPN Nov 04 02:31:53.925614 ip-10-0-134-125 kubenswrapper[2579]: W1104 02:31:53.922011 2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 04 02:31:53.925614 ip-10-0-134-125 kubenswrapper[2579]: W1104 02:31:53.922016 2579 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 04 02:31:53.925614 ip-10-0-134-125 kubenswrapper[2579]: W1104 02:31:53.922021 2579 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 04 02:31:53.925614 ip-10-0-134-125 kubenswrapper[2579]: W1104 02:31:53.922025 2579 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 04 02:31:53.925614 ip-10-0-134-125 kubenswrapper[2579]: W1104 02:31:53.922029 2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 04 02:31:53.925614 ip-10-0-134-125 kubenswrapper[2579]: W1104 02:31:53.922034 2579 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 04 02:31:53.925614 ip-10-0-134-125 kubenswrapper[2579]: W1104 02:31:53.922039 2579 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 04 02:31:53.925614 ip-10-0-134-125 kubenswrapper[2579]: W1104 02:31:53.922046 2579 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 04 02:31:53.925614 ip-10-0-134-125 kubenswrapper[2579]: W1104 02:31:53.922051 2579 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 04 02:31:53.925614 ip-10-0-134-125 kubenswrapper[2579]: W1104 02:31:53.922055 2579 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 04 02:31:53.925614 ip-10-0-134-125 kubenswrapper[2579]: W1104 02:31:53.922059 2579 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 04 02:31:53.926534 ip-10-0-134-125 kubenswrapper[2579]: W1104 02:31:53.922064 2579 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 04 02:31:53.926534 ip-10-0-134-125 kubenswrapper[2579]: W1104 02:31:53.922068 2579 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 04 02:31:53.926534 ip-10-0-134-125 kubenswrapper[2579]: W1104 02:31:53.922072 2579 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 04 02:31:53.926534 ip-10-0-134-125 kubenswrapper[2579]: W1104 02:31:53.922076 2579 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 04 02:31:53.926534 ip-10-0-134-125 kubenswrapper[2579]: W1104 02:31:53.922080 2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 04 02:31:53.926534 ip-10-0-134-125 kubenswrapper[2579]: W1104 02:31:53.922084 2579 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 04 02:31:53.926534 ip-10-0-134-125 kubenswrapper[2579]: W1104 02:31:53.922088 2579 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 04 02:31:53.926534 ip-10-0-134-125 kubenswrapper[2579]: W1104 02:31:53.922093 2579 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 04 02:31:53.926534 ip-10-0-134-125 kubenswrapper[2579]: W1104 02:31:53.922097 2579 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 04 02:31:53.926534 ip-10-0-134-125 kubenswrapper[2579]: W1104 02:31:53.922101 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 04 02:31:53.926534 ip-10-0-134-125 kubenswrapper[2579]: W1104 02:31:53.922106 2579 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 04 02:31:53.926534 ip-10-0-134-125 kubenswrapper[2579]: W1104 02:31:53.922110 2579 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 04 02:31:53.926534 ip-10-0-134-125 kubenswrapper[2579]: W1104 02:31:53.922114 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 04 02:31:53.926534 ip-10-0-134-125 kubenswrapper[2579]: W1104 02:31:53.922119 2579 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 04 02:31:53.926534 ip-10-0-134-125 kubenswrapper[2579]: W1104 02:31:53.922123 2579 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 04 02:31:53.926534 ip-10-0-134-125 kubenswrapper[2579]: W1104 02:31:53.922127 2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 04 02:31:53.926534 ip-10-0-134-125 kubenswrapper[2579]: W1104 02:31:53.922131 2579 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 04 02:31:53.926534 ip-10-0-134-125 kubenswrapper[2579]: W1104 02:31:53.922136 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 04 02:31:53.926534 ip-10-0-134-125 kubenswrapper[2579]: W1104 02:31:53.922140 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 04 02:31:53.926534 ip-10-0-134-125 kubenswrapper[2579]: W1104 02:31:53.922144 2579 feature_gate.go:386] unrecognized feature gate: Example Nov 04 02:31:53.927407 ip-10-0-134-125 kubenswrapper[2579]: W1104 02:31:53.922148 2579 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 04 02:31:53.927407 ip-10-0-134-125 kubenswrapper[2579]: W1104 02:31:53.922153 2579 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 04 02:31:53.927407 ip-10-0-134-125 kubenswrapper[2579]: W1104 02:31:53.922157 2579 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 04 02:31:53.927407 ip-10-0-134-125 kubenswrapper[2579]: W1104 02:31:53.922162 2579 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 04 02:31:53.927407 ip-10-0-134-125 kubenswrapper[2579]: W1104 02:31:53.922166 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 04 02:31:53.927407 ip-10-0-134-125 kubenswrapper[2579]: W1104 02:31:53.922170 2579 feature_gate.go:386] unrecognized feature gate: Example2 Nov 04 02:31:53.927407 ip-10-0-134-125 kubenswrapper[2579]: W1104 02:31:53.922175 2579 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 04 02:31:53.927407 ip-10-0-134-125 kubenswrapper[2579]: W1104 02:31:53.922179 2579 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 04 02:31:53.927407 ip-10-0-134-125 kubenswrapper[2579]: W1104 02:31:53.922185 2579 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 04 02:31:53.927407 ip-10-0-134-125 kubenswrapper[2579]: W1104 02:31:53.922190 2579 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 04 02:31:53.927407 ip-10-0-134-125 kubenswrapper[2579]: W1104 02:31:53.922194 2579 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 04 02:31:53.927407 ip-10-0-134-125 kubenswrapper[2579]: W1104 02:31:53.922198 2579 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 04 02:31:53.927407 ip-10-0-134-125 kubenswrapper[2579]: W1104 02:31:53.922203 2579 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 04 02:31:53.927407 ip-10-0-134-125 kubenswrapper[2579]: W1104 02:31:53.922207 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 04 02:31:53.927407 ip-10-0-134-125 kubenswrapper[2579]: W1104 02:31:53.922211 2579 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 04 02:31:53.927407 ip-10-0-134-125 kubenswrapper[2579]: W1104 02:31:53.922216 2579 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 04 02:31:53.927407 ip-10-0-134-125 kubenswrapper[2579]: W1104 02:31:53.922220 2579 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 04 02:31:53.927407 ip-10-0-134-125 kubenswrapper[2579]: W1104 02:31:53.922224 2579 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 04 02:31:53.927407 ip-10-0-134-125 kubenswrapper[2579]: W1104 02:31:53.922228 2579 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 04 02:31:53.928086 ip-10-0-134-125 kubenswrapper[2579]: W1104 02:31:53.922233 2579 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 04 02:31:53.928086 ip-10-0-134-125 kubenswrapper[2579]: W1104 02:31:53.922238 2579 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 04 02:31:53.928086 ip-10-0-134-125 kubenswrapper[2579]: W1104 02:31:53.922242 2579 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 04 02:31:53.928086 ip-10-0-134-125 kubenswrapper[2579]: W1104 02:31:53.922246 2579 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 04 02:31:53.928086 ip-10-0-134-125 kubenswrapper[2579]: W1104 02:31:53.922250 2579 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 04 02:31:53.928086 ip-10-0-134-125 kubenswrapper[2579]: W1104 02:31:53.922254 2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 04 02:31:53.928086 ip-10-0-134-125 kubenswrapper[2579]: W1104 02:31:53.922259 2579 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 04 02:31:53.928086 ip-10-0-134-125 kubenswrapper[2579]: W1104 02:31:53.922263 2579 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 04 02:31:53.928086 ip-10-0-134-125 kubenswrapper[2579]: W1104 02:31:53.922269 2579 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 04 02:31:53.928086 ip-10-0-134-125 kubenswrapper[2579]: W1104 02:31:53.922273 2579 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 04 02:31:53.928086 ip-10-0-134-125 kubenswrapper[2579]: W1104 02:31:53.922278 2579 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 04 02:31:53.928086 ip-10-0-134-125 kubenswrapper[2579]: W1104 02:31:53.922282 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 04 02:31:53.928086 ip-10-0-134-125 kubenswrapper[2579]: W1104 02:31:53.922286 2579 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 04 02:31:53.928086 ip-10-0-134-125 kubenswrapper[2579]: W1104 02:31:53.922291 2579 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 04 02:31:53.928086 ip-10-0-134-125 kubenswrapper[2579]: W1104 02:31:53.922295 2579 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 04 02:31:53.928086 ip-10-0-134-125 kubenswrapper[2579]: W1104 02:31:53.923150 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 04 02:31:53.928086 ip-10-0-134-125 kubenswrapper[2579]: W1104 02:31:53.923160 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 04 02:31:53.928086 ip-10-0-134-125 kubenswrapper[2579]: W1104 02:31:53.923165 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 04 02:31:53.928086 ip-10-0-134-125 kubenswrapper[2579]: W1104 02:31:53.923170 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 04 02:31:53.928736 ip-10-0-134-125 kubenswrapper[2579]: W1104 02:31:53.923175 2579 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 04 02:31:53.928736 ip-10-0-134-125 kubenswrapper[2579]: W1104 02:31:53.923180 2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 04 02:31:53.928736 ip-10-0-134-125 kubenswrapper[2579]: W1104 02:31:53.923186 2579 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 04 02:31:53.928736 ip-10-0-134-125 kubenswrapper[2579]: W1104 02:31:53.923191 2579 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 04 02:31:53.928736 ip-10-0-134-125 kubenswrapper[2579]: W1104 02:31:53.923195 2579 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 04 02:31:53.928736 ip-10-0-134-125 kubenswrapper[2579]: W1104 02:31:53.923200 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 04 02:31:53.928736 ip-10-0-134-125 kubenswrapper[2579]: W1104 02:31:53.923204 2579 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 04 02:31:53.928736 ip-10-0-134-125 kubenswrapper[2579]: W1104 02:31:53.923208 2579 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 04 02:31:53.928736 ip-10-0-134-125 kubenswrapper[2579]: W1104 02:31:53.923213 2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 04 02:31:53.928736 ip-10-0-134-125 kubenswrapper[2579]: W1104 02:31:53.923217 2579 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 04 02:31:53.928736 ip-10-0-134-125 kubenswrapper[2579]: W1104 02:31:53.923221 2579 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 04 02:31:53.928736 ip-10-0-134-125 kubenswrapper[2579]: W1104 02:31:53.923226 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 04 02:31:53.928736 ip-10-0-134-125 kubenswrapper[2579]: W1104 02:31:53.923230 2579 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 04 02:31:53.928736 ip-10-0-134-125 kubenswrapper[2579]: W1104 02:31:53.923235 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 04 02:31:53.928736 ip-10-0-134-125 kubenswrapper[2579]: W1104 02:31:53.923239 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 04 02:31:53.928736 ip-10-0-134-125 kubenswrapper[2579]: W1104 02:31:53.923243 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 04 02:31:53.928736 ip-10-0-134-125 kubenswrapper[2579]: W1104 02:31:53.923247 2579 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 04 02:31:53.928736 ip-10-0-134-125 kubenswrapper[2579]: W1104 02:31:53.923252 2579 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 04 02:31:53.928736 ip-10-0-134-125 kubenswrapper[2579]: W1104 02:31:53.923256 2579 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 04 02:31:53.929241 ip-10-0-134-125 kubenswrapper[2579]: W1104 02:31:53.923261 2579 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 04 02:31:53.929241 ip-10-0-134-125 kubenswrapper[2579]: W1104 02:31:53.923265 2579 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 04 02:31:53.929241 ip-10-0-134-125 kubenswrapper[2579]: W1104 02:31:53.923269 2579 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 04 02:31:53.929241 ip-10-0-134-125 kubenswrapper[2579]: W1104 02:31:53.923274 2579 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 04 02:31:53.929241 ip-10-0-134-125 kubenswrapper[2579]: W1104 02:31:53.923278 2579 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 04 02:31:53.929241 ip-10-0-134-125 kubenswrapper[2579]: W1104 02:31:53.923282 2579 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 04 02:31:53.929241 ip-10-0-134-125 kubenswrapper[2579]: W1104 02:31:53.923286 2579 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 04 02:31:53.929241 ip-10-0-134-125 kubenswrapper[2579]: W1104 02:31:53.923291 2579 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 04 02:31:53.929241 ip-10-0-134-125 kubenswrapper[2579]: W1104 02:31:53.923295 2579 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 04 02:31:53.929241 ip-10-0-134-125 kubenswrapper[2579]: W1104 02:31:53.923301 2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 04 02:31:53.929241 ip-10-0-134-125 kubenswrapper[2579]: W1104 02:31:53.923306 2579 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 04 02:31:53.929241 ip-10-0-134-125 kubenswrapper[2579]: W1104 02:31:53.923310 2579 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 04 02:31:53.929241 ip-10-0-134-125 kubenswrapper[2579]: W1104 02:31:53.923314 2579 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 04 02:31:53.929241 ip-10-0-134-125 kubenswrapper[2579]: W1104 02:31:53.923319 2579 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 04 02:31:53.929241 ip-10-0-134-125 kubenswrapper[2579]: W1104 02:31:53.923323 2579 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 04 02:31:53.929241 ip-10-0-134-125 kubenswrapper[2579]: W1104 02:31:53.923329 2579 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 04 02:31:53.929241 ip-10-0-134-125 kubenswrapper[2579]: W1104 02:31:53.923333 2579 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 04 02:31:53.929241 ip-10-0-134-125 kubenswrapper[2579]: W1104 02:31:53.923337 2579 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 04 02:31:53.929241 ip-10-0-134-125 kubenswrapper[2579]: W1104 02:31:53.923341 2579 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 04 02:31:53.929241 ip-10-0-134-125 kubenswrapper[2579]: W1104 02:31:53.923346 2579 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 04 02:31:53.929868 ip-10-0-134-125 kubenswrapper[2579]: W1104 02:31:53.923350 2579 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 04 02:31:53.929868 ip-10-0-134-125 kubenswrapper[2579]: W1104 02:31:53.923354 2579 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 04 02:31:53.929868 ip-10-0-134-125 kubenswrapper[2579]: W1104 02:31:53.923359 2579 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 04 02:31:53.929868 ip-10-0-134-125 kubenswrapper[2579]: W1104 02:31:53.923363 2579 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 04 02:31:53.929868 ip-10-0-134-125 kubenswrapper[2579]: W1104 02:31:53.923367 2579 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 04 02:31:53.929868 ip-10-0-134-125 kubenswrapper[2579]: W1104 02:31:53.923372 2579 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 04 02:31:53.929868 ip-10-0-134-125 kubenswrapper[2579]: W1104 02:31:53.923376 2579 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 04 02:31:53.929868 ip-10-0-134-125 kubenswrapper[2579]: W1104 02:31:53.923380 2579 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 04 02:31:53.929868 ip-10-0-134-125 kubenswrapper[2579]: W1104 02:31:53.923384 2579 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 04 02:31:53.929868 ip-10-0-134-125 kubenswrapper[2579]: W1104 02:31:53.923390 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 04 02:31:53.929868 ip-10-0-134-125 kubenswrapper[2579]: W1104 02:31:53.923395 2579 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 04 02:31:53.929868 ip-10-0-134-125 kubenswrapper[2579]: W1104 02:31:53.923399 2579 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 04 02:31:53.929868 ip-10-0-134-125 kubenswrapper[2579]: W1104 02:31:53.923403 2579 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 04 02:31:53.929868 ip-10-0-134-125 kubenswrapper[2579]: W1104 02:31:53.923408 2579 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 04 02:31:53.929868 ip-10-0-134-125 kubenswrapper[2579]: W1104 02:31:53.923412 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 04 02:31:53.929868 ip-10-0-134-125 kubenswrapper[2579]: W1104 02:31:53.923418 2579 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 04 02:31:53.929868 ip-10-0-134-125 kubenswrapper[2579]: W1104 02:31:53.923423 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 04 02:31:53.929868 ip-10-0-134-125 kubenswrapper[2579]: W1104 02:31:53.923427 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 04 02:31:53.929868 ip-10-0-134-125 kubenswrapper[2579]: W1104 02:31:53.923431 2579 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 04 02:31:53.930510 ip-10-0-134-125 kubenswrapper[2579]: W1104 02:31:53.923435 2579 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 04 02:31:53.930510 ip-10-0-134-125 kubenswrapper[2579]: W1104 02:31:53.923439 2579 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 04 02:31:53.930510 ip-10-0-134-125 kubenswrapper[2579]: W1104 02:31:53.923443 2579 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 04 02:31:53.930510 ip-10-0-134-125 kubenswrapper[2579]: W1104 02:31:53.923448 2579 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 04 02:31:53.930510 ip-10-0-134-125 kubenswrapper[2579]: W1104 02:31:53.923453 2579 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 04 02:31:53.930510 ip-10-0-134-125 kubenswrapper[2579]: W1104 02:31:53.923457 2579 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 04 02:31:53.930510 ip-10-0-134-125 kubenswrapper[2579]: W1104 02:31:53.923461 2579 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 04 02:31:53.930510 ip-10-0-134-125 kubenswrapper[2579]: W1104 02:31:53.923465 2579 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 04 02:31:53.930510 ip-10-0-134-125 kubenswrapper[2579]: W1104 02:31:53.923473 2579 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 04 02:31:53.930510 ip-10-0-134-125 kubenswrapper[2579]: W1104 02:31:53.923479 2579 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 04 02:31:53.930510 ip-10-0-134-125 kubenswrapper[2579]: W1104 02:31:53.923484 2579 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 04 02:31:53.930510 ip-10-0-134-125 kubenswrapper[2579]: W1104 02:31:53.923489 2579 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 04 02:31:53.930510 ip-10-0-134-125 kubenswrapper[2579]: W1104 02:31:53.923494 2579 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 04 02:31:53.930510 ip-10-0-134-125 kubenswrapper[2579]: W1104 02:31:53.923498 2579 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 04 02:31:53.930510 ip-10-0-134-125 kubenswrapper[2579]: W1104 02:31:53.923502 2579 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 04 02:31:53.930510 ip-10-0-134-125 kubenswrapper[2579]: W1104 02:31:53.923506 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 04 02:31:53.930510 ip-10-0-134-125 kubenswrapper[2579]: W1104 02:31:53.923510 2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 04 02:31:53.930510 ip-10-0-134-125 kubenswrapper[2579]: W1104 02:31:53.923515 2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 04 02:31:53.930510 ip-10-0-134-125 kubenswrapper[2579]: W1104 02:31:53.923519 2579 feature_gate.go:386] unrecognized feature gate: EVPN Nov 04 02:31:53.930510 ip-10-0-134-125 kubenswrapper[2579]: W1104 02:31:53.923523 2579 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 04 02:31:53.931045 ip-10-0-134-125 kubenswrapper[2579]: W1104 02:31:53.923527 2579 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 04 02:31:53.931045 ip-10-0-134-125 kubenswrapper[2579]: W1104 02:31:53.923532 2579 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 04 02:31:53.931045 ip-10-0-134-125 kubenswrapper[2579]: W1104 02:31:53.923537 2579 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 04 02:31:53.931045 ip-10-0-134-125 kubenswrapper[2579]: W1104 02:31:53.923542 2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 04 02:31:53.931045 ip-10-0-134-125 kubenswrapper[2579]: W1104 02:31:53.923546 2579 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 04 02:31:53.931045 ip-10-0-134-125 kubenswrapper[2579]: W1104 02:31:53.923550 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 04 02:31:53.931045 ip-10-0-134-125 kubenswrapper[2579]: W1104 02:31:53.923557 2579 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 04 02:31:53.931045 ip-10-0-134-125 kubenswrapper[2579]: W1104 02:31:53.923562 2579 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 04 02:31:53.931045 ip-10-0-134-125 kubenswrapper[2579]: W1104 02:31:53.923568 2579 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 04 02:31:53.931045 ip-10-0-134-125 kubenswrapper[2579]: W1104 02:31:53.923573 2579 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 04 02:31:53.931045 ip-10-0-134-125 kubenswrapper[2579]: W1104 02:31:53.923577 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 04 02:31:53.931045 ip-10-0-134-125 kubenswrapper[2579]: W1104 02:31:53.923581 2579 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 04 02:31:53.931045 ip-10-0-134-125 kubenswrapper[2579]: W1104 02:31:53.923586 2579 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 04 02:31:53.931045 ip-10-0-134-125 kubenswrapper[2579]: W1104 02:31:53.923590 2579 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 04 02:31:53.931045 ip-10-0-134-125 kubenswrapper[2579]: W1104 02:31:53.923595 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 04 02:31:53.931045 ip-10-0-134-125 kubenswrapper[2579]: W1104 02:31:53.923599 2579 feature_gate.go:386] unrecognized feature gate: Example Nov 04 02:31:53.931045 ip-10-0-134-125 kubenswrapper[2579]: W1104 02:31:53.923604 2579 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 04 02:31:53.931045 ip-10-0-134-125 kubenswrapper[2579]: W1104 02:31:53.923608 2579 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 04 02:31:53.931045 ip-10-0-134-125 kubenswrapper[2579]: W1104 02:31:53.923614 2579 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 04 02:31:53.931045 ip-10-0-134-125 kubenswrapper[2579]: W1104 02:31:53.923618 2579 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 04 02:31:53.931557 ip-10-0-134-125 kubenswrapper[2579]: W1104 02:31:53.923624 2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 04 02:31:53.931557 ip-10-0-134-125 kubenswrapper[2579]: W1104 02:31:53.923629 2579 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 04 02:31:53.931557 ip-10-0-134-125 kubenswrapper[2579]: W1104 02:31:53.923633 2579 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 04 02:31:53.931557 ip-10-0-134-125 kubenswrapper[2579]: W1104 02:31:53.923637 2579 feature_gate.go:386] unrecognized feature gate: Example2 Nov 04 02:31:53.931557 ip-10-0-134-125 kubenswrapper[2579]: I1104 02:31:53.924913 2579 flags.go:64] FLAG: --address="0.0.0.0" Nov 04 02:31:53.931557 ip-10-0-134-125 kubenswrapper[2579]: I1104 02:31:53.924932 2579 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Nov 04 02:31:53.931557 ip-10-0-134-125 kubenswrapper[2579]: I1104 02:31:53.924943 2579 flags.go:64] FLAG: --anonymous-auth="true" Nov 04 02:31:53.931557 ip-10-0-134-125 kubenswrapper[2579]: I1104 02:31:53.924951 2579 flags.go:64] FLAG: --application-metrics-count-limit="100" Nov 04 02:31:53.931557 ip-10-0-134-125 kubenswrapper[2579]: I1104 02:31:53.924959 2579 flags.go:64] FLAG: --authentication-token-webhook="false" Nov 04 02:31:53.931557 ip-10-0-134-125 kubenswrapper[2579]: I1104 02:31:53.924964 2579 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Nov 04 02:31:53.931557 ip-10-0-134-125 kubenswrapper[2579]: I1104 02:31:53.924971 2579 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Nov 04 02:31:53.931557 ip-10-0-134-125 kubenswrapper[2579]: I1104 02:31:53.924978 2579 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Nov 04 02:31:53.931557 ip-10-0-134-125 kubenswrapper[2579]: I1104 02:31:53.924983 2579 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Nov 04 02:31:53.931557 ip-10-0-134-125 kubenswrapper[2579]: I1104 02:31:53.924988 2579 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Nov 04 02:31:53.931557 ip-10-0-134-125 kubenswrapper[2579]: I1104 02:31:53.924996 2579 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Nov 04 02:31:53.931557 ip-10-0-134-125 kubenswrapper[2579]: I1104 02:31:53.925002 2579 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Nov 04 02:31:53.931557 ip-10-0-134-125 kubenswrapper[2579]: I1104 02:31:53.925007 2579 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Nov 04 02:31:53.931557 ip-10-0-134-125 kubenswrapper[2579]: I1104 02:31:53.925012 2579 flags.go:64] FLAG: --cgroup-root="" Nov 04 02:31:53.931557 ip-10-0-134-125 kubenswrapper[2579]: I1104 02:31:53.925017 2579 flags.go:64] FLAG: --cgroups-per-qos="true" Nov 04 02:31:53.931557 ip-10-0-134-125 kubenswrapper[2579]: I1104 02:31:53.925021 2579 flags.go:64] FLAG: --client-ca-file="" Nov 04 02:31:53.931557 ip-10-0-134-125 kubenswrapper[2579]: I1104 02:31:53.925026 2579 flags.go:64] FLAG: --cloud-provider="external" Nov 04 02:31:53.931557 ip-10-0-134-125 kubenswrapper[2579]: I1104 02:31:53.925031 2579 flags.go:64] FLAG: --cluster-dns="[]" Nov 04 02:31:53.931557 ip-10-0-134-125 kubenswrapper[2579]: I1104 02:31:53.925038 2579 flags.go:64] FLAG: --cluster-domain="" Nov 04 02:31:53.931557 ip-10-0-134-125 kubenswrapper[2579]: I1104 02:31:53.925043 2579 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Nov 04 02:31:53.939207 ip-10-0-134-125 kubenswrapper[2579]: I1104 02:31:53.925049 2579 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Nov 04 02:31:53.939207 ip-10-0-134-125 kubenswrapper[2579]: I1104 02:31:53.925054 2579 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Nov 04 02:31:53.939207 ip-10-0-134-125 kubenswrapper[2579]: I1104 02:31:53.925060 2579 flags.go:64] FLAG: --container-log-max-files="5" Nov 04 02:31:53.939207 ip-10-0-134-125 kubenswrapper[2579]: I1104 02:31:53.925066 2579 flags.go:64] FLAG: --container-log-max-size="10Mi" Nov 04 02:31:53.939207 ip-10-0-134-125 kubenswrapper[2579]: I1104 02:31:53.925071 2579 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Nov 04 02:31:53.939207 ip-10-0-134-125 kubenswrapper[2579]: I1104 02:31:53.925077 2579 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Nov 04 02:31:53.939207 ip-10-0-134-125 kubenswrapper[2579]: I1104 02:31:53.925082 2579 flags.go:64] FLAG: --containerd-namespace="k8s.io" Nov 04 02:31:53.939207 ip-10-0-134-125 kubenswrapper[2579]: I1104 02:31:53.925087 2579 flags.go:64] FLAG: --contention-profiling="false" Nov 04 02:31:53.939207 ip-10-0-134-125 kubenswrapper[2579]: I1104 02:31:53.925094 2579 flags.go:64] FLAG: --cpu-cfs-quota="true" Nov 04 02:31:53.939207 ip-10-0-134-125 kubenswrapper[2579]: I1104 02:31:53.925099 2579 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Nov 04 02:31:53.939207 ip-10-0-134-125 kubenswrapper[2579]: I1104 02:31:53.925104 2579 flags.go:64] FLAG: --cpu-manager-policy="none" Nov 04 02:31:53.939207 ip-10-0-134-125 kubenswrapper[2579]: I1104 02:31:53.925109 2579 flags.go:64] FLAG: --cpu-manager-policy-options="" Nov 04 02:31:53.939207 ip-10-0-134-125 kubenswrapper[2579]: I1104 02:31:53.925116 2579 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Nov 04 02:31:53.939207 ip-10-0-134-125 kubenswrapper[2579]: I1104 02:31:53.925121 2579 flags.go:64] FLAG: --enable-controller-attach-detach="true" Nov 04 02:31:53.939207 ip-10-0-134-125 kubenswrapper[2579]: I1104 02:31:53.925126 2579 flags.go:64] FLAG: --enable-debugging-handlers="true" Nov 04 02:31:53.939207 ip-10-0-134-125 kubenswrapper[2579]: I1104 02:31:53.925131 2579 flags.go:64] FLAG: --enable-load-reader="false" Nov 04 02:31:53.939207 ip-10-0-134-125 kubenswrapper[2579]: I1104 02:31:53.925136 2579 flags.go:64] FLAG: --enable-server="true" Nov 04 02:31:53.939207 ip-10-0-134-125 kubenswrapper[2579]: I1104 02:31:53.925141 2579 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Nov 04 02:31:53.939207 ip-10-0-134-125 kubenswrapper[2579]: I1104 02:31:53.925148 2579 flags.go:64] FLAG: --event-burst="100" Nov 04 02:31:53.939207 ip-10-0-134-125 kubenswrapper[2579]: I1104 02:31:53.925153 2579 flags.go:64] FLAG: --event-qps="50" Nov 04 02:31:53.939207 ip-10-0-134-125 kubenswrapper[2579]: I1104 02:31:53.925158 2579 flags.go:64] FLAG: --event-storage-age-limit="default=0" Nov 04 02:31:53.939207 ip-10-0-134-125 kubenswrapper[2579]: I1104 02:31:53.925163 2579 flags.go:64] FLAG: --event-storage-event-limit="default=0" Nov 04 02:31:53.939207 ip-10-0-134-125 kubenswrapper[2579]: I1104 02:31:53.925169 2579 flags.go:64] FLAG: --eviction-hard="" Nov 04 02:31:53.939207 ip-10-0-134-125 kubenswrapper[2579]: I1104 02:31:53.925176 2579 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Nov 04 02:31:53.939910 ip-10-0-134-125 kubenswrapper[2579]: I1104 02:31:53.925181 2579 flags.go:64] FLAG: --eviction-minimum-reclaim="" Nov 04 02:31:53.939910 ip-10-0-134-125 kubenswrapper[2579]: I1104 02:31:53.925186 2579 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Nov 04 02:31:53.939910 ip-10-0-134-125 kubenswrapper[2579]: I1104 02:31:53.925191 2579 flags.go:64] FLAG: --eviction-soft="" Nov 04 02:31:53.939910 ip-10-0-134-125 kubenswrapper[2579]: I1104 02:31:53.925196 2579 flags.go:64] FLAG: --eviction-soft-grace-period="" Nov 04 02:31:53.939910 ip-10-0-134-125 kubenswrapper[2579]: I1104 02:31:53.925201 2579 flags.go:64] FLAG: --exit-on-lock-contention="false" Nov 04 02:31:53.939910 ip-10-0-134-125 kubenswrapper[2579]: I1104 02:31:53.925206 2579 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Nov 04 02:31:53.939910 ip-10-0-134-125 kubenswrapper[2579]: I1104 02:31:53.925211 2579 flags.go:64] FLAG: --experimental-mounter-path="" Nov 04 02:31:53.939910 ip-10-0-134-125 kubenswrapper[2579]: I1104 02:31:53.925215 2579 flags.go:64] FLAG: --fail-cgroupv1="true" Nov 04 02:31:53.939910 ip-10-0-134-125 kubenswrapper[2579]: I1104 02:31:53.925220 2579 flags.go:64] FLAG: --fail-swap-on="true" Nov 04 02:31:53.939910 ip-10-0-134-125 kubenswrapper[2579]: I1104 02:31:53.925225 2579 flags.go:64] FLAG: --feature-gates="" Nov 04 02:31:53.939910 ip-10-0-134-125 kubenswrapper[2579]: I1104 02:31:53.925232 2579 flags.go:64] FLAG: --file-check-frequency="20s" Nov 04 02:31:53.939910 ip-10-0-134-125 kubenswrapper[2579]: I1104 02:31:53.925237 2579 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Nov 04 02:31:53.939910 ip-10-0-134-125 kubenswrapper[2579]: I1104 02:31:53.925242 2579 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Nov 04 02:31:53.939910 ip-10-0-134-125 kubenswrapper[2579]: I1104 02:31:53.925247 2579 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Nov 04 02:31:53.939910 ip-10-0-134-125 kubenswrapper[2579]: I1104 02:31:53.925253 2579 flags.go:64] FLAG: --healthz-port="10248" Nov 04 02:31:53.939910 ip-10-0-134-125 kubenswrapper[2579]: I1104 02:31:53.925258 2579 flags.go:64] FLAG: --help="false" Nov 04 02:31:53.939910 ip-10-0-134-125 kubenswrapper[2579]: I1104 02:31:53.925264 2579 flags.go:64] FLAG: --hostname-override="ip-10-0-134-125.us-west-1.compute.internal" Nov 04 02:31:53.939910 ip-10-0-134-125 kubenswrapper[2579]: I1104 02:31:53.925269 2579 flags.go:64] FLAG: --housekeeping-interval="10s" Nov 04 02:31:53.939910 ip-10-0-134-125 kubenswrapper[2579]: I1104 02:31:53.925274 2579 flags.go:64] FLAG: --http-check-frequency="20s" Nov 04 02:31:53.939910 ip-10-0-134-125 kubenswrapper[2579]: I1104 02:31:53.925279 2579 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Nov 04 02:31:53.939910 ip-10-0-134-125 kubenswrapper[2579]: I1104 02:31:53.925285 2579 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Nov 04 02:31:53.939910 ip-10-0-134-125 kubenswrapper[2579]: I1104 02:31:53.925291 2579 flags.go:64] FLAG: --image-gc-high-threshold="85" Nov 04 02:31:53.939910 ip-10-0-134-125 kubenswrapper[2579]: I1104 02:31:53.925296 2579 flags.go:64] FLAG: --image-gc-low-threshold="80" Nov 04 02:31:53.940500 ip-10-0-134-125 kubenswrapper[2579]: I1104 02:31:53.925301 2579 flags.go:64] FLAG: --image-service-endpoint="" Nov 04 02:31:53.940500 ip-10-0-134-125 kubenswrapper[2579]: I1104 02:31:53.925306 2579 flags.go:64] FLAG: --kernel-memcg-notification="false" Nov 04 02:31:53.940500 ip-10-0-134-125 kubenswrapper[2579]: I1104 02:31:53.925310 2579 flags.go:64] FLAG: --kube-api-burst="100" Nov 04 02:31:53.940500 ip-10-0-134-125 kubenswrapper[2579]: I1104 02:31:53.925315 2579 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Nov 04 02:31:53.940500 ip-10-0-134-125 kubenswrapper[2579]: I1104 02:31:53.925321 2579 flags.go:64] FLAG: --kube-api-qps="50" Nov 04 02:31:53.940500 ip-10-0-134-125 kubenswrapper[2579]: I1104 02:31:53.925325 2579 flags.go:64] FLAG: --kube-reserved="" Nov 04 02:31:53.940500 ip-10-0-134-125 kubenswrapper[2579]: I1104 02:31:53.925330 2579 flags.go:64] FLAG: --kube-reserved-cgroup="" Nov 04 02:31:53.940500 ip-10-0-134-125 kubenswrapper[2579]: I1104 02:31:53.925337 2579 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Nov 04 02:31:53.940500 ip-10-0-134-125 kubenswrapper[2579]: I1104 02:31:53.925342 2579 flags.go:64] FLAG: --kubelet-cgroups="" Nov 04 02:31:53.940500 ip-10-0-134-125 kubenswrapper[2579]: I1104 02:31:53.925347 2579 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Nov 04 02:31:53.940500 ip-10-0-134-125 kubenswrapper[2579]: I1104 02:31:53.925352 2579 flags.go:64] FLAG: --lock-file="" Nov 04 02:31:53.940500 ip-10-0-134-125 kubenswrapper[2579]: I1104 02:31:53.925356 2579 flags.go:64] FLAG: --log-cadvisor-usage="false" Nov 04 02:31:53.940500 ip-10-0-134-125 kubenswrapper[2579]: I1104 02:31:53.925361 2579 flags.go:64] FLAG: --log-flush-frequency="5s" Nov 04 02:31:53.940500 ip-10-0-134-125 kubenswrapper[2579]: I1104 02:31:53.925367 2579 flags.go:64] FLAG: --log-json-info-buffer-size="0" Nov 04 02:31:53.940500 ip-10-0-134-125 kubenswrapper[2579]: I1104 02:31:53.925376 2579 flags.go:64] FLAG: --log-json-split-stream="false" Nov 04 02:31:53.940500 ip-10-0-134-125 kubenswrapper[2579]: I1104 02:31:53.925381 2579 flags.go:64] FLAG: --log-text-info-buffer-size="0" Nov 04 02:31:53.940500 ip-10-0-134-125 kubenswrapper[2579]: I1104 02:31:53.925385 2579 flags.go:64] FLAG: --log-text-split-stream="false" Nov 04 02:31:53.940500 ip-10-0-134-125 kubenswrapper[2579]: I1104 02:31:53.925390 2579 flags.go:64] FLAG: --logging-format="text" Nov 04 02:31:53.940500 ip-10-0-134-125 kubenswrapper[2579]: I1104 02:31:53.925395 2579 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Nov 04 02:31:53.940500 ip-10-0-134-125 kubenswrapper[2579]: I1104 02:31:53.925401 2579 flags.go:64] FLAG: --make-iptables-util-chains="true" Nov 04 02:31:53.940500 ip-10-0-134-125 kubenswrapper[2579]: I1104 02:31:53.925405 2579 flags.go:64] FLAG: --manifest-url="" Nov 04 02:31:53.940500 ip-10-0-134-125 kubenswrapper[2579]: I1104 02:31:53.925411 2579 flags.go:64] FLAG: --manifest-url-header="" Nov 04 02:31:53.940500 ip-10-0-134-125 kubenswrapper[2579]: I1104 02:31:53.925418 2579 flags.go:64] FLAG: --max-housekeeping-interval="15s" Nov 04 02:31:53.940500 ip-10-0-134-125 kubenswrapper[2579]: I1104 02:31:53.925423 2579 flags.go:64] FLAG: --max-open-files="1000000" Nov 04 02:31:53.940500 ip-10-0-134-125 kubenswrapper[2579]: I1104 02:31:53.925430 2579 flags.go:64] FLAG: --max-pods="110" Nov 04 02:31:53.941258 ip-10-0-134-125 kubenswrapper[2579]: I1104 02:31:53.925436 2579 flags.go:64] FLAG: --maximum-dead-containers="-1" Nov 04 02:31:53.941258 ip-10-0-134-125 kubenswrapper[2579]: I1104 02:31:53.925440 2579 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Nov 04 02:31:53.941258 ip-10-0-134-125 kubenswrapper[2579]: I1104 02:31:53.925446 2579 flags.go:64] FLAG: --memory-manager-policy="None" Nov 04 02:31:53.941258 ip-10-0-134-125 kubenswrapper[2579]: I1104 02:31:53.925450 2579 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Nov 04 02:31:53.941258 ip-10-0-134-125 kubenswrapper[2579]: I1104 02:31:53.925455 2579 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Nov 04 02:31:53.941258 ip-10-0-134-125 kubenswrapper[2579]: I1104 02:31:53.925460 2579 flags.go:64] FLAG: --node-ip="0.0.0.0" Nov 04 02:31:53.941258 ip-10-0-134-125 kubenswrapper[2579]: I1104 02:31:53.925465 2579 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Nov 04 02:31:53.941258 ip-10-0-134-125 kubenswrapper[2579]: I1104 02:31:53.925477 2579 flags.go:64] FLAG: --node-status-max-images="50" Nov 04 02:31:53.941258 ip-10-0-134-125 kubenswrapper[2579]: I1104 02:31:53.925482 2579 flags.go:64] FLAG: --node-status-update-frequency="10s" Nov 04 02:31:53.941258 ip-10-0-134-125 kubenswrapper[2579]: I1104 02:31:53.925488 2579 flags.go:64] FLAG: --oom-score-adj="-999" Nov 04 02:31:53.941258 ip-10-0-134-125 kubenswrapper[2579]: I1104 02:31:53.925493 2579 flags.go:64] FLAG: --pod-cidr="" Nov 04 02:31:53.941258 ip-10-0-134-125 kubenswrapper[2579]: I1104 02:31:53.925497 2579 flags.go:64] FLAG: --pod-manifest-path="" Nov 04 02:31:53.941258 ip-10-0-134-125 kubenswrapper[2579]: I1104 02:31:53.925510 2579 flags.go:64] FLAG: --pod-max-pids="-1" Nov 04 02:31:53.941258 ip-10-0-134-125 kubenswrapper[2579]: I1104 02:31:53.925515 2579 flags.go:64] FLAG: --pods-per-core="0" Nov 04 02:31:53.941258 ip-10-0-134-125 kubenswrapper[2579]: I1104 02:31:53.925523 2579 flags.go:64] FLAG: --port="10250" Nov 04 02:31:53.941258 ip-10-0-134-125 kubenswrapper[2579]: I1104 02:31:53.925528 2579 flags.go:64] FLAG: --protect-kernel-defaults="false" Nov 04 02:31:53.941258 ip-10-0-134-125 kubenswrapper[2579]: I1104 02:31:53.925533 2579 flags.go:64] FLAG: --provider-id="aws:///us-west-1a/i-0e1ce850fea14055b" Nov 04 02:31:53.941258 ip-10-0-134-125 kubenswrapper[2579]: I1104 02:31:53.925539 2579 flags.go:64] FLAG: --qos-reserved="" Nov 04 02:31:53.941258 ip-10-0-134-125 kubenswrapper[2579]: I1104 02:31:53.925544 2579 flags.go:64] FLAG: --read-only-port="10255" Nov 04 02:31:53.941258 ip-10-0-134-125 kubenswrapper[2579]: I1104 02:31:53.925549 2579 flags.go:64] FLAG: --register-node="true" Nov 04 02:31:53.941258 ip-10-0-134-125 kubenswrapper[2579]: I1104 02:31:53.925553 2579 flags.go:64] FLAG: --register-with-taints="" Nov 04 02:31:53.941258 ip-10-0-134-125 kubenswrapper[2579]: I1104 02:31:53.925560 2579 flags.go:64] FLAG: --registry-burst="10" Nov 04 02:31:53.941258 ip-10-0-134-125 kubenswrapper[2579]: I1104 02:31:53.925564 2579 flags.go:64] FLAG: --registry-qps="5" Nov 04 02:31:53.941258 ip-10-0-134-125 kubenswrapper[2579]: I1104 02:31:53.925569 2579 flags.go:64] FLAG: --reserved-cpus="" Nov 04 02:31:53.941258 ip-10-0-134-125 kubenswrapper[2579]: I1104 02:31:53.925573 2579 flags.go:64] FLAG: --reserved-memory="" Nov 04 02:31:53.941258 ip-10-0-134-125 kubenswrapper[2579]: I1104 02:31:53.925580 2579 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Nov 04 02:31:53.941957 ip-10-0-134-125 kubenswrapper[2579]: I1104 02:31:53.925585 2579 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Nov 04 02:31:53.941957 ip-10-0-134-125 kubenswrapper[2579]: I1104 02:31:53.925591 2579 flags.go:64] FLAG: --rotate-certificates="false" Nov 04 02:31:53.941957 ip-10-0-134-125 kubenswrapper[2579]: I1104 02:31:53.925596 2579 flags.go:64] FLAG: --rotate-server-certificates="false" Nov 04 02:31:53.941957 ip-10-0-134-125 kubenswrapper[2579]: I1104 02:31:53.925601 2579 flags.go:64] FLAG: --runonce="false" Nov 04 02:31:53.941957 ip-10-0-134-125 kubenswrapper[2579]: I1104 02:31:53.925606 2579 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Nov 04 02:31:53.941957 ip-10-0-134-125 kubenswrapper[2579]: I1104 02:31:53.925611 2579 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Nov 04 02:31:53.941957 ip-10-0-134-125 kubenswrapper[2579]: I1104 02:31:53.925617 2579 flags.go:64] FLAG: --seccomp-default="false" Nov 04 02:31:53.941957 ip-10-0-134-125 kubenswrapper[2579]: I1104 02:31:53.925621 2579 flags.go:64] FLAG: --serialize-image-pulls="true" Nov 04 02:31:53.941957 ip-10-0-134-125 kubenswrapper[2579]: I1104 02:31:53.925626 2579 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Nov 04 02:31:53.941957 ip-10-0-134-125 kubenswrapper[2579]: I1104 02:31:53.925631 2579 flags.go:64] FLAG: --storage-driver-db="cadvisor" Nov 04 02:31:53.941957 ip-10-0-134-125 kubenswrapper[2579]: I1104 02:31:53.925635 2579 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Nov 04 02:31:53.941957 ip-10-0-134-125 kubenswrapper[2579]: I1104 02:31:53.925640 2579 flags.go:64] FLAG: --storage-driver-password="root" Nov 04 02:31:53.941957 ip-10-0-134-125 kubenswrapper[2579]: I1104 02:31:53.925645 2579 flags.go:64] FLAG: --storage-driver-secure="false" Nov 04 02:31:53.941957 ip-10-0-134-125 kubenswrapper[2579]: I1104 02:31:53.925650 2579 flags.go:64] FLAG: --storage-driver-table="stats" Nov 04 02:31:53.941957 ip-10-0-134-125 kubenswrapper[2579]: I1104 02:31:53.925655 2579 flags.go:64] FLAG: --storage-driver-user="root" Nov 04 02:31:53.941957 ip-10-0-134-125 kubenswrapper[2579]: I1104 02:31:53.925660 2579 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Nov 04 02:31:53.941957 ip-10-0-134-125 kubenswrapper[2579]: I1104 02:31:53.925681 2579 flags.go:64] FLAG: --sync-frequency="1m0s" Nov 04 02:31:53.941957 ip-10-0-134-125 kubenswrapper[2579]: I1104 02:31:53.925687 2579 flags.go:64] FLAG: --system-cgroups="" Nov 04 02:31:53.941957 ip-10-0-134-125 kubenswrapper[2579]: I1104 02:31:53.925692 2579 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Nov 04 02:31:53.941957 ip-10-0-134-125 kubenswrapper[2579]: I1104 02:31:53.925701 2579 flags.go:64] FLAG: --system-reserved-cgroup="" Nov 04 02:31:53.941957 ip-10-0-134-125 kubenswrapper[2579]: I1104 02:31:53.925707 2579 flags.go:64] FLAG: --tls-cert-file="" Nov 04 02:31:53.941957 ip-10-0-134-125 kubenswrapper[2579]: I1104 02:31:53.925712 2579 flags.go:64] FLAG: --tls-cipher-suites="[]" Nov 04 02:31:53.941957 ip-10-0-134-125 kubenswrapper[2579]: I1104 02:31:53.925719 2579 flags.go:64] FLAG: --tls-min-version="" Nov 04 02:31:53.941957 ip-10-0-134-125 kubenswrapper[2579]: I1104 02:31:53.925724 2579 flags.go:64] FLAG: --tls-private-key-file="" Nov 04 02:31:53.941957 ip-10-0-134-125 kubenswrapper[2579]: I1104 02:31:53.925728 2579 flags.go:64] FLAG: --topology-manager-policy="none" Nov 04 02:31:53.942593 ip-10-0-134-125 kubenswrapper[2579]: I1104 02:31:53.925733 2579 flags.go:64] FLAG: --topology-manager-policy-options="" Nov 04 02:31:53.942593 ip-10-0-134-125 kubenswrapper[2579]: I1104 02:31:53.925738 2579 flags.go:64] FLAG: --topology-manager-scope="container" Nov 04 02:31:53.942593 ip-10-0-134-125 kubenswrapper[2579]: I1104 02:31:53.925745 2579 flags.go:64] FLAG: --v="2" Nov 04 02:31:53.942593 ip-10-0-134-125 kubenswrapper[2579]: I1104 02:31:53.925752 2579 flags.go:64] FLAG: --version="false" Nov 04 02:31:53.942593 ip-10-0-134-125 kubenswrapper[2579]: I1104 02:31:53.925759 2579 flags.go:64] FLAG: --vmodule="" Nov 04 02:31:53.942593 ip-10-0-134-125 kubenswrapper[2579]: I1104 02:31:53.925765 2579 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Nov 04 02:31:53.942593 ip-10-0-134-125 kubenswrapper[2579]: I1104 02:31:53.925771 2579 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Nov 04 02:31:53.942593 ip-10-0-134-125 kubenswrapper[2579]: W1104 02:31:53.926024 2579 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 04 02:31:53.942593 ip-10-0-134-125 kubenswrapper[2579]: W1104 02:31:53.926032 2579 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 04 02:31:53.942593 ip-10-0-134-125 kubenswrapper[2579]: W1104 02:31:53.926037 2579 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 04 02:31:53.942593 ip-10-0-134-125 kubenswrapper[2579]: W1104 02:31:53.926042 2579 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 04 02:31:53.942593 ip-10-0-134-125 kubenswrapper[2579]: W1104 02:31:53.926048 2579 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 04 02:31:53.942593 ip-10-0-134-125 kubenswrapper[2579]: W1104 02:31:53.926052 2579 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 04 02:31:53.942593 ip-10-0-134-125 kubenswrapper[2579]: W1104 02:31:53.926057 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 04 02:31:53.942593 ip-10-0-134-125 kubenswrapper[2579]: W1104 02:31:53.926063 2579 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 04 02:31:53.942593 ip-10-0-134-125 kubenswrapper[2579]: W1104 02:31:53.926068 2579 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 04 02:31:53.942593 ip-10-0-134-125 kubenswrapper[2579]: W1104 02:31:53.926073 2579 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 04 02:31:53.942593 ip-10-0-134-125 kubenswrapper[2579]: W1104 02:31:53.926077 2579 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 04 02:31:53.942593 ip-10-0-134-125 kubenswrapper[2579]: W1104 02:31:53.926081 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 04 02:31:53.942593 ip-10-0-134-125 kubenswrapper[2579]: W1104 02:31:53.926086 2579 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 04 02:31:53.942593 ip-10-0-134-125 kubenswrapper[2579]: W1104 02:31:53.926090 2579 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 04 02:31:53.943213 ip-10-0-134-125 kubenswrapper[2579]: W1104 02:31:53.926095 2579 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 04 02:31:53.943213 ip-10-0-134-125 kubenswrapper[2579]: W1104 02:31:53.926099 2579 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 04 02:31:53.943213 ip-10-0-134-125 kubenswrapper[2579]: W1104 02:31:53.926103 2579 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 04 02:31:53.943213 ip-10-0-134-125 kubenswrapper[2579]: W1104 02:31:53.926107 2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 04 02:31:53.943213 ip-10-0-134-125 kubenswrapper[2579]: W1104 02:31:53.926112 2579 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 04 02:31:53.943213 ip-10-0-134-125 kubenswrapper[2579]: W1104 02:31:53.926117 2579 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 04 02:31:53.943213 ip-10-0-134-125 kubenswrapper[2579]: W1104 02:31:53.926123 2579 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 04 02:31:53.943213 ip-10-0-134-125 kubenswrapper[2579]: W1104 02:31:53.926127 2579 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 04 02:31:53.943213 ip-10-0-134-125 kubenswrapper[2579]: W1104 02:31:53.926132 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 04 02:31:53.943213 ip-10-0-134-125 kubenswrapper[2579]: W1104 02:31:53.926136 2579 feature_gate.go:386] unrecognized feature gate: Example2 Nov 04 02:31:53.943213 ip-10-0-134-125 kubenswrapper[2579]: W1104 02:31:53.926157 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 04 02:31:53.943213 ip-10-0-134-125 kubenswrapper[2579]: W1104 02:31:53.926163 2579 feature_gate.go:386] unrecognized feature gate: EVPN Nov 04 02:31:53.943213 ip-10-0-134-125 kubenswrapper[2579]: W1104 02:31:53.926169 2579 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 04 02:31:53.943213 ip-10-0-134-125 kubenswrapper[2579]: W1104 02:31:53.926175 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 04 02:31:53.943213 ip-10-0-134-125 kubenswrapper[2579]: W1104 02:31:53.926179 2579 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 04 02:31:53.943213 ip-10-0-134-125 kubenswrapper[2579]: W1104 02:31:53.926183 2579 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 04 02:31:53.943213 ip-10-0-134-125 kubenswrapper[2579]: W1104 02:31:53.926188 2579 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 04 02:31:53.943213 ip-10-0-134-125 kubenswrapper[2579]: W1104 02:31:53.926192 2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 04 02:31:53.943213 ip-10-0-134-125 kubenswrapper[2579]: W1104 02:31:53.926196 2579 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 04 02:31:53.943213 ip-10-0-134-125 kubenswrapper[2579]: W1104 02:31:53.926201 2579 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 04 02:31:53.943893 ip-10-0-134-125 kubenswrapper[2579]: W1104 02:31:53.926205 2579 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 04 02:31:53.943893 ip-10-0-134-125 kubenswrapper[2579]: W1104 02:31:53.926209 2579 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 04 02:31:53.943893 ip-10-0-134-125 kubenswrapper[2579]: W1104 02:31:53.926214 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 04 02:31:53.943893 ip-10-0-134-125 kubenswrapper[2579]: W1104 02:31:53.926218 2579 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 04 02:31:53.943893 ip-10-0-134-125 kubenswrapper[2579]: W1104 02:31:53.926223 2579 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 04 02:31:53.943893 ip-10-0-134-125 kubenswrapper[2579]: W1104 02:31:53.926227 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 04 02:31:53.943893 ip-10-0-134-125 kubenswrapper[2579]: W1104 02:31:53.926232 2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 04 02:31:53.943893 ip-10-0-134-125 kubenswrapper[2579]: W1104 02:31:53.926236 2579 feature_gate.go:386] unrecognized feature gate: Example Nov 04 02:31:53.943893 ip-10-0-134-125 kubenswrapper[2579]: W1104 02:31:53.926240 2579 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 04 02:31:53.943893 ip-10-0-134-125 kubenswrapper[2579]: W1104 02:31:53.926245 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 04 02:31:53.943893 ip-10-0-134-125 kubenswrapper[2579]: W1104 02:31:53.926249 2579 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 04 02:31:53.943893 ip-10-0-134-125 kubenswrapper[2579]: W1104 02:31:53.926253 2579 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 04 02:31:53.943893 ip-10-0-134-125 kubenswrapper[2579]: W1104 02:31:53.926258 2579 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 04 02:31:53.943893 ip-10-0-134-125 kubenswrapper[2579]: W1104 02:31:53.926262 2579 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 04 02:31:53.943893 ip-10-0-134-125 kubenswrapper[2579]: W1104 02:31:53.926266 2579 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 04 02:31:53.943893 ip-10-0-134-125 kubenswrapper[2579]: W1104 02:31:53.926271 2579 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 04 02:31:53.943893 ip-10-0-134-125 kubenswrapper[2579]: W1104 02:31:53.926275 2579 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 04 02:31:53.943893 ip-10-0-134-125 kubenswrapper[2579]: W1104 02:31:53.926279 2579 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 04 02:31:53.943893 ip-10-0-134-125 kubenswrapper[2579]: W1104 02:31:53.926288 2579 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 04 02:31:53.944626 ip-10-0-134-125 kubenswrapper[2579]: W1104 02:31:53.926294 2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 04 02:31:53.944626 ip-10-0-134-125 kubenswrapper[2579]: W1104 02:31:53.926299 2579 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 04 02:31:53.944626 ip-10-0-134-125 kubenswrapper[2579]: W1104 02:31:53.926304 2579 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 04 02:31:53.944626 ip-10-0-134-125 kubenswrapper[2579]: W1104 02:31:53.926308 2579 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 04 02:31:53.944626 ip-10-0-134-125 kubenswrapper[2579]: W1104 02:31:53.926312 2579 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 04 02:31:53.944626 ip-10-0-134-125 kubenswrapper[2579]: W1104 02:31:53.926316 2579 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 04 02:31:53.944626 ip-10-0-134-125 kubenswrapper[2579]: W1104 02:31:53.926323 2579 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 04 02:31:53.944626 ip-10-0-134-125 kubenswrapper[2579]: W1104 02:31:53.926327 2579 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 04 02:31:53.944626 ip-10-0-134-125 kubenswrapper[2579]: W1104 02:31:53.926332 2579 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 04 02:31:53.944626 ip-10-0-134-125 kubenswrapper[2579]: W1104 02:31:53.926336 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 04 02:31:53.944626 ip-10-0-134-125 kubenswrapper[2579]: W1104 02:31:53.926341 2579 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 04 02:31:53.944626 ip-10-0-134-125 kubenswrapper[2579]: W1104 02:31:53.926345 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 04 02:31:53.944626 ip-10-0-134-125 kubenswrapper[2579]: W1104 02:31:53.926349 2579 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 04 02:31:53.944626 ip-10-0-134-125 kubenswrapper[2579]: W1104 02:31:53.926354 2579 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 04 02:31:53.944626 ip-10-0-134-125 kubenswrapper[2579]: W1104 02:31:53.926358 2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 04 02:31:53.944626 ip-10-0-134-125 kubenswrapper[2579]: W1104 02:31:53.926362 2579 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 04 02:31:53.944626 ip-10-0-134-125 kubenswrapper[2579]: W1104 02:31:53.926366 2579 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 04 02:31:53.944626 ip-10-0-134-125 kubenswrapper[2579]: W1104 02:31:53.926371 2579 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 04 02:31:53.944626 ip-10-0-134-125 kubenswrapper[2579]: W1104 02:31:53.926375 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 04 02:31:53.944626 ip-10-0-134-125 kubenswrapper[2579]: W1104 02:31:53.926379 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 04 02:31:53.945176 ip-10-0-134-125 kubenswrapper[2579]: W1104 02:31:53.926385 2579 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 04 02:31:53.945176 ip-10-0-134-125 kubenswrapper[2579]: W1104 02:31:53.926391 2579 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 04 02:31:53.945176 ip-10-0-134-125 kubenswrapper[2579]: W1104 02:31:53.926396 2579 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 04 02:31:53.945176 ip-10-0-134-125 kubenswrapper[2579]: W1104 02:31:53.926401 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 04 02:31:53.945176 ip-10-0-134-125 kubenswrapper[2579]: W1104 02:31:53.926405 2579 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 04 02:31:53.945176 ip-10-0-134-125 kubenswrapper[2579]: W1104 02:31:53.926410 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 04 02:31:53.945176 ip-10-0-134-125 kubenswrapper[2579]: W1104 02:31:53.926414 2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 04 02:31:53.945176 ip-10-0-134-125 kubenswrapper[2579]: W1104 02:31:53.926419 2579 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 04 02:31:53.945176 ip-10-0-134-125 kubenswrapper[2579]: W1104 02:31:53.926423 2579 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 04 02:31:53.945176 ip-10-0-134-125 kubenswrapper[2579]: W1104 02:31:53.926428 2579 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 04 02:31:53.945176 ip-10-0-134-125 kubenswrapper[2579]: W1104 02:31:53.926432 2579 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 04 02:31:53.945176 ip-10-0-134-125 kubenswrapper[2579]: W1104 02:31:53.926438 2579 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 04 02:31:53.945176 ip-10-0-134-125 kubenswrapper[2579]: W1104 02:31:53.926442 2579 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 04 02:31:53.945176 ip-10-0-134-125 kubenswrapper[2579]: W1104 02:31:53.926446 2579 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 04 02:31:53.945176 ip-10-0-134-125 kubenswrapper[2579]: W1104 02:31:53.926451 2579 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 04 02:31:53.945176 ip-10-0-134-125 kubenswrapper[2579]: W1104 02:31:53.926455 2579 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 04 02:31:53.945176 ip-10-0-134-125 kubenswrapper[2579]: W1104 02:31:53.926459 2579 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 04 02:31:53.945176 ip-10-0-134-125 kubenswrapper[2579]: W1104 02:31:53.926463 2579 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 04 02:31:53.945176 ip-10-0-134-125 kubenswrapper[2579]: W1104 02:31:53.926469 2579 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 04 02:31:53.945686 ip-10-0-134-125 kubenswrapper[2579]: W1104 02:31:53.926474 2579 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 04 02:31:53.945686 ip-10-0-134-125 kubenswrapper[2579]: W1104 02:31:53.926478 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 04 02:31:53.945686 ip-10-0-134-125 kubenswrapper[2579]: W1104 02:31:53.926482 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 04 02:31:53.945686 ip-10-0-134-125 kubenswrapper[2579]: W1104 02:31:53.926486 2579 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 04 02:31:53.945686 ip-10-0-134-125 kubenswrapper[2579]: W1104 02:31:53.926490 2579 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 04 02:31:53.945686 ip-10-0-134-125 kubenswrapper[2579]: W1104 02:31:53.926494 2579 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 04 02:31:53.945686 ip-10-0-134-125 kubenswrapper[2579]: W1104 02:31:53.926499 2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 04 02:31:53.945686 ip-10-0-134-125 kubenswrapper[2579]: W1104 02:31:53.926503 2579 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 04 02:31:53.945686 ip-10-0-134-125 kubenswrapper[2579]: W1104 02:31:53.926507 2579 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 04 02:31:53.945686 ip-10-0-134-125 kubenswrapper[2579]: W1104 02:31:53.926512 2579 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 04 02:31:53.945686 ip-10-0-134-125 kubenswrapper[2579]: W1104 02:31:53.926516 2579 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 04 02:31:53.945686 ip-10-0-134-125 kubenswrapper[2579]: W1104 02:31:53.926521 2579 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 04 02:31:53.945686 ip-10-0-134-125 kubenswrapper[2579]: W1104 02:31:53.926525 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 04 02:31:53.945686 ip-10-0-134-125 kubenswrapper[2579]: W1104 02:31:53.926530 2579 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 04 02:31:53.945686 ip-10-0-134-125 kubenswrapper[2579]: I1104 02:31:53.926573 2579 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Nov 04 02:31:53.945686 ip-10-0-134-125 kubenswrapper[2579]: W1104 02:31:53.926630 2579 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 04 02:31:53.946107 ip-10-0-134-125 kubenswrapper[2579]: W1104 02:31:53.926638 2579 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 04 02:31:53.946107 ip-10-0-134-125 kubenswrapper[2579]: W1104 02:31:53.926643 2579 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 04 02:31:53.946107 ip-10-0-134-125 kubenswrapper[2579]: W1104 02:31:53.926648 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 04 02:31:53.946107 ip-10-0-134-125 kubenswrapper[2579]: W1104 02:31:53.926653 2579 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 04 02:31:53.946107 ip-10-0-134-125 kubenswrapper[2579]: W1104 02:31:53.926657 2579 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 04 02:31:53.946107 ip-10-0-134-125 kubenswrapper[2579]: W1104 02:31:53.926662 2579 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 04 02:31:53.946107 ip-10-0-134-125 kubenswrapper[2579]: W1104 02:31:53.926686 2579 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 04 02:31:53.946107 ip-10-0-134-125 kubenswrapper[2579]: W1104 02:31:53.926690 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 04 02:31:53.946107 ip-10-0-134-125 kubenswrapper[2579]: W1104 02:31:53.926696 2579 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 04 02:31:53.946107 ip-10-0-134-125 kubenswrapper[2579]: W1104 02:31:53.926701 2579 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 04 02:31:53.946107 ip-10-0-134-125 kubenswrapper[2579]: W1104 02:31:53.926705 2579 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 04 02:31:53.946107 ip-10-0-134-125 kubenswrapper[2579]: W1104 02:31:53.926710 2579 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 04 02:31:53.946107 ip-10-0-134-125 kubenswrapper[2579]: W1104 02:31:53.926714 2579 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 04 02:31:53.946107 ip-10-0-134-125 kubenswrapper[2579]: W1104 02:31:53.926718 2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 04 02:31:53.946107 ip-10-0-134-125 kubenswrapper[2579]: W1104 02:31:53.926722 2579 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 04 02:31:53.946107 ip-10-0-134-125 kubenswrapper[2579]: W1104 02:31:53.926727 2579 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 04 02:31:53.946107 ip-10-0-134-125 kubenswrapper[2579]: W1104 02:31:53.926732 2579 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 04 02:31:53.946107 ip-10-0-134-125 kubenswrapper[2579]: W1104 02:31:53.926737 2579 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 04 02:31:53.946107 ip-10-0-134-125 kubenswrapper[2579]: W1104 02:31:53.926741 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 04 02:31:53.946107 ip-10-0-134-125 kubenswrapper[2579]: W1104 02:31:53.926746 2579 feature_gate.go:386] unrecognized feature gate: Example2 Nov 04 02:31:53.946615 ip-10-0-134-125 kubenswrapper[2579]: W1104 02:31:53.926750 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 04 02:31:53.946615 ip-10-0-134-125 kubenswrapper[2579]: W1104 02:31:53.926755 2579 feature_gate.go:386] unrecognized feature gate: EVPN Nov 04 02:31:53.946615 ip-10-0-134-125 kubenswrapper[2579]: W1104 02:31:53.926759 2579 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 04 02:31:53.946615 ip-10-0-134-125 kubenswrapper[2579]: W1104 02:31:53.926764 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 04 02:31:53.946615 ip-10-0-134-125 kubenswrapper[2579]: W1104 02:31:53.926768 2579 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 04 02:31:53.946615 ip-10-0-134-125 kubenswrapper[2579]: W1104 02:31:53.926772 2579 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 04 02:31:53.946615 ip-10-0-134-125 kubenswrapper[2579]: W1104 02:31:53.926777 2579 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 04 02:31:53.946615 ip-10-0-134-125 kubenswrapper[2579]: W1104 02:31:53.926781 2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 04 02:31:53.946615 ip-10-0-134-125 kubenswrapper[2579]: W1104 02:31:53.926785 2579 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 04 02:31:53.946615 ip-10-0-134-125 kubenswrapper[2579]: W1104 02:31:53.926790 2579 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 04 02:31:53.946615 ip-10-0-134-125 kubenswrapper[2579]: W1104 02:31:53.926794 2579 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 04 02:31:53.946615 ip-10-0-134-125 kubenswrapper[2579]: W1104 02:31:53.926800 2579 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 04 02:31:53.946615 ip-10-0-134-125 kubenswrapper[2579]: W1104 02:31:53.926804 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 04 02:31:53.946615 ip-10-0-134-125 kubenswrapper[2579]: W1104 02:31:53.926809 2579 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 04 02:31:53.946615 ip-10-0-134-125 kubenswrapper[2579]: W1104 02:31:53.926813 2579 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 04 02:31:53.946615 ip-10-0-134-125 kubenswrapper[2579]: W1104 02:31:53.926818 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 04 02:31:53.946615 ip-10-0-134-125 kubenswrapper[2579]: W1104 02:31:53.926822 2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 04 02:31:53.946615 ip-10-0-134-125 kubenswrapper[2579]: W1104 02:31:53.926826 2579 feature_gate.go:386] unrecognized feature gate: Example Nov 04 02:31:53.946615 ip-10-0-134-125 kubenswrapper[2579]: W1104 02:31:53.926830 2579 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 04 02:31:53.947197 ip-10-0-134-125 kubenswrapper[2579]: W1104 02:31:53.926835 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 04 02:31:53.947197 ip-10-0-134-125 kubenswrapper[2579]: W1104 02:31:53.926842 2579 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 04 02:31:53.947197 ip-10-0-134-125 kubenswrapper[2579]: W1104 02:31:53.926846 2579 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 04 02:31:53.947197 ip-10-0-134-125 kubenswrapper[2579]: W1104 02:31:53.926850 2579 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 04 02:31:53.947197 ip-10-0-134-125 kubenswrapper[2579]: W1104 02:31:53.926855 2579 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 04 02:31:53.947197 ip-10-0-134-125 kubenswrapper[2579]: W1104 02:31:53.926859 2579 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 04 02:31:53.947197 ip-10-0-134-125 kubenswrapper[2579]: W1104 02:31:53.926864 2579 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 04 02:31:53.947197 ip-10-0-134-125 kubenswrapper[2579]: W1104 02:31:53.926868 2579 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 04 02:31:53.947197 ip-10-0-134-125 kubenswrapper[2579]: W1104 02:31:53.926873 2579 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 04 02:31:53.947197 ip-10-0-134-125 kubenswrapper[2579]: W1104 02:31:53.926883 2579 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 04 02:31:53.947197 ip-10-0-134-125 kubenswrapper[2579]: W1104 02:31:53.926889 2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 04 02:31:53.947197 ip-10-0-134-125 kubenswrapper[2579]: W1104 02:31:53.926894 2579 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 04 02:31:53.947197 ip-10-0-134-125 kubenswrapper[2579]: W1104 02:31:53.926898 2579 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 04 02:31:53.947197 ip-10-0-134-125 kubenswrapper[2579]: W1104 02:31:53.926903 2579 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 04 02:31:53.947197 ip-10-0-134-125 kubenswrapper[2579]: W1104 02:31:53.926907 2579 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 04 02:31:53.947197 ip-10-0-134-125 kubenswrapper[2579]: W1104 02:31:53.926911 2579 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 04 02:31:53.947197 ip-10-0-134-125 kubenswrapper[2579]: W1104 02:31:53.926915 2579 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 04 02:31:53.947197 ip-10-0-134-125 kubenswrapper[2579]: W1104 02:31:53.926920 2579 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 04 02:31:53.947197 ip-10-0-134-125 kubenswrapper[2579]: W1104 02:31:53.926924 2579 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 04 02:31:53.947693 ip-10-0-134-125 kubenswrapper[2579]: W1104 02:31:53.926929 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 04 02:31:53.947693 ip-10-0-134-125 kubenswrapper[2579]: W1104 02:31:53.926933 2579 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 04 02:31:53.947693 ip-10-0-134-125 kubenswrapper[2579]: W1104 02:31:53.926937 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 04 02:31:53.947693 ip-10-0-134-125 kubenswrapper[2579]: W1104 02:31:53.926942 2579 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 04 02:31:53.947693 ip-10-0-134-125 kubenswrapper[2579]: W1104 02:31:53.926946 2579 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 04 02:31:53.947693 ip-10-0-134-125 kubenswrapper[2579]: W1104 02:31:53.926950 2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 04 02:31:53.947693 ip-10-0-134-125 kubenswrapper[2579]: W1104 02:31:53.926955 2579 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 04 02:31:53.947693 ip-10-0-134-125 kubenswrapper[2579]: W1104 02:31:53.926960 2579 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 04 02:31:53.947693 ip-10-0-134-125 kubenswrapper[2579]: W1104 02:31:53.926965 2579 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 04 02:31:53.947693 ip-10-0-134-125 kubenswrapper[2579]: W1104 02:31:53.926969 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 04 02:31:53.947693 ip-10-0-134-125 kubenswrapper[2579]: W1104 02:31:53.926973 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 04 02:31:53.947693 ip-10-0-134-125 kubenswrapper[2579]: W1104 02:31:53.926978 2579 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 04 02:31:53.947693 ip-10-0-134-125 kubenswrapper[2579]: W1104 02:31:53.926984 2579 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 04 02:31:53.947693 ip-10-0-134-125 kubenswrapper[2579]: W1104 02:31:53.926991 2579 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 04 02:31:53.947693 ip-10-0-134-125 kubenswrapper[2579]: W1104 02:31:53.926998 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 04 02:31:53.947693 ip-10-0-134-125 kubenswrapper[2579]: W1104 02:31:53.927002 2579 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 04 02:31:53.947693 ip-10-0-134-125 kubenswrapper[2579]: W1104 02:31:53.927007 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 04 02:31:53.947693 ip-10-0-134-125 kubenswrapper[2579]: W1104 02:31:53.927011 2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 04 02:31:53.947693 ip-10-0-134-125 kubenswrapper[2579]: W1104 02:31:53.927015 2579 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 04 02:31:53.948217 ip-10-0-134-125 kubenswrapper[2579]: W1104 02:31:53.927020 2579 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 04 02:31:53.948217 ip-10-0-134-125 kubenswrapper[2579]: W1104 02:31:53.927024 2579 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 04 02:31:53.948217 ip-10-0-134-125 kubenswrapper[2579]: W1104 02:31:53.927028 2579 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 04 02:31:53.948217 ip-10-0-134-125 kubenswrapper[2579]: W1104 02:31:53.927035 2579 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 04 02:31:53.948217 ip-10-0-134-125 kubenswrapper[2579]: W1104 02:31:53.927039 2579 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 04 02:31:53.948217 ip-10-0-134-125 kubenswrapper[2579]: W1104 02:31:53.927043 2579 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 04 02:31:53.948217 ip-10-0-134-125 kubenswrapper[2579]: W1104 02:31:53.927048 2579 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 04 02:31:53.948217 ip-10-0-134-125 kubenswrapper[2579]: W1104 02:31:53.927052 2579 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 04 02:31:53.948217 ip-10-0-134-125 kubenswrapper[2579]: W1104 02:31:53.927056 2579 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 04 02:31:53.948217 ip-10-0-134-125 kubenswrapper[2579]: W1104 02:31:53.927061 2579 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 04 02:31:53.948217 ip-10-0-134-125 kubenswrapper[2579]: W1104 02:31:53.927065 2579 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 04 02:31:53.948217 ip-10-0-134-125 kubenswrapper[2579]: W1104 02:31:53.927069 2579 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 04 02:31:53.948217 ip-10-0-134-125 kubenswrapper[2579]: W1104 02:31:53.927074 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 04 02:31:53.948217 ip-10-0-134-125 kubenswrapper[2579]: W1104 02:31:53.927078 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 04 02:31:53.948217 ip-10-0-134-125 kubenswrapper[2579]: W1104 02:31:53.927082 2579 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 04 02:31:53.948217 ip-10-0-134-125 kubenswrapper[2579]: W1104 02:31:53.927086 2579 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 04 02:31:53.948217 ip-10-0-134-125 kubenswrapper[2579]: W1104 02:31:53.927091 2579 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 04 02:31:53.948217 ip-10-0-134-125 kubenswrapper[2579]: W1104 02:31:53.927095 2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 04 02:31:53.948217 ip-10-0-134-125 kubenswrapper[2579]: W1104 02:31:53.927100 2579 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 04 02:31:53.948217 ip-10-0-134-125 kubenswrapper[2579]: W1104 02:31:53.927105 2579 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 04 02:31:53.948838 ip-10-0-134-125 kubenswrapper[2579]: W1104 02:31:53.927111 2579 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 04 02:31:53.948838 ip-10-0-134-125 kubenswrapper[2579]: W1104 02:31:53.927115 2579 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 04 02:31:53.948838 ip-10-0-134-125 kubenswrapper[2579]: W1104 02:31:53.927119 2579 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 04 02:31:53.948838 ip-10-0-134-125 kubenswrapper[2579]: W1104 02:31:53.927125 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 04 02:31:53.948838 ip-10-0-134-125 kubenswrapper[2579]: W1104 02:31:53.927129 2579 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 04 02:31:53.948838 ip-10-0-134-125 kubenswrapper[2579]: W1104 02:31:53.927133 2579 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 04 02:31:53.948838 ip-10-0-134-125 kubenswrapper[2579]: W1104 02:31:53.927137 2579 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 04 02:31:53.948838 ip-10-0-134-125 kubenswrapper[2579]: W1104 02:31:53.927143 2579 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 04 02:31:53.948838 ip-10-0-134-125 kubenswrapper[2579]: I1104 02:31:53.940464 2579 server.go:525] "Kubelet version" kubeletVersion="v1.35.9" Nov 04 02:31:53.948838 ip-10-0-134-125 kubenswrapper[2579]: I1104 02:31:53.940485 2579 server.go:527] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Nov 04 02:31:53.948838 ip-10-0-134-125 kubenswrapper[2579]: I1104 02:31:53.940503 2579 watchdog_linux.go:95] "Systemd watchdog is not enabled" Nov 04 02:31:53.948838 ip-10-0-134-125 kubenswrapper[2579]: I1104 02:31:53.940508 2579 watchdog_linux.go:138] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Nov 04 02:31:53.948838 ip-10-0-134-125 kubenswrapper[2579]: W1104 02:31:53.940568 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 04 02:31:53.948838 ip-10-0-134-125 kubenswrapper[2579]: W1104 02:31:53.940573 2579 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 04 02:31:53.948838 ip-10-0-134-125 kubenswrapper[2579]: W1104 02:31:53.940576 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 04 02:31:53.948838 ip-10-0-134-125 kubenswrapper[2579]: W1104 02:31:53.940579 2579 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 04 02:31:53.948838 ip-10-0-134-125 kubenswrapper[2579]: W1104 02:31:53.940581 2579 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 04 02:31:53.948838 ip-10-0-134-125 kubenswrapper[2579]: W1104 02:31:53.940584 2579 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 04 02:31:53.948838 ip-10-0-134-125 kubenswrapper[2579]: W1104 02:31:53.940587 2579 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 04 02:31:53.949377 ip-10-0-134-125 kubenswrapper[2579]: W1104 02:31:53.940590 2579 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 04 02:31:53.949377 ip-10-0-134-125 kubenswrapper[2579]: W1104 02:31:53.940592 2579 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 04 02:31:53.949377 ip-10-0-134-125 kubenswrapper[2579]: W1104 02:31:53.940595 2579 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 04 02:31:53.949377 ip-10-0-134-125 kubenswrapper[2579]: W1104 02:31:53.940598 2579 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 04 02:31:53.949377 ip-10-0-134-125 kubenswrapper[2579]: W1104 02:31:53.940600 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 04 02:31:53.949377 ip-10-0-134-125 kubenswrapper[2579]: W1104 02:31:53.940603 2579 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 04 02:31:53.949377 ip-10-0-134-125 kubenswrapper[2579]: W1104 02:31:53.940606 2579 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 04 02:31:53.949377 ip-10-0-134-125 kubenswrapper[2579]: W1104 02:31:53.940608 2579 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 04 02:31:53.949377 ip-10-0-134-125 kubenswrapper[2579]: W1104 02:31:53.940611 2579 feature_gate.go:386] unrecognized feature gate: Example2 Nov 04 02:31:53.949377 ip-10-0-134-125 kubenswrapper[2579]: W1104 02:31:53.940614 2579 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 04 02:31:53.949377 ip-10-0-134-125 kubenswrapper[2579]: W1104 02:31:53.940616 2579 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 04 02:31:53.949377 ip-10-0-134-125 kubenswrapper[2579]: W1104 02:31:53.940619 2579 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 04 02:31:53.949377 ip-10-0-134-125 kubenswrapper[2579]: W1104 02:31:53.940621 2579 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 04 02:31:53.949377 ip-10-0-134-125 kubenswrapper[2579]: W1104 02:31:53.940624 2579 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 04 02:31:53.949377 ip-10-0-134-125 kubenswrapper[2579]: W1104 02:31:53.940626 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 04 02:31:53.949377 ip-10-0-134-125 kubenswrapper[2579]: W1104 02:31:53.940629 2579 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 04 02:31:53.949377 ip-10-0-134-125 kubenswrapper[2579]: W1104 02:31:53.940631 2579 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 04 02:31:53.949377 ip-10-0-134-125 kubenswrapper[2579]: W1104 02:31:53.940634 2579 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 04 02:31:53.949377 ip-10-0-134-125 kubenswrapper[2579]: W1104 02:31:53.940637 2579 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 04 02:31:53.949377 ip-10-0-134-125 kubenswrapper[2579]: W1104 02:31:53.940639 2579 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 04 02:31:53.949954 ip-10-0-134-125 kubenswrapper[2579]: W1104 02:31:53.940642 2579 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 04 02:31:53.949954 ip-10-0-134-125 kubenswrapper[2579]: W1104 02:31:53.940645 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 04 02:31:53.949954 ip-10-0-134-125 kubenswrapper[2579]: W1104 02:31:53.940649 2579 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 04 02:31:53.949954 ip-10-0-134-125 kubenswrapper[2579]: W1104 02:31:53.940651 2579 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 04 02:31:53.949954 ip-10-0-134-125 kubenswrapper[2579]: W1104 02:31:53.940654 2579 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 04 02:31:53.949954 ip-10-0-134-125 kubenswrapper[2579]: W1104 02:31:53.940657 2579 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 04 02:31:53.949954 ip-10-0-134-125 kubenswrapper[2579]: W1104 02:31:53.940659 2579 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 04 02:31:53.949954 ip-10-0-134-125 kubenswrapper[2579]: W1104 02:31:53.940677 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 04 02:31:53.949954 ip-10-0-134-125 kubenswrapper[2579]: W1104 02:31:53.940680 2579 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 04 02:31:53.949954 ip-10-0-134-125 kubenswrapper[2579]: W1104 02:31:53.940682 2579 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 04 02:31:53.949954 ip-10-0-134-125 kubenswrapper[2579]: W1104 02:31:53.940685 2579 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 04 02:31:53.949954 ip-10-0-134-125 kubenswrapper[2579]: W1104 02:31:53.940688 2579 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 04 02:31:53.949954 ip-10-0-134-125 kubenswrapper[2579]: W1104 02:31:53.940690 2579 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 04 02:31:53.949954 ip-10-0-134-125 kubenswrapper[2579]: W1104 02:31:53.940693 2579 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 04 02:31:53.949954 ip-10-0-134-125 kubenswrapper[2579]: W1104 02:31:53.940696 2579 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 04 02:31:53.949954 ip-10-0-134-125 kubenswrapper[2579]: W1104 02:31:53.940698 2579 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 04 02:31:53.949954 ip-10-0-134-125 kubenswrapper[2579]: W1104 02:31:53.940701 2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 04 02:31:53.949954 ip-10-0-134-125 kubenswrapper[2579]: W1104 02:31:53.940704 2579 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 04 02:31:53.949954 ip-10-0-134-125 kubenswrapper[2579]: W1104 02:31:53.940707 2579 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 04 02:31:53.950455 ip-10-0-134-125 kubenswrapper[2579]: W1104 02:31:53.940710 2579 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 04 02:31:53.950455 ip-10-0-134-125 kubenswrapper[2579]: W1104 02:31:53.940712 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 04 02:31:53.950455 ip-10-0-134-125 kubenswrapper[2579]: W1104 02:31:53.940715 2579 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 04 02:31:53.950455 ip-10-0-134-125 kubenswrapper[2579]: W1104 02:31:53.940718 2579 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 04 02:31:53.950455 ip-10-0-134-125 kubenswrapper[2579]: W1104 02:31:53.940720 2579 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 04 02:31:53.950455 ip-10-0-134-125 kubenswrapper[2579]: W1104 02:31:53.940723 2579 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 04 02:31:53.950455 ip-10-0-134-125 kubenswrapper[2579]: W1104 02:31:53.940726 2579 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 04 02:31:53.950455 ip-10-0-134-125 kubenswrapper[2579]: W1104 02:31:53.940729 2579 feature_gate.go:386] unrecognized feature gate: Example Nov 04 02:31:53.950455 ip-10-0-134-125 kubenswrapper[2579]: W1104 02:31:53.940732 2579 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 04 02:31:53.950455 ip-10-0-134-125 kubenswrapper[2579]: W1104 02:31:53.940735 2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 04 02:31:53.950455 ip-10-0-134-125 kubenswrapper[2579]: W1104 02:31:53.940737 2579 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 04 02:31:53.950455 ip-10-0-134-125 kubenswrapper[2579]: W1104 02:31:53.940740 2579 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 04 02:31:53.950455 ip-10-0-134-125 kubenswrapper[2579]: W1104 02:31:53.940742 2579 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 04 02:31:53.950455 ip-10-0-134-125 kubenswrapper[2579]: W1104 02:31:53.940745 2579 feature_gate.go:386] unrecognized feature gate: EVPN Nov 04 02:31:53.950455 ip-10-0-134-125 kubenswrapper[2579]: W1104 02:31:53.940748 2579 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 04 02:31:53.950455 ip-10-0-134-125 kubenswrapper[2579]: W1104 02:31:53.940750 2579 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 04 02:31:53.950455 ip-10-0-134-125 kubenswrapper[2579]: W1104 02:31:53.940753 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 04 02:31:53.950455 ip-10-0-134-125 kubenswrapper[2579]: W1104 02:31:53.940756 2579 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 04 02:31:53.950455 ip-10-0-134-125 kubenswrapper[2579]: W1104 02:31:53.940759 2579 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 04 02:31:53.950455 ip-10-0-134-125 kubenswrapper[2579]: W1104 02:31:53.940764 2579 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 04 02:31:53.951000 ip-10-0-134-125 kubenswrapper[2579]: W1104 02:31:53.940768 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 04 02:31:53.951000 ip-10-0-134-125 kubenswrapper[2579]: W1104 02:31:53.940771 2579 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 04 02:31:53.951000 ip-10-0-134-125 kubenswrapper[2579]: W1104 02:31:53.940774 2579 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 04 02:31:53.951000 ip-10-0-134-125 kubenswrapper[2579]: W1104 02:31:53.940777 2579 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 04 02:31:53.951000 ip-10-0-134-125 kubenswrapper[2579]: W1104 02:31:53.940779 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 04 02:31:53.951000 ip-10-0-134-125 kubenswrapper[2579]: W1104 02:31:53.940782 2579 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 04 02:31:53.951000 ip-10-0-134-125 kubenswrapper[2579]: W1104 02:31:53.940784 2579 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 04 02:31:53.951000 ip-10-0-134-125 kubenswrapper[2579]: W1104 02:31:53.940787 2579 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 04 02:31:53.951000 ip-10-0-134-125 kubenswrapper[2579]: W1104 02:31:53.940790 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 04 02:31:53.951000 ip-10-0-134-125 kubenswrapper[2579]: W1104 02:31:53.940792 2579 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 04 02:31:53.951000 ip-10-0-134-125 kubenswrapper[2579]: W1104 02:31:53.940795 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 04 02:31:53.951000 ip-10-0-134-125 kubenswrapper[2579]: W1104 02:31:53.940798 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 04 02:31:53.951000 ip-10-0-134-125 kubenswrapper[2579]: W1104 02:31:53.940801 2579 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 04 02:31:53.951000 ip-10-0-134-125 kubenswrapper[2579]: W1104 02:31:53.940803 2579 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 04 02:31:53.951000 ip-10-0-134-125 kubenswrapper[2579]: W1104 02:31:53.940806 2579 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 04 02:31:53.951000 ip-10-0-134-125 kubenswrapper[2579]: W1104 02:31:53.940808 2579 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 04 02:31:53.951000 ip-10-0-134-125 kubenswrapper[2579]: W1104 02:31:53.940811 2579 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 04 02:31:53.951000 ip-10-0-134-125 kubenswrapper[2579]: W1104 02:31:53.940813 2579 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 04 02:31:53.951000 ip-10-0-134-125 kubenswrapper[2579]: W1104 02:31:53.940816 2579 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 04 02:31:53.952006 ip-10-0-134-125 kubenswrapper[2579]: W1104 02:31:53.940819 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 04 02:31:53.952006 ip-10-0-134-125 kubenswrapper[2579]: W1104 02:31:53.940822 2579 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 04 02:31:53.952006 ip-10-0-134-125 kubenswrapper[2579]: W1104 02:31:53.940824 2579 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 04 02:31:53.952006 ip-10-0-134-125 kubenswrapper[2579]: W1104 02:31:53.940827 2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 04 02:31:53.952006 ip-10-0-134-125 kubenswrapper[2579]: W1104 02:31:53.940830 2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 04 02:31:53.952006 ip-10-0-134-125 kubenswrapper[2579]: W1104 02:31:53.940833 2579 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 04 02:31:53.952006 ip-10-0-134-125 kubenswrapper[2579]: W1104 02:31:53.940835 2579 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 04 02:31:53.952006 ip-10-0-134-125 kubenswrapper[2579]: W1104 02:31:53.940838 2579 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 04 02:31:53.952006 ip-10-0-134-125 kubenswrapper[2579]: W1104 02:31:53.940840 2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 04 02:31:53.952006 ip-10-0-134-125 kubenswrapper[2579]: W1104 02:31:53.940843 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 04 02:31:53.952006 ip-10-0-134-125 kubenswrapper[2579]: W1104 02:31:53.940846 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 04 02:31:53.952006 ip-10-0-134-125 kubenswrapper[2579]: W1104 02:31:53.940848 2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 04 02:31:53.952006 ip-10-0-134-125 kubenswrapper[2579]: W1104 02:31:53.940851 2579 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 04 02:31:53.952006 ip-10-0-134-125 kubenswrapper[2579]: W1104 02:31:53.940854 2579 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 04 02:31:53.952006 ip-10-0-134-125 kubenswrapper[2579]: W1104 02:31:53.940856 2579 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 04 02:31:53.952006 ip-10-0-134-125 kubenswrapper[2579]: W1104 02:31:53.940859 2579 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 04 02:31:53.952006 ip-10-0-134-125 kubenswrapper[2579]: W1104 02:31:53.940861 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 04 02:31:53.952006 ip-10-0-134-125 kubenswrapper[2579]: W1104 02:31:53.940863 2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 04 02:31:53.952006 ip-10-0-134-125 kubenswrapper[2579]: W1104 02:31:53.940868 2579 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 04 02:31:53.952496 ip-10-0-134-125 kubenswrapper[2579]: W1104 02:31:53.940871 2579 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 04 02:31:53.952496 ip-10-0-134-125 kubenswrapper[2579]: W1104 02:31:53.940874 2579 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 04 02:31:53.952496 ip-10-0-134-125 kubenswrapper[2579]: I1104 02:31:53.940906 2579 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Nov 04 02:31:53.952496 ip-10-0-134-125 kubenswrapper[2579]: W1104 02:31:53.941052 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 04 02:31:53.952496 ip-10-0-134-125 kubenswrapper[2579]: W1104 02:31:53.941057 2579 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 04 02:31:53.952496 ip-10-0-134-125 kubenswrapper[2579]: W1104 02:31:53.941061 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 04 02:31:53.952496 ip-10-0-134-125 kubenswrapper[2579]: W1104 02:31:53.941064 2579 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 04 02:31:53.952496 ip-10-0-134-125 kubenswrapper[2579]: W1104 02:31:53.941067 2579 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 04 02:31:53.952496 ip-10-0-134-125 kubenswrapper[2579]: W1104 02:31:53.941070 2579 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 04 02:31:53.952496 ip-10-0-134-125 kubenswrapper[2579]: W1104 02:31:53.941073 2579 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 04 02:31:53.952496 ip-10-0-134-125 kubenswrapper[2579]: W1104 02:31:53.941076 2579 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 04 02:31:53.952496 ip-10-0-134-125 kubenswrapper[2579]: W1104 02:31:53.941079 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 04 02:31:53.952496 ip-10-0-134-125 kubenswrapper[2579]: W1104 02:31:53.941081 2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 04 02:31:53.952496 ip-10-0-134-125 kubenswrapper[2579]: W1104 02:31:53.941084 2579 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 04 02:31:53.952496 ip-10-0-134-125 kubenswrapper[2579]: W1104 02:31:53.941087 2579 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 04 02:31:53.952496 ip-10-0-134-125 kubenswrapper[2579]: W1104 02:31:53.941089 2579 feature_gate.go:386] unrecognized feature gate: EVPN Nov 04 02:31:53.952927 ip-10-0-134-125 kubenswrapper[2579]: W1104 02:31:53.941092 2579 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 04 02:31:53.952927 ip-10-0-134-125 kubenswrapper[2579]: W1104 02:31:53.941095 2579 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 04 02:31:53.952927 ip-10-0-134-125 kubenswrapper[2579]: W1104 02:31:53.941097 2579 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 04 02:31:53.952927 ip-10-0-134-125 kubenswrapper[2579]: W1104 02:31:53.941100 2579 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 04 02:31:53.952927 ip-10-0-134-125 kubenswrapper[2579]: W1104 02:31:53.941103 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 04 02:31:53.952927 ip-10-0-134-125 kubenswrapper[2579]: W1104 02:31:53.941105 2579 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 04 02:31:53.952927 ip-10-0-134-125 kubenswrapper[2579]: W1104 02:31:53.941108 2579 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 04 02:31:53.952927 ip-10-0-134-125 kubenswrapper[2579]: W1104 02:31:53.941110 2579 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 04 02:31:53.952927 ip-10-0-134-125 kubenswrapper[2579]: W1104 02:31:53.941113 2579 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 04 02:31:53.952927 ip-10-0-134-125 kubenswrapper[2579]: W1104 02:31:53.941116 2579 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 04 02:31:53.952927 ip-10-0-134-125 kubenswrapper[2579]: W1104 02:31:53.941119 2579 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 04 02:31:53.952927 ip-10-0-134-125 kubenswrapper[2579]: W1104 02:31:53.941121 2579 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 04 02:31:53.952927 ip-10-0-134-125 kubenswrapper[2579]: W1104 02:31:53.941123 2579 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 04 02:31:53.952927 ip-10-0-134-125 kubenswrapper[2579]: W1104 02:31:53.941126 2579 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 04 02:31:53.952927 ip-10-0-134-125 kubenswrapper[2579]: W1104 02:31:53.941128 2579 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 04 02:31:53.952927 ip-10-0-134-125 kubenswrapper[2579]: W1104 02:31:53.941131 2579 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 04 02:31:53.952927 ip-10-0-134-125 kubenswrapper[2579]: W1104 02:31:53.941133 2579 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 04 02:31:53.952927 ip-10-0-134-125 kubenswrapper[2579]: W1104 02:31:53.941136 2579 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 04 02:31:53.952927 ip-10-0-134-125 kubenswrapper[2579]: W1104 02:31:53.941140 2579 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 04 02:31:53.952927 ip-10-0-134-125 kubenswrapper[2579]: W1104 02:31:53.941143 2579 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 04 02:31:53.953423 ip-10-0-134-125 kubenswrapper[2579]: W1104 02:31:53.941146 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 04 02:31:53.953423 ip-10-0-134-125 kubenswrapper[2579]: W1104 02:31:53.941149 2579 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 04 02:31:53.953423 ip-10-0-134-125 kubenswrapper[2579]: W1104 02:31:53.941152 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 04 02:31:53.953423 ip-10-0-134-125 kubenswrapper[2579]: W1104 02:31:53.941155 2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 04 02:31:53.953423 ip-10-0-134-125 kubenswrapper[2579]: W1104 02:31:53.941157 2579 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 04 02:31:53.953423 ip-10-0-134-125 kubenswrapper[2579]: W1104 02:31:53.941160 2579 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 04 02:31:53.953423 ip-10-0-134-125 kubenswrapper[2579]: W1104 02:31:53.941163 2579 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 04 02:31:53.953423 ip-10-0-134-125 kubenswrapper[2579]: W1104 02:31:53.941166 2579 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 04 02:31:53.953423 ip-10-0-134-125 kubenswrapper[2579]: W1104 02:31:53.941168 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 04 02:31:53.953423 ip-10-0-134-125 kubenswrapper[2579]: W1104 02:31:53.941171 2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 04 02:31:53.953423 ip-10-0-134-125 kubenswrapper[2579]: W1104 02:31:53.941173 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 04 02:31:53.953423 ip-10-0-134-125 kubenswrapper[2579]: W1104 02:31:53.941176 2579 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 04 02:31:53.953423 ip-10-0-134-125 kubenswrapper[2579]: W1104 02:31:53.941179 2579 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 04 02:31:53.953423 ip-10-0-134-125 kubenswrapper[2579]: W1104 02:31:53.941181 2579 feature_gate.go:386] unrecognized feature gate: Example2 Nov 04 02:31:53.953423 ip-10-0-134-125 kubenswrapper[2579]: W1104 02:31:53.941184 2579 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 04 02:31:53.953423 ip-10-0-134-125 kubenswrapper[2579]: W1104 02:31:53.941187 2579 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 04 02:31:53.953423 ip-10-0-134-125 kubenswrapper[2579]: W1104 02:31:53.941190 2579 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 04 02:31:53.953423 ip-10-0-134-125 kubenswrapper[2579]: W1104 02:31:53.941192 2579 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 04 02:31:53.953423 ip-10-0-134-125 kubenswrapper[2579]: W1104 02:31:53.941195 2579 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 04 02:31:53.953980 ip-10-0-134-125 kubenswrapper[2579]: W1104 02:31:53.941197 2579 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 04 02:31:53.953980 ip-10-0-134-125 kubenswrapper[2579]: W1104 02:31:53.941200 2579 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 04 02:31:53.953980 ip-10-0-134-125 kubenswrapper[2579]: W1104 02:31:53.941203 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 04 02:31:53.953980 ip-10-0-134-125 kubenswrapper[2579]: W1104 02:31:53.941205 2579 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 04 02:31:53.953980 ip-10-0-134-125 kubenswrapper[2579]: W1104 02:31:53.941208 2579 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 04 02:31:53.953980 ip-10-0-134-125 kubenswrapper[2579]: W1104 02:31:53.941210 2579 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 04 02:31:53.953980 ip-10-0-134-125 kubenswrapper[2579]: W1104 02:31:53.941213 2579 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 04 02:31:53.953980 ip-10-0-134-125 kubenswrapper[2579]: W1104 02:31:53.941216 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 04 02:31:53.953980 ip-10-0-134-125 kubenswrapper[2579]: W1104 02:31:53.941218 2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 04 02:31:53.953980 ip-10-0-134-125 kubenswrapper[2579]: W1104 02:31:53.941221 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 04 02:31:53.953980 ip-10-0-134-125 kubenswrapper[2579]: W1104 02:31:53.941223 2579 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 04 02:31:53.953980 ip-10-0-134-125 kubenswrapper[2579]: W1104 02:31:53.941226 2579 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 04 02:31:53.953980 ip-10-0-134-125 kubenswrapper[2579]: W1104 02:31:53.941228 2579 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 04 02:31:53.953980 ip-10-0-134-125 kubenswrapper[2579]: W1104 02:31:53.941231 2579 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 04 02:31:53.953980 ip-10-0-134-125 kubenswrapper[2579]: W1104 02:31:53.941233 2579 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 04 02:31:53.953980 ip-10-0-134-125 kubenswrapper[2579]: W1104 02:31:53.941236 2579 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 04 02:31:53.953980 ip-10-0-134-125 kubenswrapper[2579]: W1104 02:31:53.941239 2579 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 04 02:31:53.953980 ip-10-0-134-125 kubenswrapper[2579]: W1104 02:31:53.941242 2579 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 04 02:31:53.953980 ip-10-0-134-125 kubenswrapper[2579]: W1104 02:31:53.941246 2579 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 04 02:31:53.954472 ip-10-0-134-125 kubenswrapper[2579]: W1104 02:31:53.941249 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 04 02:31:53.954472 ip-10-0-134-125 kubenswrapper[2579]: W1104 02:31:53.941252 2579 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 04 02:31:53.954472 ip-10-0-134-125 kubenswrapper[2579]: W1104 02:31:53.941254 2579 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 04 02:31:53.954472 ip-10-0-134-125 kubenswrapper[2579]: W1104 02:31:53.941257 2579 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 04 02:31:53.954472 ip-10-0-134-125 kubenswrapper[2579]: W1104 02:31:53.941260 2579 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 04 02:31:53.954472 ip-10-0-134-125 kubenswrapper[2579]: W1104 02:31:53.941263 2579 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 04 02:31:53.954472 ip-10-0-134-125 kubenswrapper[2579]: W1104 02:31:53.941266 2579 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 04 02:31:53.954472 ip-10-0-134-125 kubenswrapper[2579]: W1104 02:31:53.941269 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 04 02:31:53.954472 ip-10-0-134-125 kubenswrapper[2579]: W1104 02:31:53.941271 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 04 02:31:53.954472 ip-10-0-134-125 kubenswrapper[2579]: W1104 02:31:53.941274 2579 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 04 02:31:53.954472 ip-10-0-134-125 kubenswrapper[2579]: W1104 02:31:53.941276 2579 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 04 02:31:53.954472 ip-10-0-134-125 kubenswrapper[2579]: W1104 02:31:53.941279 2579 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 04 02:31:53.954472 ip-10-0-134-125 kubenswrapper[2579]: W1104 02:31:53.941282 2579 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 04 02:31:53.954472 ip-10-0-134-125 kubenswrapper[2579]: W1104 02:31:53.941285 2579 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 04 02:31:53.954472 ip-10-0-134-125 kubenswrapper[2579]: W1104 02:31:53.941287 2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 04 02:31:53.954472 ip-10-0-134-125 kubenswrapper[2579]: W1104 02:31:53.941290 2579 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 04 02:31:53.954472 ip-10-0-134-125 kubenswrapper[2579]: W1104 02:31:53.941293 2579 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 04 02:31:53.954472 ip-10-0-134-125 kubenswrapper[2579]: W1104 02:31:53.941295 2579 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 04 02:31:53.954472 ip-10-0-134-125 kubenswrapper[2579]: W1104 02:31:53.941298 2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 04 02:31:53.954472 ip-10-0-134-125 kubenswrapper[2579]: W1104 02:31:53.941300 2579 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 04 02:31:53.955040 ip-10-0-134-125 kubenswrapper[2579]: W1104 02:31:53.941303 2579 feature_gate.go:386] unrecognized feature gate: Example Nov 04 02:31:53.955040 ip-10-0-134-125 kubenswrapper[2579]: W1104 02:31:53.941305 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 04 02:31:53.955040 ip-10-0-134-125 kubenswrapper[2579]: W1104 02:31:53.941308 2579 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 04 02:31:53.955040 ip-10-0-134-125 kubenswrapper[2579]: W1104 02:31:53.941310 2579 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 04 02:31:53.955040 ip-10-0-134-125 kubenswrapper[2579]: W1104 02:31:53.941313 2579 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 04 02:31:53.955040 ip-10-0-134-125 kubenswrapper[2579]: W1104 02:31:53.941315 2579 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 04 02:31:53.955040 ip-10-0-134-125 kubenswrapper[2579]: W1104 02:31:53.941318 2579 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 04 02:31:53.955040 ip-10-0-134-125 kubenswrapper[2579]: W1104 02:31:53.941320 2579 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 04 02:31:53.955040 ip-10-0-134-125 kubenswrapper[2579]: W1104 02:31:53.941323 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 04 02:31:53.955040 ip-10-0-134-125 kubenswrapper[2579]: W1104 02:31:53.941327 2579 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 04 02:31:53.955040 ip-10-0-134-125 kubenswrapper[2579]: W1104 02:31:53.941329 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 04 02:31:53.955040 ip-10-0-134-125 kubenswrapper[2579]: W1104 02:31:53.941332 2579 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 04 02:31:53.955040 ip-10-0-134-125 kubenswrapper[2579]: W1104 02:31:53.941338 2579 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 04 02:31:53.955040 ip-10-0-134-125 kubenswrapper[2579]: W1104 02:31:53.941340 2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 04 02:31:53.955040 ip-10-0-134-125 kubenswrapper[2579]: W1104 02:31:53.941343 2579 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 04 02:31:53.955040 ip-10-0-134-125 kubenswrapper[2579]: I1104 02:31:53.941366 2579 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Nov 04 02:31:53.955444 ip-10-0-134-125 kubenswrapper[2579]: W1104 02:31:53.941398 2579 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 04 02:31:53.955444 ip-10-0-134-125 kubenswrapper[2579]: W1104 02:31:53.941402 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 04 02:31:53.955444 ip-10-0-134-125 kubenswrapper[2579]: W1104 02:31:53.941405 2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 04 02:31:53.955444 ip-10-0-134-125 kubenswrapper[2579]: W1104 02:31:53.941408 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 04 02:31:53.955444 ip-10-0-134-125 kubenswrapper[2579]: W1104 02:31:53.941411 2579 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 04 02:31:53.955444 ip-10-0-134-125 kubenswrapper[2579]: W1104 02:31:53.941414 2579 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 04 02:31:53.955444 ip-10-0-134-125 kubenswrapper[2579]: W1104 02:31:53.941416 2579 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 04 02:31:53.955444 ip-10-0-134-125 kubenswrapper[2579]: W1104 02:31:53.941419 2579 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 04 02:31:53.955444 ip-10-0-134-125 kubenswrapper[2579]: W1104 02:31:53.941421 2579 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 04 02:31:53.955444 ip-10-0-134-125 kubenswrapper[2579]: W1104 02:31:53.941424 2579 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 04 02:31:53.955444 ip-10-0-134-125 kubenswrapper[2579]: W1104 02:31:53.941426 2579 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 04 02:31:53.955444 ip-10-0-134-125 kubenswrapper[2579]: W1104 02:31:53.941430 2579 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 04 02:31:53.955444 ip-10-0-134-125 kubenswrapper[2579]: W1104 02:31:53.941434 2579 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 04 02:31:53.955444 ip-10-0-134-125 kubenswrapper[2579]: W1104 02:31:53.941437 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 04 02:31:53.955444 ip-10-0-134-125 kubenswrapper[2579]: W1104 02:31:53.941440 2579 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 04 02:31:53.955444 ip-10-0-134-125 kubenswrapper[2579]: W1104 02:31:53.941443 2579 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 04 02:31:53.955444 ip-10-0-134-125 kubenswrapper[2579]: W1104 02:31:53.941446 2579 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 04 02:31:53.955444 ip-10-0-134-125 kubenswrapper[2579]: W1104 02:31:53.941448 2579 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 04 02:31:53.955444 ip-10-0-134-125 kubenswrapper[2579]: W1104 02:31:53.941451 2579 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 04 02:31:53.955444 ip-10-0-134-125 kubenswrapper[2579]: W1104 02:31:53.941453 2579 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 04 02:31:53.955974 ip-10-0-134-125 kubenswrapper[2579]: W1104 02:31:53.941456 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 04 02:31:53.955974 ip-10-0-134-125 kubenswrapper[2579]: W1104 02:31:53.941458 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 04 02:31:53.955974 ip-10-0-134-125 kubenswrapper[2579]: W1104 02:31:53.941461 2579 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 04 02:31:53.955974 ip-10-0-134-125 kubenswrapper[2579]: W1104 02:31:53.941464 2579 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 04 02:31:53.955974 ip-10-0-134-125 kubenswrapper[2579]: W1104 02:31:53.941467 2579 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 04 02:31:53.955974 ip-10-0-134-125 kubenswrapper[2579]: W1104 02:31:53.941469 2579 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 04 02:31:53.955974 ip-10-0-134-125 kubenswrapper[2579]: W1104 02:31:53.941472 2579 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 04 02:31:53.955974 ip-10-0-134-125 kubenswrapper[2579]: W1104 02:31:53.941475 2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 04 02:31:53.955974 ip-10-0-134-125 kubenswrapper[2579]: W1104 02:31:53.941477 2579 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 04 02:31:53.955974 ip-10-0-134-125 kubenswrapper[2579]: W1104 02:31:53.941481 2579 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 04 02:31:53.955974 ip-10-0-134-125 kubenswrapper[2579]: W1104 02:31:53.941484 2579 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 04 02:31:53.955974 ip-10-0-134-125 kubenswrapper[2579]: W1104 02:31:53.941486 2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 04 02:31:53.955974 ip-10-0-134-125 kubenswrapper[2579]: W1104 02:31:53.941489 2579 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 04 02:31:53.955974 ip-10-0-134-125 kubenswrapper[2579]: W1104 02:31:53.941492 2579 feature_gate.go:386] unrecognized feature gate: Example Nov 04 02:31:53.955974 ip-10-0-134-125 kubenswrapper[2579]: W1104 02:31:53.941494 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 04 02:31:53.955974 ip-10-0-134-125 kubenswrapper[2579]: W1104 02:31:53.941497 2579 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 04 02:31:53.955974 ip-10-0-134-125 kubenswrapper[2579]: W1104 02:31:53.941499 2579 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 04 02:31:53.955974 ip-10-0-134-125 kubenswrapper[2579]: W1104 02:31:53.941536 2579 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 04 02:31:53.955974 ip-10-0-134-125 kubenswrapper[2579]: W1104 02:31:53.941541 2579 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 04 02:31:53.956488 ip-10-0-134-125 kubenswrapper[2579]: W1104 02:31:53.941543 2579 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 04 02:31:53.956488 ip-10-0-134-125 kubenswrapper[2579]: W1104 02:31:53.941546 2579 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 04 02:31:53.956488 ip-10-0-134-125 kubenswrapper[2579]: W1104 02:31:53.941551 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 04 02:31:53.956488 ip-10-0-134-125 kubenswrapper[2579]: W1104 02:31:53.941554 2579 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 04 02:31:53.956488 ip-10-0-134-125 kubenswrapper[2579]: W1104 02:31:53.941557 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 04 02:31:53.956488 ip-10-0-134-125 kubenswrapper[2579]: W1104 02:31:53.941559 2579 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 04 02:31:53.956488 ip-10-0-134-125 kubenswrapper[2579]: W1104 02:31:53.941562 2579 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 04 02:31:53.956488 ip-10-0-134-125 kubenswrapper[2579]: W1104 02:31:53.941565 2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 04 02:31:53.956488 ip-10-0-134-125 kubenswrapper[2579]: W1104 02:31:53.941567 2579 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 04 02:31:53.956488 ip-10-0-134-125 kubenswrapper[2579]: W1104 02:31:53.941570 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 04 02:31:53.956488 ip-10-0-134-125 kubenswrapper[2579]: W1104 02:31:53.941573 2579 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 04 02:31:53.956488 ip-10-0-134-125 kubenswrapper[2579]: W1104 02:31:53.941575 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 04 02:31:53.956488 ip-10-0-134-125 kubenswrapper[2579]: W1104 02:31:53.941578 2579 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 04 02:31:53.956488 ip-10-0-134-125 kubenswrapper[2579]: W1104 02:31:53.941580 2579 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 04 02:31:53.956488 ip-10-0-134-125 kubenswrapper[2579]: W1104 02:31:53.941583 2579 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 04 02:31:53.956488 ip-10-0-134-125 kubenswrapper[2579]: W1104 02:31:53.941586 2579 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 04 02:31:53.956488 ip-10-0-134-125 kubenswrapper[2579]: W1104 02:31:53.941588 2579 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 04 02:31:53.956488 ip-10-0-134-125 kubenswrapper[2579]: W1104 02:31:53.941591 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 04 02:31:53.956488 ip-10-0-134-125 kubenswrapper[2579]: W1104 02:31:53.941593 2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 04 02:31:53.957112 ip-10-0-134-125 kubenswrapper[2579]: W1104 02:31:53.941596 2579 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 04 02:31:53.957112 ip-10-0-134-125 kubenswrapper[2579]: W1104 02:31:53.941599 2579 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 04 02:31:53.957112 ip-10-0-134-125 kubenswrapper[2579]: W1104 02:31:53.941601 2579 feature_gate.go:386] unrecognized feature gate: EVPN Nov 04 02:31:53.957112 ip-10-0-134-125 kubenswrapper[2579]: W1104 02:31:53.941605 2579 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 04 02:31:53.957112 ip-10-0-134-125 kubenswrapper[2579]: W1104 02:31:53.941608 2579 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 04 02:31:53.957112 ip-10-0-134-125 kubenswrapper[2579]: W1104 02:31:53.941610 2579 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 04 02:31:53.957112 ip-10-0-134-125 kubenswrapper[2579]: W1104 02:31:53.941613 2579 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 04 02:31:53.957112 ip-10-0-134-125 kubenswrapper[2579]: W1104 02:31:53.941616 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 04 02:31:53.957112 ip-10-0-134-125 kubenswrapper[2579]: W1104 02:31:53.941618 2579 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 04 02:31:53.957112 ip-10-0-134-125 kubenswrapper[2579]: W1104 02:31:53.941621 2579 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 04 02:31:53.957112 ip-10-0-134-125 kubenswrapper[2579]: W1104 02:31:53.941623 2579 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 04 02:31:53.957112 ip-10-0-134-125 kubenswrapper[2579]: W1104 02:31:53.941626 2579 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 04 02:31:53.957112 ip-10-0-134-125 kubenswrapper[2579]: W1104 02:31:53.941629 2579 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 04 02:31:53.957112 ip-10-0-134-125 kubenswrapper[2579]: W1104 02:31:53.941631 2579 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 04 02:31:53.957112 ip-10-0-134-125 kubenswrapper[2579]: W1104 02:31:53.941634 2579 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 04 02:31:53.957112 ip-10-0-134-125 kubenswrapper[2579]: W1104 02:31:53.941636 2579 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 04 02:31:53.957112 ip-10-0-134-125 kubenswrapper[2579]: W1104 02:31:53.941639 2579 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 04 02:31:53.957112 ip-10-0-134-125 kubenswrapper[2579]: W1104 02:31:53.941641 2579 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 04 02:31:53.957112 ip-10-0-134-125 kubenswrapper[2579]: W1104 02:31:53.941644 2579 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 04 02:31:53.957112 ip-10-0-134-125 kubenswrapper[2579]: W1104 02:31:53.941647 2579 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 04 02:31:53.957686 ip-10-0-134-125 kubenswrapper[2579]: W1104 02:31:53.941649 2579 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 04 02:31:53.957686 ip-10-0-134-125 kubenswrapper[2579]: W1104 02:31:53.941654 2579 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 04 02:31:53.957686 ip-10-0-134-125 kubenswrapper[2579]: W1104 02:31:53.941658 2579 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 04 02:31:53.957686 ip-10-0-134-125 kubenswrapper[2579]: W1104 02:31:53.941661 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 04 02:31:53.957686 ip-10-0-134-125 kubenswrapper[2579]: W1104 02:31:53.941679 2579 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 04 02:31:53.957686 ip-10-0-134-125 kubenswrapper[2579]: W1104 02:31:53.941682 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 04 02:31:53.957686 ip-10-0-134-125 kubenswrapper[2579]: W1104 02:31:53.941685 2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 04 02:31:53.957686 ip-10-0-134-125 kubenswrapper[2579]: W1104 02:31:53.941687 2579 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 04 02:31:53.957686 ip-10-0-134-125 kubenswrapper[2579]: W1104 02:31:53.941690 2579 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 04 02:31:53.957686 ip-10-0-134-125 kubenswrapper[2579]: W1104 02:31:53.941693 2579 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 04 02:31:53.957686 ip-10-0-134-125 kubenswrapper[2579]: W1104 02:31:53.941696 2579 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 04 02:31:53.957686 ip-10-0-134-125 kubenswrapper[2579]: W1104 02:31:53.941699 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 04 02:31:53.957686 ip-10-0-134-125 kubenswrapper[2579]: W1104 02:31:53.941701 2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 04 02:31:53.957686 ip-10-0-134-125 kubenswrapper[2579]: W1104 02:31:53.941704 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 04 02:31:53.957686 ip-10-0-134-125 kubenswrapper[2579]: W1104 02:31:53.941707 2579 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 04 02:31:53.957686 ip-10-0-134-125 kubenswrapper[2579]: W1104 02:31:53.941710 2579 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 04 02:31:53.957686 ip-10-0-134-125 kubenswrapper[2579]: W1104 02:31:53.941713 2579 feature_gate.go:386] unrecognized feature gate: Example2 Nov 04 02:31:53.957686 ip-10-0-134-125 kubenswrapper[2579]: W1104 02:31:53.941716 2579 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 04 02:31:53.957686 ip-10-0-134-125 kubenswrapper[2579]: W1104 02:31:53.941720 2579 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 04 02:31:53.958179 ip-10-0-134-125 kubenswrapper[2579]: W1104 02:31:53.941722 2579 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 04 02:31:53.958179 ip-10-0-134-125 kubenswrapper[2579]: W1104 02:31:53.941725 2579 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 04 02:31:53.958179 ip-10-0-134-125 kubenswrapper[2579]: W1104 02:31:53.941727 2579 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 04 02:31:53.958179 ip-10-0-134-125 kubenswrapper[2579]: W1104 02:31:53.941730 2579 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 04 02:31:53.958179 ip-10-0-134-125 kubenswrapper[2579]: W1104 02:31:53.941732 2579 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 04 02:31:53.958179 ip-10-0-134-125 kubenswrapper[2579]: W1104 02:31:53.941735 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 04 02:31:53.958179 ip-10-0-134-125 kubenswrapper[2579]: W1104 02:31:53.941737 2579 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 04 02:31:53.958179 ip-10-0-134-125 kubenswrapper[2579]: W1104 02:31:53.941740 2579 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 04 02:31:53.958179 ip-10-0-134-125 kubenswrapper[2579]: W1104 02:31:53.941743 2579 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 04 02:31:53.958179 ip-10-0-134-125 kubenswrapper[2579]: I1104 02:31:53.941803 2579 server.go:957] "Client rotation is on, will bootstrap in background" Nov 04 02:31:53.958179 ip-10-0-134-125 kubenswrapper[2579]: I1104 02:31:53.944785 2579 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir" Nov 04 02:31:53.958179 ip-10-0-134-125 kubenswrapper[2579]: I1104 02:31:53.945686 2579 server.go:1014] "Starting client certificate rotation" Nov 04 02:31:53.958179 ip-10-0-134-125 kubenswrapper[2579]: I1104 02:31:53.945779 2579 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Nov 04 02:31:53.958179 ip-10-0-134-125 kubenswrapper[2579]: I1104 02:31:53.946280 2579 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet" Nov 04 02:31:53.958179 ip-10-0-134-125 kubenswrapper[2579]: I1104 02:31:53.947790 2579 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Nov 04 02:31:53.958179 ip-10-0-134-125 kubenswrapper[2579]: I1104 02:31:53.949007 2579 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Nov 04 02:31:53.959726 ip-10-0-134-125 kubenswrapper[2579]: I1104 02:31:53.959711 2579 log.go:25] "Validated CRI v1 runtime API" Nov 04 02:31:53.963589 ip-10-0-134-125 kubenswrapper[2579]: I1104 02:31:53.963573 2579 log.go:25] "Validated CRI v1 image API" Nov 04 02:31:53.965814 ip-10-0-134-125 kubenswrapper[2579]: I1104 02:31:53.965785 2579 server.go:1424] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Nov 04 02:31:53.968274 ip-10-0-134-125 kubenswrapper[2579]: I1104 02:31:53.968254 2579 fs.go:135] Filesystem UUIDs: map[19404a3c-de09-4630-be9f-0cbcbcb7c656:/dev/nvme0n1p4 2a6e893d-37f1-444b-9f75-98f43a4d48bf:/dev/nvme0n1p3 7B77-95E7:/dev/nvme0n1p2] Nov 04 02:31:53.968319 ip-10-0-134-125 kubenswrapper[2579]: I1104 02:31:53.968275 2579 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:24 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:26 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:32 fsType:tmpfs blockSize:0} composefs_0-34:{mountpoint:/ major:0 minor:34 fsType:overlay blockSize:0}] Nov 04 02:31:53.974063 ip-10-0-134-125 kubenswrapper[2579]: I1104 02:31:53.973949 2579 manager.go:217] Machine: {Timestamp:2026-11-04 02:31:53.972928267 +0000 UTC m=+0.328035361 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3107795 MemoryCapacity:32812011520 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2833fcb94c4b503126b825579d6ddb SystemUUID:ec2833fc-b94c-4b50-3126-b825579d6ddb BootID:2cd229f4-4cf0-4425-8796-f243d5685184 Filesystems:[{Device:/dev/shm DeviceMajor:0 DeviceMinor:24 Capacity:16406003712 Type:vfs Inodes:4005372 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:26 Capacity:6562402304 Type:vfs Inodes:819200 HasInodes:true} {Device:composefs_0-34 DeviceMajor:0 DeviceMinor:34 Capacity:6422528 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:32 Capacity:16406007808 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:2c:40:68:17:55 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:2c:40:68:17:55 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:c2:d9:78:6b:88:6d Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:32812011520 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:34603008 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Nov 04 02:31:53.974465 ip-10-0-134-125 kubenswrapper[2579]: I1104 02:31:53.974454 2579 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Nov 04 02:31:53.974596 ip-10-0-134-125 kubenswrapper[2579]: I1104 02:31:53.974585 2579 manager.go:233] Version: {KernelVersion:5.14.0-687.60.1.el9_8.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.8.20261023-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Nov 04 02:31:53.976636 ip-10-0-134-125 kubenswrapper[2579]: I1104 02:31:53.976608 2579 container_manager_linux.go:275] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Nov 04 02:31:53.976789 ip-10-0-134-125 kubenswrapper[2579]: I1104 02:31:53.976638 2579 container_manager_linux.go:280] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-134-125.us-west-1.compute.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"/system.slice","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{},"system-reserved-compressible":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Nov 04 02:31:53.976840 ip-10-0-134-125 kubenswrapper[2579]: I1104 02:31:53.976791 2579 topology_manager.go:143] "Creating topology manager with none policy" Nov 04 02:31:53.976840 ip-10-0-134-125 kubenswrapper[2579]: I1104 02:31:53.976804 2579 container_manager_linux.go:311] "Creating device plugin manager" Nov 04 02:31:53.976840 ip-10-0-134-125 kubenswrapper[2579]: I1104 02:31:53.976815 2579 manager.go:144] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Nov 04 02:31:53.977597 ip-10-0-134-125 kubenswrapper[2579]: I1104 02:31:53.977585 2579 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Nov 04 02:31:53.978230 ip-10-0-134-125 kubenswrapper[2579]: I1104 02:31:53.978219 2579 container_manager_linux.go:320] "Creating Dynamic Resource Allocation (DRA) manager" Nov 04 02:31:53.981834 ip-10-0-134-125 kubenswrapper[2579]: I1104 02:31:53.981821 2579 state_mem.go:41] "Initialized" logger="CPUManager state memory" Nov 04 02:31:53.981948 ip-10-0-134-125 kubenswrapper[2579]: I1104 02:31:53.981938 2579 server.go:1235] "Using root directory" path="/var/lib/kubelet" Nov 04 02:31:53.982736 ip-10-0-134-125 kubenswrapper[2579]: I1104 02:31:53.982726 2579 kubelet.go:493] "Attempting to sync node with API server" Nov 04 02:31:53.982770 ip-10-0-134-125 kubenswrapper[2579]: I1104 02:31:53.982740 2579 kubelet.go:394] "Adding static pod path" path="/etc/kubernetes/manifests" Nov 04 02:31:53.982770 ip-10-0-134-125 kubenswrapper[2579]: I1104 02:31:53.982751 2579 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Nov 04 02:31:53.982770 ip-10-0-134-125 kubenswrapper[2579]: I1104 02:31:53.982760 2579 kubelet.go:405] "Adding apiserver pod source" Nov 04 02:31:53.982770 ip-10-0-134-125 kubenswrapper[2579]: I1104 02:31:53.982768 2579 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Nov 04 02:31:53.983363 ip-10-0-134-125 kubenswrapper[2579]: I1104 02:31:53.983347 2579 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Nov 04 02:31:53.987814 ip-10-0-134-125 kubenswrapper[2579]: I1104 02:31:53.987791 2579 kuberuntime_manager.go:304] "Container runtime initialized" containerRuntime="cri-o" version="1.35.10-3.rhaos4.22.git9383502.el9" apiVersion="v1" Nov 04 02:31:53.992110 ip-10-0-134-125 kubenswrapper[2579]: I1104 02:31:53.992096 2579 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Nov 04 02:31:53.992853 ip-10-0-134-125 kubenswrapper[2579]: I1104 02:31:53.992840 2579 kubelet.go:963] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Nov 04 02:31:53.992891 ip-10-0-134-125 kubenswrapper[2579]: I1104 02:31:53.992872 2579 kubelet.go:990] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Nov 04 02:31:53.993028 ip-10-0-134-125 kubenswrapper[2579]: I1104 02:31:53.993018 2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Nov 04 02:31:53.993063 ip-10-0-134-125 kubenswrapper[2579]: I1104 02:31:53.993035 2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Nov 04 02:31:53.993063 ip-10-0-134-125 kubenswrapper[2579]: I1104 02:31:53.993041 2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Nov 04 02:31:53.993063 ip-10-0-134-125 kubenswrapper[2579]: I1104 02:31:53.993047 2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Nov 04 02:31:53.993063 ip-10-0-134-125 kubenswrapper[2579]: I1104 02:31:53.993052 2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Nov 04 02:31:53.993063 ip-10-0-134-125 kubenswrapper[2579]: I1104 02:31:53.993058 2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Nov 04 02:31:53.993532 ip-10-0-134-125 kubenswrapper[2579]: I1104 02:31:53.993523 2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Nov 04 02:31:53.993565 ip-10-0-134-125 kubenswrapper[2579]: I1104 02:31:53.993533 2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Nov 04 02:31:53.993565 ip-10-0-134-125 kubenswrapper[2579]: I1104 02:31:53.993540 2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Nov 04 02:31:53.993565 ip-10-0-134-125 kubenswrapper[2579]: I1104 02:31:53.993546 2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Nov 04 02:31:53.993565 ip-10-0-134-125 kubenswrapper[2579]: I1104 02:31:53.993555 2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Nov 04 02:31:53.993565 ip-10-0-134-125 kubenswrapper[2579]: I1104 02:31:53.993564 2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Nov 04 02:31:53.994561 ip-10-0-134-125 kubenswrapper[2579]: I1104 02:31:53.994550 2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Nov 04 02:31:53.994591 ip-10-0-134-125 kubenswrapper[2579]: I1104 02:31:53.994565 2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Nov 04 02:31:53.998465 ip-10-0-134-125 kubenswrapper[2579]: I1104 02:31:53.998445 2579 server.go:1263] "Started kubelet" Nov 04 02:31:53.998586 ip-10-0-134-125 kubenswrapper[2579]: I1104 02:31:53.998542 2579 server.go:182] "Starting to listen" address="0.0.0.0" port=10250 Nov 04 02:31:53.998586 ip-10-0-134-125 kubenswrapper[2579]: I1104 02:31:53.998538 2579 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Nov 04 02:31:53.998720 ip-10-0-134-125 kubenswrapper[2579]: I1104 02:31:53.998612 2579 server_v1.go:49] "podresources" method="list" useActivePods=true Nov 04 02:31:53.999260 ip-10-0-134-125 systemd[1]: Started Kubernetes Kubelet. Nov 04 02:31:53.999779 ip-10-0-134-125 kubenswrapper[2579]: I1104 02:31:53.999558 2579 server.go:254] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Nov 04 02:31:54.000482 ip-10-0-134-125 kubenswrapper[2579]: I1104 02:31:54.000391 2579 server.go:317] "Adding debug handlers to kubelet server" Nov 04 02:31:54.003470 ip-10-0-134-125 kubenswrapper[2579]: E1104 02:31:54.003449 2579 kubelet.go:1679] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache" Nov 04 02:31:54.005106 ip-10-0-134-125 kubenswrapper[2579]: I1104 02:31:54.005091 2579 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Nov 04 02:31:54.006129 ip-10-0-134-125 kubenswrapper[2579]: I1104 02:31:54.006115 2579 fs_resource_analyzer.go:69] "Starting FS ResourceAnalyzer" Nov 04 02:31:54.006247 ip-10-0-134-125 kubenswrapper[2579]: E1104 02:31:54.006226 2579 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-134-125.us-west-1.compute.internal\" not found" Nov 04 02:31:54.006628 ip-10-0-134-125 kubenswrapper[2579]: I1104 02:31:54.006610 2579 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Nov 04 02:31:54.006628 ip-10-0-134-125 kubenswrapper[2579]: I1104 02:31:54.006613 2579 volume_manager.go:309] "The desired_state_of_world populator starts" Nov 04 02:31:54.006793 ip-10-0-134-125 kubenswrapper[2579]: I1104 02:31:54.006636 2579 volume_manager.go:311] "Starting Kubelet Volume Manager" Nov 04 02:31:54.006852 ip-10-0-134-125 kubenswrapper[2579]: I1104 02:31:54.006823 2579 reconstruct.go:97] "Volume reconstruction finished" Nov 04 02:31:54.006852 ip-10-0-134-125 kubenswrapper[2579]: I1104 02:31:54.006833 2579 reconciler.go:29] "Reconciler: start to sync state" Nov 04 02:31:54.007004 ip-10-0-134-125 kubenswrapper[2579]: I1104 02:31:54.006991 2579 factory.go:55] Registering systemd factory Nov 04 02:31:54.007034 ip-10-0-134-125 kubenswrapper[2579]: I1104 02:31:54.007008 2579 factory.go:223] Registration of the systemd container factory successfully Nov 04 02:31:54.007254 ip-10-0-134-125 kubenswrapper[2579]: I1104 02:31:54.007242 2579 factory.go:153] Registering CRI-O factory Nov 04 02:31:54.007306 ip-10-0-134-125 kubenswrapper[2579]: I1104 02:31:54.007255 2579 factory.go:223] Registration of the crio container factory successfully Nov 04 02:31:54.007336 ip-10-0-134-125 kubenswrapper[2579]: I1104 02:31:54.007328 2579 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Nov 04 02:31:54.007381 ip-10-0-134-125 kubenswrapper[2579]: I1104 02:31:54.007342 2579 factory.go:103] Registering Raw factory Nov 04 02:31:54.007381 ip-10-0-134-125 kubenswrapper[2579]: I1104 02:31:54.007352 2579 manager.go:1199] Started watching for new ooms in manager Nov 04 02:31:54.008221 ip-10-0-134-125 kubenswrapper[2579]: I1104 02:31:54.008205 2579 manager.go:319] Starting recovery of all containers Nov 04 02:31:54.019400 ip-10-0-134-125 kubenswrapper[2579]: I1104 02:31:54.019376 2579 manager.go:324] Recovery completed Nov 04 02:31:54.023899 ip-10-0-134-125 kubenswrapper[2579]: I1104 02:31:54.023883 2579 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 04 02:31:54.025434 ip-10-0-134-125 kubenswrapper[2579]: I1104 02:31:54.025404 2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-125.us-west-1.compute.internal" event="NodeHasSufficientMemory" Nov 04 02:31:54.025539 ip-10-0-134-125 kubenswrapper[2579]: I1104 02:31:54.025443 2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-125.us-west-1.compute.internal" event="NodeHasNoDiskPressure" Nov 04 02:31:54.025539 ip-10-0-134-125 kubenswrapper[2579]: I1104 02:31:54.025454 2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-125.us-west-1.compute.internal" event="NodeHasSufficientPID" Nov 04 02:31:54.026022 ip-10-0-134-125 kubenswrapper[2579]: I1104 02:31:54.026009 2579 cpu_manager.go:226] "Starting" policy="none" Nov 04 02:31:54.026022 ip-10-0-134-125 kubenswrapper[2579]: I1104 02:31:54.026018 2579 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 04 02:31:54.026099 ip-10-0-134-125 kubenswrapper[2579]: I1104 02:31:54.026020 2579 cpu_manager.go:227] "Reconciling" reconcilePeriod="10s" Nov 04 02:31:54.026099 ip-10-0-134-125 kubenswrapper[2579]: I1104 02:31:54.026049 2579 state_mem.go:41] "Initialized" logger="CPUManager state checkpoint.CPUManager state memory" Nov 04 02:31:54.026744 ip-10-0-134-125 kubenswrapper[2579]: I1104 02:31:54.026727 2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-125.us-west-1.compute.internal" event="NodeHasSufficientMemory" Nov 04 02:31:54.026817 ip-10-0-134-125 kubenswrapper[2579]: I1104 02:31:54.026758 2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-125.us-west-1.compute.internal" event="NodeHasNoDiskPressure" Nov 04 02:31:54.026817 ip-10-0-134-125 kubenswrapper[2579]: I1104 02:31:54.026770 2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-125.us-west-1.compute.internal" event="NodeHasSufficientPID" Nov 04 02:31:54.028179 ip-10-0-134-125 kubenswrapper[2579]: I1104 02:31:54.028166 2579 policy_none.go:50] "Start" Nov 04 02:31:54.028238 ip-10-0-134-125 kubenswrapper[2579]: I1104 02:31:54.028184 2579 memory_manager.go:187] "Starting memorymanager" policy="None" Nov 04 02:31:54.028238 ip-10-0-134-125 kubenswrapper[2579]: I1104 02:31:54.028197 2579 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Nov 04 02:31:54.030144 ip-10-0-134-125 kubenswrapper[2579]: I1104 02:31:54.030130 2579 policy_none.go:44] "Start" Nov 04 02:31:54.065440 ip-10-0-134-125 kubenswrapper[2579]: I1104 02:31:54.065268 2579 node_container_manager_linux.go:130] "Enforcing system reserved compressible on cgroup" cgroupName="/system.slice" limits={"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"} Nov 04 02:31:54.069971 ip-10-0-134-125 kubenswrapper[2579]: I1104 02:31:54.069958 2579 manager.go:341] "Starting Device Plugin manager" Nov 04 02:31:54.070062 ip-10-0-134-125 kubenswrapper[2579]: E1104 02:31:54.069995 2579 manager.go:525] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Nov 04 02:31:54.070062 ip-10-0-134-125 kubenswrapper[2579]: I1104 02:31:54.070006 2579 server.go:87] "Starting device plugin registration server" Nov 04 02:31:54.070238 ip-10-0-134-125 kubenswrapper[2579]: I1104 02:31:54.070224 2579 eviction_manager.go:194] "Eviction manager: starting control loop" Nov 04 02:31:54.070287 ip-10-0-134-125 kubenswrapper[2579]: I1104 02:31:54.070238 2579 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Nov 04 02:31:54.070348 ip-10-0-134-125 kubenswrapper[2579]: I1104 02:31:54.070330 2579 plugin_watcher.go:53] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Nov 04 02:31:54.070473 ip-10-0-134-125 kubenswrapper[2579]: I1104 02:31:54.070460 2579 plugin_manager.go:119] "The desired_state_of_world populator (plugin watcher) starts" Nov 04 02:31:54.070542 ip-10-0-134-125 kubenswrapper[2579]: I1104 02:31:54.070474 2579 plugin_manager.go:121] "Starting Kubelet Plugin Manager" Nov 04 02:31:54.070920 ip-10-0-134-125 kubenswrapper[2579]: E1104 02:31:54.070902 2579 eviction_manager.go:272] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Nov 04 02:31:54.070988 ip-10-0-134-125 kubenswrapper[2579]: E1104 02:31:54.070964 2579 eviction_manager.go:297] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-134-125.us-west-1.compute.internal\" not found" Nov 04 02:31:54.147654 ip-10-0-134-125 kubenswrapper[2579]: I1104 02:31:54.147624 2579 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Nov 04 02:31:54.149517 ip-10-0-134-125 kubenswrapper[2579]: I1104 02:31:54.149009 2579 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Nov 04 02:31:54.149517 ip-10-0-134-125 kubenswrapper[2579]: I1104 02:31:54.149033 2579 status_manager.go:249] "Starting to sync pod status with apiserver" Nov 04 02:31:54.149517 ip-10-0-134-125 kubenswrapper[2579]: I1104 02:31:54.149057 2579 kubelet.go:2521] "Starting kubelet main sync loop" Nov 04 02:31:54.149517 ip-10-0-134-125 kubenswrapper[2579]: E1104 02:31:54.149098 2579 kubelet.go:2545] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful" Nov 04 02:31:54.151783 ip-10-0-134-125 kubenswrapper[2579]: E1104 02:31:54.151757 2579 controller.go:201] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-134-125.us-west-1.compute.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms" Nov 04 02:31:54.151783 ip-10-0-134-125 kubenswrapper[2579]: I1104 02:31:54.151757 2579 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-134-125.us-west-1.compute.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Nov 04 02:31:54.170556 ip-10-0-134-125 kubenswrapper[2579]: I1104 02:31:54.170489 2579 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 04 02:31:54.171366 ip-10-0-134-125 kubenswrapper[2579]: I1104 02:31:54.171350 2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-125.us-west-1.compute.internal" event="NodeHasSufficientMemory" Nov 04 02:31:54.171445 ip-10-0-134-125 kubenswrapper[2579]: I1104 02:31:54.171382 2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-125.us-west-1.compute.internal" event="NodeHasNoDiskPressure" Nov 04 02:31:54.171445 ip-10-0-134-125 kubenswrapper[2579]: I1104 02:31:54.171392 2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-125.us-west-1.compute.internal" event="NodeHasSufficientPID" Nov 04 02:31:54.171445 ip-10-0-134-125 kubenswrapper[2579]: I1104 02:31:54.171418 2579 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-134-125.us-west-1.compute.internal" Nov 04 02:31:54.200829 ip-10-0-134-125 kubenswrapper[2579]: E1104 02:31:54.200177 2579 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-134-125.us-west-1.compute.internal.18e4b6c6345a45d6 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-134-125.us-west-1.compute.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-134-125.us-west-1.compute.internal,},FirstTimestamp:2026-11-04 02:31:53.998411222 +0000 UTC m=+0.353518316,LastTimestamp:2026-11-04 02:31:53.998411222 +0000 UTC m=+0.353518316,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-134-125.us-west-1.compute.internal,}" Nov 04 02:31:54.219202 ip-10-0-134-125 kubenswrapper[2579]: I1104 02:31:54.219180 2579 reflector.go:446] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Nov 04 02:31:54.219305 ip-10-0-134-125 kubenswrapper[2579]: I1104 02:31:54.219221 2579 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-h8ndj" Nov 04 02:31:54.250194 ip-10-0-134-125 kubenswrapper[2579]: I1104 02:31:54.250142 2579 kubelet.go:2608] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-125.us-west-1.compute.internal","kube-system/kube-apiserver-proxy-ip-10-0-134-125.us-west-1.compute.internal"] Nov 04 02:31:54.250330 ip-10-0-134-125 kubenswrapper[2579]: I1104 02:31:54.250230 2579 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 04 02:31:54.251145 ip-10-0-134-125 kubenswrapper[2579]: I1104 02:31:54.251127 2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-125.us-west-1.compute.internal" event="NodeHasSufficientMemory" Nov 04 02:31:54.251216 ip-10-0-134-125 kubenswrapper[2579]: I1104 02:31:54.251166 2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-125.us-west-1.compute.internal" event="NodeHasNoDiskPressure" Nov 04 02:31:54.251216 ip-10-0-134-125 kubenswrapper[2579]: I1104 02:31:54.251181 2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-125.us-west-1.compute.internal" event="NodeHasSufficientPID" Nov 04 02:31:54.253367 ip-10-0-134-125 kubenswrapper[2579]: I1104 02:31:54.253354 2579 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 04 02:31:54.253547 ip-10-0-134-125 kubenswrapper[2579]: I1104 02:31:54.253531 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-125.us-west-1.compute.internal" Nov 04 02:31:54.253601 ip-10-0-134-125 kubenswrapper[2579]: I1104 02:31:54.253566 2579 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 04 02:31:54.254108 ip-10-0-134-125 kubenswrapper[2579]: I1104 02:31:54.254092 2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-125.us-west-1.compute.internal" event="NodeHasSufficientMemory" Nov 04 02:31:54.254159 ip-10-0-134-125 kubenswrapper[2579]: I1104 02:31:54.254116 2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-125.us-west-1.compute.internal" event="NodeHasNoDiskPressure" Nov 04 02:31:54.254159 ip-10-0-134-125 kubenswrapper[2579]: I1104 02:31:54.254126 2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-125.us-west-1.compute.internal" event="NodeHasSufficientPID" Nov 04 02:31:54.254240 ip-10-0-134-125 kubenswrapper[2579]: I1104 02:31:54.254177 2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-125.us-west-1.compute.internal" event="NodeHasSufficientMemory" Nov 04 02:31:54.254240 ip-10-0-134-125 kubenswrapper[2579]: I1104 02:31:54.254200 2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-125.us-west-1.compute.internal" event="NodeHasNoDiskPressure" Nov 04 02:31:54.254240 ip-10-0-134-125 kubenswrapper[2579]: I1104 02:31:54.254216 2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-125.us-west-1.compute.internal" event="NodeHasSufficientPID" Nov 04 02:31:54.256189 ip-10-0-134-125 kubenswrapper[2579]: I1104 02:31:54.256176 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-134-125.us-west-1.compute.internal" Nov 04 02:31:54.256252 ip-10-0-134-125 kubenswrapper[2579]: I1104 02:31:54.256200 2579 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 04 02:31:54.256982 ip-10-0-134-125 kubenswrapper[2579]: I1104 02:31:54.256969 2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-125.us-west-1.compute.internal" event="NodeHasSufficientMemory" Nov 04 02:31:54.257052 ip-10-0-134-125 kubenswrapper[2579]: I1104 02:31:54.256995 2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-125.us-west-1.compute.internal" event="NodeHasNoDiskPressure" Nov 04 02:31:54.257052 ip-10-0-134-125 kubenswrapper[2579]: I1104 02:31:54.257005 2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-125.us-west-1.compute.internal" event="NodeHasSufficientPID" Nov 04 02:31:54.275138 ip-10-0-134-125 kubenswrapper[2579]: E1104 02:31:54.275121 2579 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-134-125.us-west-1.compute.internal\" not found" node="ip-10-0-134-125.us-west-1.compute.internal" Nov 04 02:31:54.277376 ip-10-0-134-125 kubenswrapper[2579]: E1104 02:31:54.277359 2579 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-134-125.us-west-1.compute.internal\" not found" node="ip-10-0-134-125.us-west-1.compute.internal" Nov 04 02:31:54.308572 ip-10-0-134-125 kubenswrapper[2579]: I1104 02:31:54.308549 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/c7038986364b9e4776a5785da4dd2f01-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-134-125.us-west-1.compute.internal\" (UID: \"c7038986364b9e4776a5785da4dd2f01\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-125.us-west-1.compute.internal" Nov 04 02:31:54.308739 ip-10-0-134-125 kubenswrapper[2579]: I1104 02:31:54.308574 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/c7038986364b9e4776a5785da4dd2f01-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-134-125.us-west-1.compute.internal\" (UID: \"c7038986364b9e4776a5785da4dd2f01\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-125.us-west-1.compute.internal" Nov 04 02:31:54.308739 ip-10-0-134-125 kubenswrapper[2579]: I1104 02:31:54.308596 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/ee7b213d76bf8a5295d150c84b46451e-config\") pod \"kube-apiserver-proxy-ip-10-0-134-125.us-west-1.compute.internal\" (UID: \"ee7b213d76bf8a5295d150c84b46451e\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-134-125.us-west-1.compute.internal" Nov 04 02:31:54.310101 ip-10-0-134-125 kubenswrapper[2579]: I1104 02:31:54.310076 2579 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161" Nov 04 02:31:54.361145 ip-10-0-134-125 kubenswrapper[2579]: I1104 02:31:54.361108 2579 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161" Nov 04 02:31:54.408092 ip-10-0-134-125 kubenswrapper[2579]: I1104 02:31:54.408061 2579 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161" Nov 04 02:31:54.409255 ip-10-0-134-125 kubenswrapper[2579]: I1104 02:31:54.409238 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/c7038986364b9e4776a5785da4dd2f01-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-134-125.us-west-1.compute.internal\" (UID: \"c7038986364b9e4776a5785da4dd2f01\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-125.us-west-1.compute.internal" Nov 04 02:31:54.409308 ip-10-0-134-125 kubenswrapper[2579]: I1104 02:31:54.409264 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/c7038986364b9e4776a5785da4dd2f01-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-134-125.us-west-1.compute.internal\" (UID: \"c7038986364b9e4776a5785da4dd2f01\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-125.us-west-1.compute.internal" Nov 04 02:31:54.409308 ip-10-0-134-125 kubenswrapper[2579]: I1104 02:31:54.409283 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/ee7b213d76bf8a5295d150c84b46451e-config\") pod \"kube-apiserver-proxy-ip-10-0-134-125.us-west-1.compute.internal\" (UID: \"ee7b213d76bf8a5295d150c84b46451e\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-134-125.us-west-1.compute.internal" Nov 04 02:31:54.409433 ip-10-0-134-125 kubenswrapper[2579]: I1104 02:31:54.409336 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/ee7b213d76bf8a5295d150c84b46451e-config\") pod \"kube-apiserver-proxy-ip-10-0-134-125.us-west-1.compute.internal\" (UID: \"ee7b213d76bf8a5295d150c84b46451e\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-134-125.us-west-1.compute.internal" Nov 04 02:31:54.409433 ip-10-0-134-125 kubenswrapper[2579]: I1104 02:31:54.409346 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/c7038986364b9e4776a5785da4dd2f01-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-134-125.us-west-1.compute.internal\" (UID: \"c7038986364b9e4776a5785da4dd2f01\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-125.us-west-1.compute.internal" Nov 04 02:31:54.409433 ip-10-0-134-125 kubenswrapper[2579]: I1104 02:31:54.409375 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/c7038986364b9e4776a5785da4dd2f01-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-134-125.us-west-1.compute.internal\" (UID: \"c7038986364b9e4776a5785da4dd2f01\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-125.us-west-1.compute.internal" Nov 04 02:31:54.409941 ip-10-0-134-125 kubenswrapper[2579]: I1104 02:31:54.409925 2579 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161" Nov 04 02:31:54.410202 ip-10-0-134-125 kubenswrapper[2579]: I1104 02:31:54.410188 2579 kubelet_node_status.go:80] "Successfully registered node" node="ip-10-0-134-125.us-west-1.compute.internal" Nov 04 02:31:54.410240 ip-10-0-134-125 kubenswrapper[2579]: E1104 02:31:54.410210 2579 kubelet_node_status.go:524] "Error updating node status, will retry" err="error getting node \"ip-10-0-134-125.us-west-1.compute.internal\": node \"ip-10-0-134-125.us-west-1.compute.internal\" not found" Nov 04 02:31:54.506816 ip-10-0-134-125 kubenswrapper[2579]: I1104 02:31:54.506737 2579 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-125.us-west-1.compute.internal" Nov 04 02:31:54.567398 ip-10-0-134-125 kubenswrapper[2579]: I1104 02:31:54.567364 2579 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Nov 04 02:31:54.568414 ip-10-0-134-125 kubenswrapper[2579]: I1104 02:31:54.568396 2579 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-134-125.us-west-1.compute.internal" Nov 04 02:31:54.578053 ip-10-0-134-125 kubenswrapper[2579]: I1104 02:31:54.578026 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-125.us-west-1.compute.internal" Nov 04 02:31:54.580195 ip-10-0-134-125 kubenswrapper[2579]: I1104 02:31:54.580161 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-134-125.us-west-1.compute.internal" Nov 04 02:31:54.626071 ip-10-0-134-125 kubenswrapper[2579]: I1104 02:31:54.626048 2579 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Nov 04 02:31:54.945540 ip-10-0-134-125 kubenswrapper[2579]: I1104 02:31:54.945461 2579 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials" Nov 04 02:31:54.946070 ip-10-0-134-125 kubenswrapper[2579]: I1104 02:31:54.945649 2579 reflector.go:578] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.CSIDriver" err="very short watch: k8s.io/client-go/informers/factory.go:161: Unexpected watch close - watch lasted less than a second and no items received" Nov 04 02:31:54.946070 ip-10-0-134-125 kubenswrapper[2579]: E1104 02:31:54.945600 2579 event.go:368] "Unable to write event (may retry after sleeping)" err="Patch \"https://a7878fea1d4a142dfa615cdc355a2760-4128f5fe3c6c09e5.elb.us-east-2.amazonaws.com:6443/api/v1/namespaces/default/events/ip-10-0-134-125.us-west-1.compute.internal.18e4b6c635f70041\": read tcp 10.0.134.125:34590->18.227.234.46:6443: use of closed network connection" event="&Event{ObjectMeta:{ip-10-0-134-125.us-west-1.compute.internal.18e4b6c635f70041 default 5146 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-134-125.us-west-1.compute.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientPID,Message:Node ip-10-0-134-125.us-west-1.compute.internal status is now: NodeHasSufficientPID,Source:EventSource{Component:kubelet,Host:ip-10-0-134-125.us-west-1.compute.internal,},FirstTimestamp:2026-11-04 02:31:54 +0000 UTC,LastTimestamp:2026-11-04 02:31:54.171396108 +0000 UTC m=+0.526503205,Count:3,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-134-125.us-west-1.compute.internal,}" Nov 04 02:31:54.946070 ip-10-0-134-125 kubenswrapper[2579]: I1104 02:31:54.945682 2579 reflector.go:578] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:161: Unexpected watch close - watch lasted less than a second and no items received" Nov 04 02:31:54.946070 ip-10-0-134-125 kubenswrapper[2579]: I1104 02:31:54.945710 2579 reflector.go:578] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.Service" err="very short watch: k8s.io/client-go/informers/factory.go:161: Unexpected watch close - watch lasted less than a second and no items received" Nov 04 02:31:54.983528 ip-10-0-134-125 kubenswrapper[2579]: I1104 02:31:54.983505 2579 apiserver.go:52] "Watching apiserver" Nov 04 02:31:55.005751 ip-10-0-134-125 kubenswrapper[2579]: I1104 02:31:55.005732 2579 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving" Nov 04 02:31:55.223477 ip-10-0-134-125 kubenswrapper[2579]: I1104 02:31:55.223374 2579 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-11-03 02:26:54 +0000 UTC" deadline="2028-06-19 13:16:45.033463849 +0000 UTC" Nov 04 02:31:55.223477 ip-10-0-134-125 kubenswrapper[2579]: I1104 02:31:55.223423 2579 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="14242h44m49.810049179s" Nov 04 02:31:55.254865 ip-10-0-134-125 kubenswrapper[2579]: I1104 02:31:55.254835 2579 reflector.go:446] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Nov 04 02:31:55.255226 ip-10-0-134-125 kubenswrapper[2579]: I1104 02:31:55.255202 2579 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-network-diagnostics/network-check-target-zd8z9","kube-system/kube-apiserver-proxy-ip-10-0-134-125.us-west-1.compute.internal","openshift-dns/node-resolver-kzzwk","openshift-multus/multus-tglvr","kube-system/global-pull-secret-syncer-krvqk","openshift-cluster-node-tuning-operator/tuned-ncfkk","openshift-image-registry/node-ca-2h96g","openshift-multus/multus-additional-cni-plugins-sgx95","openshift-multus/network-metrics-daemon-bdcds","openshift-network-operator/iptables-alerter-sx4w7","openshift-ovn-kubernetes/ovnkube-node-vwcx8","kube-system/konnectivity-agent-wcp7z","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6mvxj","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-125.us-west-1.compute.internal"] Nov 04 02:31:55.257718 ip-10-0-134-125 kubenswrapper[2579]: I1104 02:31:55.257695 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zd8z9" Nov 04 02:31:55.257811 ip-10-0-134-125 kubenswrapper[2579]: E1104 02:31:55.257773 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zd8z9" podUID="6e27a76f-fbdb-4ce6-9df6-659cab9ad06c" Nov 04 02:31:55.259689 ip-10-0-134-125 kubenswrapper[2579]: I1104 02:31:55.259653 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-kzzwk" Nov 04 02:31:55.261842 ip-10-0-134-125 kubenswrapper[2579]: I1104 02:31:55.261823 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-tglvr" Nov 04 02:31:55.263865 ip-10-0-134-125 kubenswrapper[2579]: I1104 02:31:55.263850 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-krvqk" Nov 04 02:31:55.263935 ip-10-0-134-125 kubenswrapper[2579]: E1104 02:31:55.263917 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-krvqk" podUID="3a7ab1db-ad88-40d9-a7e9-b511bbd4584c" Nov 04 02:31:55.265923 ip-10-0-134-125 kubenswrapper[2579]: I1104 02:31:55.265908 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-ncfkk" Nov 04 02:31:55.267915 ip-10-0-134-125 kubenswrapper[2579]: I1104 02:31:55.267902 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-2h96g" Nov 04 02:31:55.269991 ip-10-0-134-125 kubenswrapper[2579]: I1104 02:31:55.269976 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-sgx95" Nov 04 02:31:55.272042 ip-10-0-134-125 kubenswrapper[2579]: I1104 02:31:55.272026 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bdcds" Nov 04 02:31:55.272130 ip-10-0-134-125 kubenswrapper[2579]: E1104 02:31:55.272079 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-bdcds" podUID="b37e5c97-77c6-4bb4-a9e4-e1ae27484457" Nov 04 02:31:55.274080 ip-10-0-134-125 kubenswrapper[2579]: I1104 02:31:55.274063 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-sx4w7" Nov 04 02:31:55.277462 ip-10-0-134-125 kubenswrapper[2579]: I1104 02:31:55.277438 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-vwcx8" Nov 04 02:31:55.278903 ip-10-0-134-125 kubenswrapper[2579]: I1104 02:31:55.278867 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-wcp7z" Nov 04 02:31:55.281069 ip-10-0-134-125 kubenswrapper[2579]: I1104 02:31:55.281055 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6mvxj" Nov 04 02:31:55.307798 ip-10-0-134-125 kubenswrapper[2579]: I1104 02:31:55.307782 2579 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Nov 04 02:31:55.312331 ip-10-0-134-125 kubenswrapper[2579]: I1104 02:31:55.312309 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Nov 04 02:31:55.312331 ip-10-0-134-125 kubenswrapper[2579]: I1104 02:31:55.312312 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Nov 04 02:31:55.312437 ip-10-0-134-125 kubenswrapper[2579]: I1104 02:31:55.312401 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-vd7gf\"" Nov 04 02:31:55.313718 ip-10-0-134-125 kubenswrapper[2579]: I1104 02:31:55.313702 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Nov 04 02:31:55.313825 ip-10-0-134-125 kubenswrapper[2579]: I1104 02:31:55.313813 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-node-identity-dockercfg-ffw8m\"" Nov 04 02:31:55.313930 ip-10-0-134-125 kubenswrapper[2579]: I1104 02:31:55.313913 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Nov 04 02:31:55.313992 ip-10-0-134-125 kubenswrapper[2579]: I1104 02:31:55.313930 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Nov 04 02:31:55.313992 ip-10-0-134-125 kubenswrapper[2579]: I1104 02:31:55.313948 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Nov 04 02:31:55.314414 ip-10-0-134-125 kubenswrapper[2579]: I1104 02:31:55.314397 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/ffbb2d83-5a93-4a5b-bce3-e40afb8f58ea-host-cni-netd\") pod \"ovnkube-node-vwcx8\" (UID: \"ffbb2d83-5a93-4a5b-bce3-e40afb8f58ea\") " pod="openshift-ovn-kubernetes/ovnkube-node-vwcx8" Nov 04 02:31:55.314483 ip-10-0-134-125 kubenswrapper[2579]: I1104 02:31:55.314421 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/ffbb2d83-5a93-4a5b-bce3-e40afb8f58ea-ovnkube-config\") pod \"ovnkube-node-vwcx8\" (UID: \"ffbb2d83-5a93-4a5b-bce3-e40afb8f58ea\") " pod="openshift-ovn-kubernetes/ovnkube-node-vwcx8" Nov 04 02:31:55.314483 ip-10-0-134-125 kubenswrapper[2579]: I1104 02:31:55.314438 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/7cb29949-bfe4-47a2-bc17-115a8d4d895d-host\") pod \"node-ca-2h96g\" (UID: \"7cb29949-bfe4-47a2-bc17-115a8d4d895d\") " pod="openshift-image-registry/node-ca-2h96g" Nov 04 02:31:55.314483 ip-10-0-134-125 kubenswrapper[2579]: I1104 02:31:55.314467 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/3a7ab1db-ad88-40d9-a7e9-b511bbd4584c-original-pull-secret\") pod \"global-pull-secret-syncer-krvqk\" (UID: \"3a7ab1db-ad88-40d9-a7e9-b511bbd4584c\") " pod="kube-system/global-pull-secret-syncer-krvqk" Nov 04 02:31:55.314660 ip-10-0-134-125 kubenswrapper[2579]: I1104 02:31:55.314505 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/cb1915a0-5505-4f32-8f79-2f638465e404-var-lib-kubelet\") pod \"tuned-ncfkk\" (UID: \"cb1915a0-5505-4f32-8f79-2f638465e404\") " pod="openshift-cluster-node-tuning-operator/tuned-ncfkk" Nov 04 02:31:55.314660 ip-10-0-134-125 kubenswrapper[2579]: I1104 02:31:55.314535 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/f58e4ac8-c132-4716-b6c0-8d8ddb9c19e5-etc-selinux\") pod \"aws-ebs-csi-driver-node-6mvxj\" (UID: \"f58e4ac8-c132-4716-b6c0-8d8ddb9c19e5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6mvxj" Nov 04 02:31:55.314660 ip-10-0-134-125 kubenswrapper[2579]: I1104 02:31:55.314558 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/5590efb1-88b8-41ad-b0ee-37b2ff9442de-cnibin\") pod \"multus-additional-cni-plugins-sgx95\" (UID: \"5590efb1-88b8-41ad-b0ee-37b2ff9442de\") " pod="openshift-multus/multus-additional-cni-plugins-sgx95" Nov 04 02:31:55.314660 ip-10-0-134-125 kubenswrapper[2579]: I1104 02:31:55.314606 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dddmg\" (UniqueName: \"kubernetes.io/projected/6e27a76f-fbdb-4ce6-9df6-659cab9ad06c-kube-api-access-dddmg\") pod \"network-check-target-zd8z9\" (UID: \"6e27a76f-fbdb-4ce6-9df6-659cab9ad06c\") " pod="openshift-network-diagnostics/network-check-target-zd8z9" Nov 04 02:31:55.315043 ip-10-0-134-125 kubenswrapper[2579]: I1104 02:31:55.314656 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/3984ce7b-96c8-464a-aa14-28487cc5b373-hostroot\") pod \"multus-tglvr\" (UID: \"3984ce7b-96c8-464a-aa14-28487cc5b373\") " pod="openshift-multus/multus-tglvr" Nov 04 02:31:55.315043 ip-10-0-134-125 kubenswrapper[2579]: I1104 02:31:55.314723 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/cb1915a0-5505-4f32-8f79-2f638465e404-etc-sysconfig\") pod \"tuned-ncfkk\" (UID: \"cb1915a0-5505-4f32-8f79-2f638465e404\") " pod="openshift-cluster-node-tuning-operator/tuned-ncfkk" Nov 04 02:31:55.315043 ip-10-0-134-125 kubenswrapper[2579]: I1104 02:31:55.314755 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/ffbb2d83-5a93-4a5b-bce3-e40afb8f58ea-log-socket\") pod \"ovnkube-node-vwcx8\" (UID: \"ffbb2d83-5a93-4a5b-bce3-e40afb8f58ea\") " pod="openshift-ovn-kubernetes/ovnkube-node-vwcx8" Nov 04 02:31:55.315043 ip-10-0-134-125 kubenswrapper[2579]: I1104 02:31:55.314788 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/3984ce7b-96c8-464a-aa14-28487cc5b373-multus-socket-dir-parent\") pod \"multus-tglvr\" (UID: \"3984ce7b-96c8-464a-aa14-28487cc5b373\") " pod="openshift-multus/multus-tglvr" Nov 04 02:31:55.315043 ip-10-0-134-125 kubenswrapper[2579]: I1104 02:31:55.314811 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/3a7ab1db-ad88-40d9-a7e9-b511bbd4584c-kubelet-config\") pod \"global-pull-secret-syncer-krvqk\" (UID: \"3a7ab1db-ad88-40d9-a7e9-b511bbd4584c\") " pod="kube-system/global-pull-secret-syncer-krvqk" Nov 04 02:31:55.315043 ip-10-0-134-125 kubenswrapper[2579]: I1104 02:31:55.314830 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/cb1915a0-5505-4f32-8f79-2f638465e404-etc-systemd\") pod \"tuned-ncfkk\" (UID: \"cb1915a0-5505-4f32-8f79-2f638465e404\") " pod="openshift-cluster-node-tuning-operator/tuned-ncfkk" Nov 04 02:31:55.315043 ip-10-0-134-125 kubenswrapper[2579]: I1104 02:31:55.314901 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/5590efb1-88b8-41ad-b0ee-37b2ff9442de-cni-binary-copy\") pod \"multus-additional-cni-plugins-sgx95\" (UID: \"5590efb1-88b8-41ad-b0ee-37b2ff9442de\") " pod="openshift-multus/multus-additional-cni-plugins-sgx95" Nov 04 02:31:55.315043 ip-10-0-134-125 kubenswrapper[2579]: I1104 02:31:55.314931 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/ffbb2d83-5a93-4a5b-bce3-e40afb8f58ea-ovnkube-script-lib\") pod \"ovnkube-node-vwcx8\" (UID: \"ffbb2d83-5a93-4a5b-bce3-e40afb8f58ea\") " pod="openshift-ovn-kubernetes/ovnkube-node-vwcx8" Nov 04 02:31:55.315043 ip-10-0-134-125 kubenswrapper[2579]: I1104 02:31:55.314976 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/3984ce7b-96c8-464a-aa14-28487cc5b373-host-run-multus-certs\") pod \"multus-tglvr\" (UID: \"3984ce7b-96c8-464a-aa14-28487cc5b373\") " pod="openshift-multus/multus-tglvr" Nov 04 02:31:55.315043 ip-10-0-134-125 kubenswrapper[2579]: I1104 02:31:55.315001 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/5b77e30b-cb0c-49c1-80b9-330a138dbfb7-agent-certs\") pod \"konnectivity-agent-wcp7z\" (UID: \"5b77e30b-cb0c-49c1-80b9-330a138dbfb7\") " pod="kube-system/konnectivity-agent-wcp7z" Nov 04 02:31:55.315043 ip-10-0-134-125 kubenswrapper[2579]: I1104 02:31:55.315024 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/f58e4ac8-c132-4716-b6c0-8d8ddb9c19e5-socket-dir\") pod \"aws-ebs-csi-driver-node-6mvxj\" (UID: \"f58e4ac8-c132-4716-b6c0-8d8ddb9c19e5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6mvxj" Nov 04 02:31:55.315571 ip-10-0-134-125 kubenswrapper[2579]: I1104 02:31:55.315076 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8ll77\" (UniqueName: \"kubernetes.io/projected/8b030f29-dfa6-42b1-81a7-07b539ad9aa9-kube-api-access-8ll77\") pod \"node-resolver-kzzwk\" (UID: \"8b030f29-dfa6-42b1-81a7-07b539ad9aa9\") " pod="openshift-dns/node-resolver-kzzwk" Nov 04 02:31:55.315571 ip-10-0-134-125 kubenswrapper[2579]: I1104 02:31:55.315107 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/cb1915a0-5505-4f32-8f79-2f638465e404-run\") pod \"tuned-ncfkk\" (UID: \"cb1915a0-5505-4f32-8f79-2f638465e404\") " pod="openshift-cluster-node-tuning-operator/tuned-ncfkk" Nov 04 02:31:55.315571 ip-10-0-134-125 kubenswrapper[2579]: I1104 02:31:55.315138 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/cb1915a0-5505-4f32-8f79-2f638465e404-lib-modules\") pod \"tuned-ncfkk\" (UID: \"cb1915a0-5505-4f32-8f79-2f638465e404\") " pod="openshift-cluster-node-tuning-operator/tuned-ncfkk" Nov 04 02:31:55.315571 ip-10-0-134-125 kubenswrapper[2579]: I1104 02:31:55.315159 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/cb1915a0-5505-4f32-8f79-2f638465e404-tmp\") pod \"tuned-ncfkk\" (UID: \"cb1915a0-5505-4f32-8f79-2f638465e404\") " pod="openshift-cluster-node-tuning-operator/tuned-ncfkk" Nov 04 02:31:55.315571 ip-10-0-134-125 kubenswrapper[2579]: I1104 02:31:55.315190 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/5590efb1-88b8-41ad-b0ee-37b2ff9442de-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-sgx95\" (UID: \"5590efb1-88b8-41ad-b0ee-37b2ff9442de\") " pod="openshift-multus/multus-additional-cni-plugins-sgx95" Nov 04 02:31:55.315571 ip-10-0-134-125 kubenswrapper[2579]: I1104 02:31:55.315222 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/ffbb2d83-5a93-4a5b-bce3-e40afb8f58ea-run-systemd\") pod \"ovnkube-node-vwcx8\" (UID: \"ffbb2d83-5a93-4a5b-bce3-e40afb8f58ea\") " pod="openshift-ovn-kubernetes/ovnkube-node-vwcx8" Nov 04 02:31:55.315571 ip-10-0-134-125 kubenswrapper[2579]: I1104 02:31:55.315252 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ffbb2d83-5a93-4a5b-bce3-e40afb8f58ea-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-vwcx8\" (UID: \"ffbb2d83-5a93-4a5b-bce3-e40afb8f58ea\") " pod="openshift-ovn-kubernetes/ovnkube-node-vwcx8" Nov 04 02:31:55.315571 ip-10-0-134-125 kubenswrapper[2579]: I1104 02:31:55.315301 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/3984ce7b-96c8-464a-aa14-28487cc5b373-host-run-k8s-cni-cncf-io\") pod \"multus-tglvr\" (UID: \"3984ce7b-96c8-464a-aa14-28487cc5b373\") " pod="openshift-multus/multus-tglvr" Nov 04 02:31:55.315571 ip-10-0-134-125 kubenswrapper[2579]: I1104 02:31:55.315329 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/5590efb1-88b8-41ad-b0ee-37b2ff9442de-system-cni-dir\") pod \"multus-additional-cni-plugins-sgx95\" (UID: \"5590efb1-88b8-41ad-b0ee-37b2ff9442de\") " pod="openshift-multus/multus-additional-cni-plugins-sgx95" Nov 04 02:31:55.315571 ip-10-0-134-125 kubenswrapper[2579]: I1104 02:31:55.315351 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/3984ce7b-96c8-464a-aa14-28487cc5b373-host-run-netns\") pod \"multus-tglvr\" (UID: \"3984ce7b-96c8-464a-aa14-28487cc5b373\") " pod="openshift-multus/multus-tglvr" Nov 04 02:31:55.315571 ip-10-0-134-125 kubenswrapper[2579]: I1104 02:31:55.315366 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/3984ce7b-96c8-464a-aa14-28487cc5b373-host-var-lib-kubelet\") pod \"multus-tglvr\" (UID: \"3984ce7b-96c8-464a-aa14-28487cc5b373\") " pod="openshift-multus/multus-tglvr" Nov 04 02:31:55.315571 ip-10-0-134-125 kubenswrapper[2579]: I1104 02:31:55.315388 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b4snb\" (UniqueName: \"kubernetes.io/projected/f58e4ac8-c132-4716-b6c0-8d8ddb9c19e5-kube-api-access-b4snb\") pod \"aws-ebs-csi-driver-node-6mvxj\" (UID: \"f58e4ac8-c132-4716-b6c0-8d8ddb9c19e5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6mvxj" Nov 04 02:31:55.315571 ip-10-0-134-125 kubenswrapper[2579]: I1104 02:31:55.315421 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/cb1915a0-5505-4f32-8f79-2f638465e404-etc-sysctl-conf\") pod \"tuned-ncfkk\" (UID: \"cb1915a0-5505-4f32-8f79-2f638465e404\") " pod="openshift-cluster-node-tuning-operator/tuned-ncfkk" Nov 04 02:31:55.315571 ip-10-0-134-125 kubenswrapper[2579]: I1104 02:31:55.315439 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/8b030f29-dfa6-42b1-81a7-07b539ad9aa9-hosts-file\") pod \"node-resolver-kzzwk\" (UID: \"8b030f29-dfa6-42b1-81a7-07b539ad9aa9\") " pod="openshift-dns/node-resolver-kzzwk" Nov 04 02:31:55.315571 ip-10-0-134-125 kubenswrapper[2579]: I1104 02:31:55.315452 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/3984ce7b-96c8-464a-aa14-28487cc5b373-host-var-lib-cni-bin\") pod \"multus-tglvr\" (UID: \"3984ce7b-96c8-464a-aa14-28487cc5b373\") " pod="openshift-multus/multus-tglvr" Nov 04 02:31:55.315571 ip-10-0-134-125 kubenswrapper[2579]: I1104 02:31:55.315466 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/cb1915a0-5505-4f32-8f79-2f638465e404-etc-sysctl-d\") pod \"tuned-ncfkk\" (UID: \"cb1915a0-5505-4f32-8f79-2f638465e404\") " pod="openshift-cluster-node-tuning-operator/tuned-ncfkk" Nov 04 02:31:55.316108 ip-10-0-134-125 kubenswrapper[2579]: I1104 02:31:55.315487 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/cb1915a0-5505-4f32-8f79-2f638465e404-host\") pod \"tuned-ncfkk\" (UID: \"cb1915a0-5505-4f32-8f79-2f638465e404\") " pod="openshift-cluster-node-tuning-operator/tuned-ncfkk" Nov 04 02:31:55.316108 ip-10-0-134-125 kubenswrapper[2579]: I1104 02:31:55.315519 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/cb1915a0-5505-4f32-8f79-2f638465e404-etc-tuned\") pod \"tuned-ncfkk\" (UID: \"cb1915a0-5505-4f32-8f79-2f638465e404\") " pod="openshift-cluster-node-tuning-operator/tuned-ncfkk" Nov 04 02:31:55.316108 ip-10-0-134-125 kubenswrapper[2579]: I1104 02:31:55.315536 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4rpxv\" (UniqueName: \"kubernetes.io/projected/5590efb1-88b8-41ad-b0ee-37b2ff9442de-kube-api-access-4rpxv\") pod \"multus-additional-cni-plugins-sgx95\" (UID: \"5590efb1-88b8-41ad-b0ee-37b2ff9442de\") " pod="openshift-multus/multus-additional-cni-plugins-sgx95" Nov 04 02:31:55.316108 ip-10-0-134-125 kubenswrapper[2579]: I1104 02:31:55.315552 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/ffbb2d83-5a93-4a5b-bce3-e40afb8f58ea-host-cni-bin\") pod \"ovnkube-node-vwcx8\" (UID: \"ffbb2d83-5a93-4a5b-bce3-e40afb8f58ea\") " pod="openshift-ovn-kubernetes/ovnkube-node-vwcx8" Nov 04 02:31:55.316108 ip-10-0-134-125 kubenswrapper[2579]: I1104 02:31:55.315576 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/3984ce7b-96c8-464a-aa14-28487cc5b373-system-cni-dir\") pod \"multus-tglvr\" (UID: \"3984ce7b-96c8-464a-aa14-28487cc5b373\") " pod="openshift-multus/multus-tglvr" Nov 04 02:31:55.316108 ip-10-0-134-125 kubenswrapper[2579]: I1104 02:31:55.315596 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-946v4\" (UniqueName: \"kubernetes.io/projected/7cb29949-bfe4-47a2-bc17-115a8d4d895d-kube-api-access-946v4\") pod \"node-ca-2h96g\" (UID: \"7cb29949-bfe4-47a2-bc17-115a8d4d895d\") " pod="openshift-image-registry/node-ca-2h96g" Nov 04 02:31:55.316108 ip-10-0-134-125 kubenswrapper[2579]: I1104 02:31:55.315615 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/5590efb1-88b8-41ad-b0ee-37b2ff9442de-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-sgx95\" (UID: \"5590efb1-88b8-41ad-b0ee-37b2ff9442de\") " pod="openshift-multus/multus-additional-cni-plugins-sgx95" Nov 04 02:31:55.316108 ip-10-0-134-125 kubenswrapper[2579]: I1104 02:31:55.315644 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/ffbb2d83-5a93-4a5b-bce3-e40afb8f58ea-etc-openvswitch\") pod \"ovnkube-node-vwcx8\" (UID: \"ffbb2d83-5a93-4a5b-bce3-e40afb8f58ea\") " pod="openshift-ovn-kubernetes/ovnkube-node-vwcx8" Nov 04 02:31:55.316108 ip-10-0-134-125 kubenswrapper[2579]: I1104 02:31:55.315688 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/ffbb2d83-5a93-4a5b-bce3-e40afb8f58ea-node-log\") pod \"ovnkube-node-vwcx8\" (UID: \"ffbb2d83-5a93-4a5b-bce3-e40afb8f58ea\") " pod="openshift-ovn-kubernetes/ovnkube-node-vwcx8" Nov 04 02:31:55.316108 ip-10-0-134-125 kubenswrapper[2579]: I1104 02:31:55.315704 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/3984ce7b-96c8-464a-aa14-28487cc5b373-host-var-lib-cni-multus\") pod \"multus-tglvr\" (UID: \"3984ce7b-96c8-464a-aa14-28487cc5b373\") " pod="openshift-multus/multus-tglvr" Nov 04 02:31:55.316108 ip-10-0-134-125 kubenswrapper[2579]: I1104 02:31:55.315716 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/cb1915a0-5505-4f32-8f79-2f638465e404-etc-kubernetes\") pod \"tuned-ncfkk\" (UID: \"cb1915a0-5505-4f32-8f79-2f638465e404\") " pod="openshift-cluster-node-tuning-operator/tuned-ncfkk" Nov 04 02:31:55.316108 ip-10-0-134-125 kubenswrapper[2579]: I1104 02:31:55.315730 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n6mc7\" (UniqueName: \"kubernetes.io/projected/cb1915a0-5505-4f32-8f79-2f638465e404-kube-api-access-n6mc7\") pod \"tuned-ncfkk\" (UID: \"cb1915a0-5505-4f32-8f79-2f638465e404\") " pod="openshift-cluster-node-tuning-operator/tuned-ncfkk" Nov 04 02:31:55.316108 ip-10-0-134-125 kubenswrapper[2579]: I1104 02:31:55.315743 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/3984ce7b-96c8-464a-aa14-28487cc5b373-multus-conf-dir\") pod \"multus-tglvr\" (UID: \"3984ce7b-96c8-464a-aa14-28487cc5b373\") " pod="openshift-multus/multus-tglvr" Nov 04 02:31:55.316108 ip-10-0-134-125 kubenswrapper[2579]: I1104 02:31:55.315770 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/3984ce7b-96c8-464a-aa14-28487cc5b373-multus-daemon-config\") pod \"multus-tglvr\" (UID: \"3984ce7b-96c8-464a-aa14-28487cc5b373\") " pod="openshift-multus/multus-tglvr" Nov 04 02:31:55.316108 ip-10-0-134-125 kubenswrapper[2579]: I1104 02:31:55.315796 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b37e5c97-77c6-4bb4-a9e4-e1ae27484457-metrics-certs\") pod \"network-metrics-daemon-bdcds\" (UID: \"b37e5c97-77c6-4bb4-a9e4-e1ae27484457\") " pod="openshift-multus/network-metrics-daemon-bdcds" Nov 04 02:31:55.316108 ip-10-0-134-125 kubenswrapper[2579]: I1104 02:31:55.315811 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zpbsm\" (UniqueName: \"kubernetes.io/projected/b37e5c97-77c6-4bb4-a9e4-e1ae27484457-kube-api-access-zpbsm\") pod \"network-metrics-daemon-bdcds\" (UID: \"b37e5c97-77c6-4bb4-a9e4-e1ae27484457\") " pod="openshift-multus/network-metrics-daemon-bdcds" Nov 04 02:31:55.316688 ip-10-0-134-125 kubenswrapper[2579]: I1104 02:31:55.315827 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/5b77e30b-cb0c-49c1-80b9-330a138dbfb7-konnectivity-ca\") pod \"konnectivity-agent-wcp7z\" (UID: \"5b77e30b-cb0c-49c1-80b9-330a138dbfb7\") " pod="kube-system/konnectivity-agent-wcp7z" Nov 04 02:31:55.316688 ip-10-0-134-125 kubenswrapper[2579]: I1104 02:31:55.315849 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/f58e4ac8-c132-4716-b6c0-8d8ddb9c19e5-registration-dir\") pod \"aws-ebs-csi-driver-node-6mvxj\" (UID: \"f58e4ac8-c132-4716-b6c0-8d8ddb9c19e5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6mvxj" Nov 04 02:31:55.316688 ip-10-0-134-125 kubenswrapper[2579]: I1104 02:31:55.315871 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/cb1915a0-5505-4f32-8f79-2f638465e404-sys\") pod \"tuned-ncfkk\" (UID: \"cb1915a0-5505-4f32-8f79-2f638465e404\") " pod="openshift-cluster-node-tuning-operator/tuned-ncfkk" Nov 04 02:31:55.316688 ip-10-0-134-125 kubenswrapper[2579]: I1104 02:31:55.315893 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/ffbb2d83-5a93-4a5b-bce3-e40afb8f58ea-run-ovn\") pod \"ovnkube-node-vwcx8\" (UID: \"ffbb2d83-5a93-4a5b-bce3-e40afb8f58ea\") " pod="openshift-ovn-kubernetes/ovnkube-node-vwcx8" Nov 04 02:31:55.316688 ip-10-0-134-125 kubenswrapper[2579]: I1104 02:31:55.315912 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/ffbb2d83-5a93-4a5b-bce3-e40afb8f58ea-ovn-node-metrics-cert\") pod \"ovnkube-node-vwcx8\" (UID: \"ffbb2d83-5a93-4a5b-bce3-e40afb8f58ea\") " pod="openshift-ovn-kubernetes/ovnkube-node-vwcx8" Nov 04 02:31:55.316688 ip-10-0-134-125 kubenswrapper[2579]: I1104 02:31:55.315932 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/f58e4ac8-c132-4716-b6c0-8d8ddb9c19e5-sys-fs\") pod \"aws-ebs-csi-driver-node-6mvxj\" (UID: \"f58e4ac8-c132-4716-b6c0-8d8ddb9c19e5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6mvxj" Nov 04 02:31:55.316688 ip-10-0-134-125 kubenswrapper[2579]: I1104 02:31:55.315949 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/ffbb2d83-5a93-4a5b-bce3-e40afb8f58ea-var-lib-openvswitch\") pod \"ovnkube-node-vwcx8\" (UID: \"ffbb2d83-5a93-4a5b-bce3-e40afb8f58ea\") " pod="openshift-ovn-kubernetes/ovnkube-node-vwcx8" Nov 04 02:31:55.316688 ip-10-0-134-125 kubenswrapper[2579]: I1104 02:31:55.315963 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ffbb2d83-5a93-4a5b-bce3-e40afb8f58ea-host-run-ovn-kubernetes\") pod \"ovnkube-node-vwcx8\" (UID: \"ffbb2d83-5a93-4a5b-bce3-e40afb8f58ea\") " pod="openshift-ovn-kubernetes/ovnkube-node-vwcx8" Nov 04 02:31:55.316688 ip-10-0-134-125 kubenswrapper[2579]: I1104 02:31:55.315978 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/3a7ab1db-ad88-40d9-a7e9-b511bbd4584c-dbus\") pod \"global-pull-secret-syncer-krvqk\" (UID: \"3a7ab1db-ad88-40d9-a7e9-b511bbd4584c\") " pod="kube-system/global-pull-secret-syncer-krvqk" Nov 04 02:31:55.316688 ip-10-0-134-125 kubenswrapper[2579]: I1104 02:31:55.316018 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/ffbb2d83-5a93-4a5b-bce3-e40afb8f58ea-host-run-netns\") pod \"ovnkube-node-vwcx8\" (UID: \"ffbb2d83-5a93-4a5b-bce3-e40afb8f58ea\") " pod="openshift-ovn-kubernetes/ovnkube-node-vwcx8" Nov 04 02:31:55.316688 ip-10-0-134-125 kubenswrapper[2579]: I1104 02:31:55.316039 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tp4jl\" (UniqueName: \"kubernetes.io/projected/ffbb2d83-5a93-4a5b-bce3-e40afb8f58ea-kube-api-access-tp4jl\") pod \"ovnkube-node-vwcx8\" (UID: \"ffbb2d83-5a93-4a5b-bce3-e40afb8f58ea\") " pod="openshift-ovn-kubernetes/ovnkube-node-vwcx8" Nov 04 02:31:55.316688 ip-10-0-134-125 kubenswrapper[2579]: I1104 02:31:55.316059 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/3984ce7b-96c8-464a-aa14-28487cc5b373-cni-binary-copy\") pod \"multus-tglvr\" (UID: \"3984ce7b-96c8-464a-aa14-28487cc5b373\") " pod="openshift-multus/multus-tglvr" Nov 04 02:31:55.316688 ip-10-0-134-125 kubenswrapper[2579]: I1104 02:31:55.316081 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-stzcf\" (UniqueName: \"kubernetes.io/projected/524e33f8-723b-4868-a4f5-5d7eb92a5fe3-kube-api-access-stzcf\") pod \"iptables-alerter-sx4w7\" (UID: \"524e33f8-723b-4868-a4f5-5d7eb92a5fe3\") " pod="openshift-network-operator/iptables-alerter-sx4w7" Nov 04 02:31:55.316688 ip-10-0-134-125 kubenswrapper[2579]: I1104 02:31:55.316104 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/f58e4ac8-c132-4716-b6c0-8d8ddb9c19e5-kubelet-dir\") pod \"aws-ebs-csi-driver-node-6mvxj\" (UID: \"f58e4ac8-c132-4716-b6c0-8d8ddb9c19e5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6mvxj" Nov 04 02:31:55.316688 ip-10-0-134-125 kubenswrapper[2579]: I1104 02:31:55.316127 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/7cb29949-bfe4-47a2-bc17-115a8d4d895d-serviceca\") pod \"node-ca-2h96g\" (UID: \"7cb29949-bfe4-47a2-bc17-115a8d4d895d\") " pod="openshift-image-registry/node-ca-2h96g" Nov 04 02:31:55.316688 ip-10-0-134-125 kubenswrapper[2579]: I1104 02:31:55.316156 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/5590efb1-88b8-41ad-b0ee-37b2ff9442de-tuning-conf-dir\") pod \"multus-additional-cni-plugins-sgx95\" (UID: \"5590efb1-88b8-41ad-b0ee-37b2ff9442de\") " pod="openshift-multus/multus-additional-cni-plugins-sgx95" Nov 04 02:31:55.317129 ip-10-0-134-125 kubenswrapper[2579]: I1104 02:31:55.316186 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/3984ce7b-96c8-464a-aa14-28487cc5b373-cnibin\") pod \"multus-tglvr\" (UID: \"3984ce7b-96c8-464a-aa14-28487cc5b373\") " pod="openshift-multus/multus-tglvr" Nov 04 02:31:55.317129 ip-10-0-134-125 kubenswrapper[2579]: I1104 02:31:55.316214 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-plfvd\" (UniqueName: \"kubernetes.io/projected/3984ce7b-96c8-464a-aa14-28487cc5b373-kube-api-access-plfvd\") pod \"multus-tglvr\" (UID: \"3984ce7b-96c8-464a-aa14-28487cc5b373\") " pod="openshift-multus/multus-tglvr" Nov 04 02:31:55.317129 ip-10-0-134-125 kubenswrapper[2579]: I1104 02:31:55.316240 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/524e33f8-723b-4868-a4f5-5d7eb92a5fe3-iptables-alerter-script\") pod \"iptables-alerter-sx4w7\" (UID: \"524e33f8-723b-4868-a4f5-5d7eb92a5fe3\") " pod="openshift-network-operator/iptables-alerter-sx4w7" Nov 04 02:31:55.317129 ip-10-0-134-125 kubenswrapper[2579]: I1104 02:31:55.316271 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/cb1915a0-5505-4f32-8f79-2f638465e404-etc-modprobe-d\") pod \"tuned-ncfkk\" (UID: \"cb1915a0-5505-4f32-8f79-2f638465e404\") " pod="openshift-cluster-node-tuning-operator/tuned-ncfkk" Nov 04 02:31:55.317129 ip-10-0-134-125 kubenswrapper[2579]: I1104 02:31:55.316299 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8b030f29-dfa6-42b1-81a7-07b539ad9aa9-tmp-dir\") pod \"node-resolver-kzzwk\" (UID: \"8b030f29-dfa6-42b1-81a7-07b539ad9aa9\") " pod="openshift-dns/node-resolver-kzzwk" Nov 04 02:31:55.317129 ip-10-0-134-125 kubenswrapper[2579]: I1104 02:31:55.316320 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/ffbb2d83-5a93-4a5b-bce3-e40afb8f58ea-systemd-units\") pod \"ovnkube-node-vwcx8\" (UID: \"ffbb2d83-5a93-4a5b-bce3-e40afb8f58ea\") " pod="openshift-ovn-kubernetes/ovnkube-node-vwcx8" Nov 04 02:31:55.317129 ip-10-0-134-125 kubenswrapper[2579]: I1104 02:31:55.316342 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/ffbb2d83-5a93-4a5b-bce3-e40afb8f58ea-env-overrides\") pod \"ovnkube-node-vwcx8\" (UID: \"ffbb2d83-5a93-4a5b-bce3-e40afb8f58ea\") " pod="openshift-ovn-kubernetes/ovnkube-node-vwcx8" Nov 04 02:31:55.317129 ip-10-0-134-125 kubenswrapper[2579]: I1104 02:31:55.316358 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/3984ce7b-96c8-464a-aa14-28487cc5b373-multus-cni-dir\") pod \"multus-tglvr\" (UID: \"3984ce7b-96c8-464a-aa14-28487cc5b373\") " pod="openshift-multus/multus-tglvr" Nov 04 02:31:55.317129 ip-10-0-134-125 kubenswrapper[2579]: I1104 02:31:55.316374 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/ffbb2d83-5a93-4a5b-bce3-e40afb8f58ea-host-kubelet\") pod \"ovnkube-node-vwcx8\" (UID: \"ffbb2d83-5a93-4a5b-bce3-e40afb8f58ea\") " pod="openshift-ovn-kubernetes/ovnkube-node-vwcx8" Nov 04 02:31:55.317129 ip-10-0-134-125 kubenswrapper[2579]: I1104 02:31:55.316409 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/ffbb2d83-5a93-4a5b-bce3-e40afb8f58ea-host-slash\") pod \"ovnkube-node-vwcx8\" (UID: \"ffbb2d83-5a93-4a5b-bce3-e40afb8f58ea\") " pod="openshift-ovn-kubernetes/ovnkube-node-vwcx8" Nov 04 02:31:55.317129 ip-10-0-134-125 kubenswrapper[2579]: I1104 02:31:55.316442 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/ffbb2d83-5a93-4a5b-bce3-e40afb8f58ea-run-openvswitch\") pod \"ovnkube-node-vwcx8\" (UID: \"ffbb2d83-5a93-4a5b-bce3-e40afb8f58ea\") " pod="openshift-ovn-kubernetes/ovnkube-node-vwcx8" Nov 04 02:31:55.317129 ip-10-0-134-125 kubenswrapper[2579]: I1104 02:31:55.316466 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/3984ce7b-96c8-464a-aa14-28487cc5b373-etc-kubernetes\") pod \"multus-tglvr\" (UID: \"3984ce7b-96c8-464a-aa14-28487cc5b373\") " pod="openshift-multus/multus-tglvr" Nov 04 02:31:55.317129 ip-10-0-134-125 kubenswrapper[2579]: I1104 02:31:55.316489 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/524e33f8-723b-4868-a4f5-5d7eb92a5fe3-host-slash\") pod \"iptables-alerter-sx4w7\" (UID: \"524e33f8-723b-4868-a4f5-5d7eb92a5fe3\") " pod="openshift-network-operator/iptables-alerter-sx4w7" Nov 04 02:31:55.317129 ip-10-0-134-125 kubenswrapper[2579]: I1104 02:31:55.316511 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/f58e4ac8-c132-4716-b6c0-8d8ddb9c19e5-device-dir\") pod \"aws-ebs-csi-driver-node-6mvxj\" (UID: \"f58e4ac8-c132-4716-b6c0-8d8ddb9c19e5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6mvxj" Nov 04 02:31:55.317590 ip-10-0-134-125 kubenswrapper[2579]: I1104 02:31:55.317575 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Nov 04 02:31:55.317868 ip-10-0-134-125 kubenswrapper[2579]: I1104 02:31:55.317854 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Nov 04 02:31:55.318033 ip-10-0-134-125 kubenswrapper[2579]: I1104 02:31:55.318018 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-gjd4x\"" Nov 04 02:31:55.319829 ip-10-0-134-125 kubenswrapper[2579]: I1104 02:31:55.319816 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-lg9c8\"" Nov 04 02:31:55.319978 ip-10-0-134-125 kubenswrapper[2579]: I1104 02:31:55.319963 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Nov 04 02:31:55.320024 ip-10-0-134-125 kubenswrapper[2579]: I1104 02:31:55.320000 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Nov 04 02:31:55.320066 ip-10-0-134-125 kubenswrapper[2579]: I1104 02:31:55.320056 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Nov 04 02:31:55.321760 ip-10-0-134-125 kubenswrapper[2579]: I1104 02:31:55.321747 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-rgzgm\"" Nov 04 02:31:55.321810 ip-10-0-134-125 kubenswrapper[2579]: I1104 02:31:55.321763 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Nov 04 02:31:55.358889 ip-10-0-134-125 kubenswrapper[2579]: I1104 02:31:55.358754 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Nov 04 02:31:55.359436 ip-10-0-134-125 kubenswrapper[2579]: I1104 02:31:55.359413 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Nov 04 02:31:55.359528 ip-10-0-134-125 kubenswrapper[2579]: I1104 02:31:55.359445 2579 reflector.go:446] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Nov 04 02:31:55.359528 ip-10-0-134-125 kubenswrapper[2579]: I1104 02:31:55.359502 2579 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-ldsmh" Nov 04 02:31:55.359817 ip-10-0-134-125 kubenswrapper[2579]: I1104 02:31:55.359801 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Nov 04 02:31:55.359892 ip-10-0-134-125 kubenswrapper[2579]: I1104 02:31:55.359836 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Nov 04 02:31:55.359892 ip-10-0-134-125 kubenswrapper[2579]: I1104 02:31:55.359865 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Nov 04 02:31:55.359892 ip-10-0-134-125 kubenswrapper[2579]: I1104 02:31:55.359869 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Nov 04 02:31:55.362534 ip-10-0-134-125 kubenswrapper[2579]: I1104 02:31:55.362458 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-4v2qx\"" Nov 04 02:31:55.362534 ip-10-0-134-125 kubenswrapper[2579]: I1104 02:31:55.362470 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-fm8qv\"" Nov 04 02:31:55.362534 ip-10-0-134-125 kubenswrapper[2579]: I1104 02:31:55.362514 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Nov 04 02:31:55.362534 ip-10-0-134-125 kubenswrapper[2579]: I1104 02:31:55.362525 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Nov 04 02:31:55.362720 ip-10-0-134-125 kubenswrapper[2579]: I1104 02:31:55.362539 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Nov 04 02:31:55.362822 ip-10-0-134-125 kubenswrapper[2579]: I1104 02:31:55.362805 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent-dockercfg-jmwv6\"" Nov 04 02:31:55.363831 ip-10-0-134-125 kubenswrapper[2579]: I1104 02:31:55.363816 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Nov 04 02:31:55.363917 ip-10-0-134-125 kubenswrapper[2579]: I1104 02:31:55.363837 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Nov 04 02:31:55.363989 ip-10-0-134-125 kubenswrapper[2579]: I1104 02:31:55.363973 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Nov 04 02:31:55.363989 ip-10-0-134-125 kubenswrapper[2579]: I1104 02:31:55.363979 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Nov 04 02:31:55.364059 ip-10-0-134-125 kubenswrapper[2579]: I1104 02:31:55.363982 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-6l7l7\"" Nov 04 02:31:55.370237 ip-10-0-134-125 kubenswrapper[2579]: I1104 02:31:55.370218 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Nov 04 02:31:55.370320 ip-10-0-134-125 kubenswrapper[2579]: I1104 02:31:55.370249 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Nov 04 02:31:55.417032 ip-10-0-134-125 kubenswrapper[2579]: I1104 02:31:55.417000 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/ffbb2d83-5a93-4a5b-bce3-e40afb8f58ea-run-systemd\") pod \"ovnkube-node-vwcx8\" (UID: \"ffbb2d83-5a93-4a5b-bce3-e40afb8f58ea\") " pod="openshift-ovn-kubernetes/ovnkube-node-vwcx8" Nov 04 02:31:55.417032 ip-10-0-134-125 kubenswrapper[2579]: I1104 02:31:55.417032 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ffbb2d83-5a93-4a5b-bce3-e40afb8f58ea-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-vwcx8\" (UID: \"ffbb2d83-5a93-4a5b-bce3-e40afb8f58ea\") " pod="openshift-ovn-kubernetes/ovnkube-node-vwcx8" Nov 04 02:31:55.417204 ip-10-0-134-125 kubenswrapper[2579]: I1104 02:31:55.417057 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/3984ce7b-96c8-464a-aa14-28487cc5b373-host-run-k8s-cni-cncf-io\") pod \"multus-tglvr\" (UID: \"3984ce7b-96c8-464a-aa14-28487cc5b373\") " pod="openshift-multus/multus-tglvr" Nov 04 02:31:55.417204 ip-10-0-134-125 kubenswrapper[2579]: I1104 02:31:55.417084 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/5590efb1-88b8-41ad-b0ee-37b2ff9442de-system-cni-dir\") pod \"multus-additional-cni-plugins-sgx95\" (UID: \"5590efb1-88b8-41ad-b0ee-37b2ff9442de\") " pod="openshift-multus/multus-additional-cni-plugins-sgx95" Nov 04 02:31:55.417204 ip-10-0-134-125 kubenswrapper[2579]: I1104 02:31:55.417106 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/3984ce7b-96c8-464a-aa14-28487cc5b373-host-run-netns\") pod \"multus-tglvr\" (UID: \"3984ce7b-96c8-464a-aa14-28487cc5b373\") " pod="openshift-multus/multus-tglvr" Nov 04 02:31:55.417204 ip-10-0-134-125 kubenswrapper[2579]: I1104 02:31:55.417129 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/3984ce7b-96c8-464a-aa14-28487cc5b373-host-var-lib-kubelet\") pod \"multus-tglvr\" (UID: \"3984ce7b-96c8-464a-aa14-28487cc5b373\") " pod="openshift-multus/multus-tglvr" Nov 04 02:31:55.417204 ip-10-0-134-125 kubenswrapper[2579]: I1104 02:31:55.417119 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ffbb2d83-5a93-4a5b-bce3-e40afb8f58ea-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-vwcx8\" (UID: \"ffbb2d83-5a93-4a5b-bce3-e40afb8f58ea\") " pod="openshift-ovn-kubernetes/ovnkube-node-vwcx8" Nov 04 02:31:55.417204 ip-10-0-134-125 kubenswrapper[2579]: I1104 02:31:55.417146 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b4snb\" (UniqueName: \"kubernetes.io/projected/f58e4ac8-c132-4716-b6c0-8d8ddb9c19e5-kube-api-access-b4snb\") pod \"aws-ebs-csi-driver-node-6mvxj\" (UID: \"f58e4ac8-c132-4716-b6c0-8d8ddb9c19e5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6mvxj" Nov 04 02:31:55.417204 ip-10-0-134-125 kubenswrapper[2579]: I1104 02:31:55.417145 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/5590efb1-88b8-41ad-b0ee-37b2ff9442de-system-cni-dir\") pod \"multus-additional-cni-plugins-sgx95\" (UID: \"5590efb1-88b8-41ad-b0ee-37b2ff9442de\") " pod="openshift-multus/multus-additional-cni-plugins-sgx95" Nov 04 02:31:55.417204 ip-10-0-134-125 kubenswrapper[2579]: I1104 02:31:55.417161 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/cb1915a0-5505-4f32-8f79-2f638465e404-etc-sysctl-conf\") pod \"tuned-ncfkk\" (UID: \"cb1915a0-5505-4f32-8f79-2f638465e404\") " pod="openshift-cluster-node-tuning-operator/tuned-ncfkk" Nov 04 02:31:55.417204 ip-10-0-134-125 kubenswrapper[2579]: I1104 02:31:55.417145 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/3984ce7b-96c8-464a-aa14-28487cc5b373-host-run-k8s-cni-cncf-io\") pod \"multus-tglvr\" (UID: \"3984ce7b-96c8-464a-aa14-28487cc5b373\") " pod="openshift-multus/multus-tglvr" Nov 04 02:31:55.417204 ip-10-0-134-125 kubenswrapper[2579]: I1104 02:31:55.417105 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/ffbb2d83-5a93-4a5b-bce3-e40afb8f58ea-run-systemd\") pod \"ovnkube-node-vwcx8\" (UID: \"ffbb2d83-5a93-4a5b-bce3-e40afb8f58ea\") " pod="openshift-ovn-kubernetes/ovnkube-node-vwcx8" Nov 04 02:31:55.417204 ip-10-0-134-125 kubenswrapper[2579]: I1104 02:31:55.417189 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/3984ce7b-96c8-464a-aa14-28487cc5b373-host-run-netns\") pod \"multus-tglvr\" (UID: \"3984ce7b-96c8-464a-aa14-28487cc5b373\") " pod="openshift-multus/multus-tglvr" Nov 04 02:31:55.417204 ip-10-0-134-125 kubenswrapper[2579]: I1104 02:31:55.417198 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/3984ce7b-96c8-464a-aa14-28487cc5b373-host-var-lib-kubelet\") pod \"multus-tglvr\" (UID: \"3984ce7b-96c8-464a-aa14-28487cc5b373\") " pod="openshift-multus/multus-tglvr" Nov 04 02:31:55.417204 ip-10-0-134-125 kubenswrapper[2579]: I1104 02:31:55.417204 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/8b030f29-dfa6-42b1-81a7-07b539ad9aa9-hosts-file\") pod \"node-resolver-kzzwk\" (UID: \"8b030f29-dfa6-42b1-81a7-07b539ad9aa9\") " pod="openshift-dns/node-resolver-kzzwk" Nov 04 02:31:55.417838 ip-10-0-134-125 kubenswrapper[2579]: I1104 02:31:55.417235 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/3984ce7b-96c8-464a-aa14-28487cc5b373-host-var-lib-cni-bin\") pod \"multus-tglvr\" (UID: \"3984ce7b-96c8-464a-aa14-28487cc5b373\") " pod="openshift-multus/multus-tglvr" Nov 04 02:31:55.417838 ip-10-0-134-125 kubenswrapper[2579]: I1104 02:31:55.417246 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/8b030f29-dfa6-42b1-81a7-07b539ad9aa9-hosts-file\") pod \"node-resolver-kzzwk\" (UID: \"8b030f29-dfa6-42b1-81a7-07b539ad9aa9\") " pod="openshift-dns/node-resolver-kzzwk" Nov 04 02:31:55.417838 ip-10-0-134-125 kubenswrapper[2579]: I1104 02:31:55.417255 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/cb1915a0-5505-4f32-8f79-2f638465e404-etc-sysctl-d\") pod \"tuned-ncfkk\" (UID: \"cb1915a0-5505-4f32-8f79-2f638465e404\") " pod="openshift-cluster-node-tuning-operator/tuned-ncfkk" Nov 04 02:31:55.417838 ip-10-0-134-125 kubenswrapper[2579]: I1104 02:31:55.417285 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/cb1915a0-5505-4f32-8f79-2f638465e404-host\") pod \"tuned-ncfkk\" (UID: \"cb1915a0-5505-4f32-8f79-2f638465e404\") " pod="openshift-cluster-node-tuning-operator/tuned-ncfkk" Nov 04 02:31:55.417838 ip-10-0-134-125 kubenswrapper[2579]: I1104 02:31:55.417287 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/3984ce7b-96c8-464a-aa14-28487cc5b373-host-var-lib-cni-bin\") pod \"multus-tglvr\" (UID: \"3984ce7b-96c8-464a-aa14-28487cc5b373\") " pod="openshift-multus/multus-tglvr" Nov 04 02:31:55.417838 ip-10-0-134-125 kubenswrapper[2579]: I1104 02:31:55.417301 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/cb1915a0-5505-4f32-8f79-2f638465e404-etc-sysctl-conf\") pod \"tuned-ncfkk\" (UID: \"cb1915a0-5505-4f32-8f79-2f638465e404\") " pod="openshift-cluster-node-tuning-operator/tuned-ncfkk" Nov 04 02:31:55.417838 ip-10-0-134-125 kubenswrapper[2579]: I1104 02:31:55.417307 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/cb1915a0-5505-4f32-8f79-2f638465e404-etc-tuned\") pod \"tuned-ncfkk\" (UID: \"cb1915a0-5505-4f32-8f79-2f638465e404\") " pod="openshift-cluster-node-tuning-operator/tuned-ncfkk" Nov 04 02:31:55.417838 ip-10-0-134-125 kubenswrapper[2579]: I1104 02:31:55.417338 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4rpxv\" (UniqueName: \"kubernetes.io/projected/5590efb1-88b8-41ad-b0ee-37b2ff9442de-kube-api-access-4rpxv\") pod \"multus-additional-cni-plugins-sgx95\" (UID: \"5590efb1-88b8-41ad-b0ee-37b2ff9442de\") " pod="openshift-multus/multus-additional-cni-plugins-sgx95" Nov 04 02:31:55.417838 ip-10-0-134-125 kubenswrapper[2579]: I1104 02:31:55.417344 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/cb1915a0-5505-4f32-8f79-2f638465e404-host\") pod \"tuned-ncfkk\" (UID: \"cb1915a0-5505-4f32-8f79-2f638465e404\") " pod="openshift-cluster-node-tuning-operator/tuned-ncfkk" Nov 04 02:31:55.417838 ip-10-0-134-125 kubenswrapper[2579]: I1104 02:31:55.417369 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/ffbb2d83-5a93-4a5b-bce3-e40afb8f58ea-host-cni-bin\") pod \"ovnkube-node-vwcx8\" (UID: \"ffbb2d83-5a93-4a5b-bce3-e40afb8f58ea\") " pod="openshift-ovn-kubernetes/ovnkube-node-vwcx8" Nov 04 02:31:55.417838 ip-10-0-134-125 kubenswrapper[2579]: I1104 02:31:55.417370 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/cb1915a0-5505-4f32-8f79-2f638465e404-etc-sysctl-d\") pod \"tuned-ncfkk\" (UID: \"cb1915a0-5505-4f32-8f79-2f638465e404\") " pod="openshift-cluster-node-tuning-operator/tuned-ncfkk" Nov 04 02:31:55.417838 ip-10-0-134-125 kubenswrapper[2579]: I1104 02:31:55.417404 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/ffbb2d83-5a93-4a5b-bce3-e40afb8f58ea-host-cni-bin\") pod \"ovnkube-node-vwcx8\" (UID: \"ffbb2d83-5a93-4a5b-bce3-e40afb8f58ea\") " pod="openshift-ovn-kubernetes/ovnkube-node-vwcx8" Nov 04 02:31:55.417838 ip-10-0-134-125 kubenswrapper[2579]: I1104 02:31:55.417515 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/3984ce7b-96c8-464a-aa14-28487cc5b373-system-cni-dir\") pod \"multus-tglvr\" (UID: \"3984ce7b-96c8-464a-aa14-28487cc5b373\") " pod="openshift-multus/multus-tglvr" Nov 04 02:31:55.417838 ip-10-0-134-125 kubenswrapper[2579]: I1104 02:31:55.417548 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-946v4\" (UniqueName: \"kubernetes.io/projected/7cb29949-bfe4-47a2-bc17-115a8d4d895d-kube-api-access-946v4\") pod \"node-ca-2h96g\" (UID: \"7cb29949-bfe4-47a2-bc17-115a8d4d895d\") " pod="openshift-image-registry/node-ca-2h96g" Nov 04 02:31:55.417838 ip-10-0-134-125 kubenswrapper[2579]: I1104 02:31:55.417597 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/5590efb1-88b8-41ad-b0ee-37b2ff9442de-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-sgx95\" (UID: \"5590efb1-88b8-41ad-b0ee-37b2ff9442de\") " pod="openshift-multus/multus-additional-cni-plugins-sgx95" Nov 04 02:31:55.417838 ip-10-0-134-125 kubenswrapper[2579]: I1104 02:31:55.417602 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/3984ce7b-96c8-464a-aa14-28487cc5b373-system-cni-dir\") pod \"multus-tglvr\" (UID: \"3984ce7b-96c8-464a-aa14-28487cc5b373\") " pod="openshift-multus/multus-tglvr" Nov 04 02:31:55.417838 ip-10-0-134-125 kubenswrapper[2579]: I1104 02:31:55.417611 2579 swap_util.go:78] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Nov 04 02:31:55.417838 ip-10-0-134-125 kubenswrapper[2579]: I1104 02:31:55.417623 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/ffbb2d83-5a93-4a5b-bce3-e40afb8f58ea-etc-openvswitch\") pod \"ovnkube-node-vwcx8\" (UID: \"ffbb2d83-5a93-4a5b-bce3-e40afb8f58ea\") " pod="openshift-ovn-kubernetes/ovnkube-node-vwcx8" Nov 04 02:31:55.418642 ip-10-0-134-125 kubenswrapper[2579]: I1104 02:31:55.417658 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/ffbb2d83-5a93-4a5b-bce3-e40afb8f58ea-node-log\") pod \"ovnkube-node-vwcx8\" (UID: \"ffbb2d83-5a93-4a5b-bce3-e40afb8f58ea\") " pod="openshift-ovn-kubernetes/ovnkube-node-vwcx8" Nov 04 02:31:55.418642 ip-10-0-134-125 kubenswrapper[2579]: I1104 02:31:55.417705 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/3984ce7b-96c8-464a-aa14-28487cc5b373-host-var-lib-cni-multus\") pod \"multus-tglvr\" (UID: \"3984ce7b-96c8-464a-aa14-28487cc5b373\") " pod="openshift-multus/multus-tglvr" Nov 04 02:31:55.418642 ip-10-0-134-125 kubenswrapper[2579]: I1104 02:31:55.417710 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/ffbb2d83-5a93-4a5b-bce3-e40afb8f58ea-etc-openvswitch\") pod \"ovnkube-node-vwcx8\" (UID: \"ffbb2d83-5a93-4a5b-bce3-e40afb8f58ea\") " pod="openshift-ovn-kubernetes/ovnkube-node-vwcx8" Nov 04 02:31:55.418642 ip-10-0-134-125 kubenswrapper[2579]: I1104 02:31:55.417727 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/ffbb2d83-5a93-4a5b-bce3-e40afb8f58ea-node-log\") pod \"ovnkube-node-vwcx8\" (UID: \"ffbb2d83-5a93-4a5b-bce3-e40afb8f58ea\") " pod="openshift-ovn-kubernetes/ovnkube-node-vwcx8" Nov 04 02:31:55.418642 ip-10-0-134-125 kubenswrapper[2579]: I1104 02:31:55.417740 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/cb1915a0-5505-4f32-8f79-2f638465e404-etc-kubernetes\") pod \"tuned-ncfkk\" (UID: \"cb1915a0-5505-4f32-8f79-2f638465e404\") " pod="openshift-cluster-node-tuning-operator/tuned-ncfkk" Nov 04 02:31:55.418642 ip-10-0-134-125 kubenswrapper[2579]: I1104 02:31:55.417753 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/3984ce7b-96c8-464a-aa14-28487cc5b373-host-var-lib-cni-multus\") pod \"multus-tglvr\" (UID: \"3984ce7b-96c8-464a-aa14-28487cc5b373\") " pod="openshift-multus/multus-tglvr" Nov 04 02:31:55.418642 ip-10-0-134-125 kubenswrapper[2579]: I1104 02:31:55.417759 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n6mc7\" (UniqueName: \"kubernetes.io/projected/cb1915a0-5505-4f32-8f79-2f638465e404-kube-api-access-n6mc7\") pod \"tuned-ncfkk\" (UID: \"cb1915a0-5505-4f32-8f79-2f638465e404\") " pod="openshift-cluster-node-tuning-operator/tuned-ncfkk" Nov 04 02:31:55.418642 ip-10-0-134-125 kubenswrapper[2579]: I1104 02:31:55.417789 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/3984ce7b-96c8-464a-aa14-28487cc5b373-multus-conf-dir\") pod \"multus-tglvr\" (UID: \"3984ce7b-96c8-464a-aa14-28487cc5b373\") " pod="openshift-multus/multus-tglvr" Nov 04 02:31:55.418642 ip-10-0-134-125 kubenswrapper[2579]: I1104 02:31:55.417812 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/3984ce7b-96c8-464a-aa14-28487cc5b373-multus-daemon-config\") pod \"multus-tglvr\" (UID: \"3984ce7b-96c8-464a-aa14-28487cc5b373\") " pod="openshift-multus/multus-tglvr" Nov 04 02:31:55.418642 ip-10-0-134-125 kubenswrapper[2579]: I1104 02:31:55.417825 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/cb1915a0-5505-4f32-8f79-2f638465e404-etc-kubernetes\") pod \"tuned-ncfkk\" (UID: \"cb1915a0-5505-4f32-8f79-2f638465e404\") " pod="openshift-cluster-node-tuning-operator/tuned-ncfkk" Nov 04 02:31:55.418642 ip-10-0-134-125 kubenswrapper[2579]: I1104 02:31:55.417855 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/3984ce7b-96c8-464a-aa14-28487cc5b373-multus-conf-dir\") pod \"multus-tglvr\" (UID: \"3984ce7b-96c8-464a-aa14-28487cc5b373\") " pod="openshift-multus/multus-tglvr" Nov 04 02:31:55.418642 ip-10-0-134-125 kubenswrapper[2579]: I1104 02:31:55.417836 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b37e5c97-77c6-4bb4-a9e4-e1ae27484457-metrics-certs\") pod \"network-metrics-daemon-bdcds\" (UID: \"b37e5c97-77c6-4bb4-a9e4-e1ae27484457\") " pod="openshift-multus/network-metrics-daemon-bdcds" Nov 04 02:31:55.418642 ip-10-0-134-125 kubenswrapper[2579]: I1104 02:31:55.417903 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zpbsm\" (UniqueName: \"kubernetes.io/projected/b37e5c97-77c6-4bb4-a9e4-e1ae27484457-kube-api-access-zpbsm\") pod \"network-metrics-daemon-bdcds\" (UID: \"b37e5c97-77c6-4bb4-a9e4-e1ae27484457\") " pod="openshift-multus/network-metrics-daemon-bdcds" Nov 04 02:31:55.418642 ip-10-0-134-125 kubenswrapper[2579]: E1104 02:31:55.417912 2579 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Nov 04 02:31:55.418642 ip-10-0-134-125 kubenswrapper[2579]: I1104 02:31:55.417959 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/5b77e30b-cb0c-49c1-80b9-330a138dbfb7-konnectivity-ca\") pod \"konnectivity-agent-wcp7z\" (UID: \"5b77e30b-cb0c-49c1-80b9-330a138dbfb7\") " pod="kube-system/konnectivity-agent-wcp7z" Nov 04 02:31:55.418642 ip-10-0-134-125 kubenswrapper[2579]: E1104 02:31:55.417994 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b37e5c97-77c6-4bb4-a9e4-e1ae27484457-metrics-certs podName:b37e5c97-77c6-4bb4-a9e4-e1ae27484457 nodeName:}" failed. No retries permitted until 2026-11-04 02:31:55.917953088 +0000 UTC m=+2.273060186 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/b37e5c97-77c6-4bb4-a9e4-e1ae27484457-metrics-certs") pod "network-metrics-daemon-bdcds" (UID: "b37e5c97-77c6-4bb4-a9e4-e1ae27484457") : object "openshift-multus"/"metrics-daemon-secret" not registered Nov 04 02:31:55.418642 ip-10-0-134-125 kubenswrapper[2579]: I1104 02:31:55.418021 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/f58e4ac8-c132-4716-b6c0-8d8ddb9c19e5-registration-dir\") pod \"aws-ebs-csi-driver-node-6mvxj\" (UID: \"f58e4ac8-c132-4716-b6c0-8d8ddb9c19e5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6mvxj" Nov 04 02:31:55.419236 ip-10-0-134-125 kubenswrapper[2579]: I1104 02:31:55.418048 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/cb1915a0-5505-4f32-8f79-2f638465e404-sys\") pod \"tuned-ncfkk\" (UID: \"cb1915a0-5505-4f32-8f79-2f638465e404\") " pod="openshift-cluster-node-tuning-operator/tuned-ncfkk" Nov 04 02:31:55.419236 ip-10-0-134-125 kubenswrapper[2579]: I1104 02:31:55.418071 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/ffbb2d83-5a93-4a5b-bce3-e40afb8f58ea-run-ovn\") pod \"ovnkube-node-vwcx8\" (UID: \"ffbb2d83-5a93-4a5b-bce3-e40afb8f58ea\") " pod="openshift-ovn-kubernetes/ovnkube-node-vwcx8" Nov 04 02:31:55.419236 ip-10-0-134-125 kubenswrapper[2579]: I1104 02:31:55.418095 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/ffbb2d83-5a93-4a5b-bce3-e40afb8f58ea-ovn-node-metrics-cert\") pod \"ovnkube-node-vwcx8\" (UID: \"ffbb2d83-5a93-4a5b-bce3-e40afb8f58ea\") " pod="openshift-ovn-kubernetes/ovnkube-node-vwcx8" Nov 04 02:31:55.419236 ip-10-0-134-125 kubenswrapper[2579]: I1104 02:31:55.418116 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/f58e4ac8-c132-4716-b6c0-8d8ddb9c19e5-sys-fs\") pod \"aws-ebs-csi-driver-node-6mvxj\" (UID: \"f58e4ac8-c132-4716-b6c0-8d8ddb9c19e5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6mvxj" Nov 04 02:31:55.419236 ip-10-0-134-125 kubenswrapper[2579]: I1104 02:31:55.418135 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/ffbb2d83-5a93-4a5b-bce3-e40afb8f58ea-var-lib-openvswitch\") pod \"ovnkube-node-vwcx8\" (UID: \"ffbb2d83-5a93-4a5b-bce3-e40afb8f58ea\") " pod="openshift-ovn-kubernetes/ovnkube-node-vwcx8" Nov 04 02:31:55.419236 ip-10-0-134-125 kubenswrapper[2579]: I1104 02:31:55.418158 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ffbb2d83-5a93-4a5b-bce3-e40afb8f58ea-host-run-ovn-kubernetes\") pod \"ovnkube-node-vwcx8\" (UID: \"ffbb2d83-5a93-4a5b-bce3-e40afb8f58ea\") " pod="openshift-ovn-kubernetes/ovnkube-node-vwcx8" Nov 04 02:31:55.419236 ip-10-0-134-125 kubenswrapper[2579]: I1104 02:31:55.418179 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/cb1915a0-5505-4f32-8f79-2f638465e404-sys\") pod \"tuned-ncfkk\" (UID: \"cb1915a0-5505-4f32-8f79-2f638465e404\") " pod="openshift-cluster-node-tuning-operator/tuned-ncfkk" Nov 04 02:31:55.419236 ip-10-0-134-125 kubenswrapper[2579]: I1104 02:31:55.418307 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/3a7ab1db-ad88-40d9-a7e9-b511bbd4584c-dbus\") pod \"global-pull-secret-syncer-krvqk\" (UID: \"3a7ab1db-ad88-40d9-a7e9-b511bbd4584c\") " pod="kube-system/global-pull-secret-syncer-krvqk" Nov 04 02:31:55.419236 ip-10-0-134-125 kubenswrapper[2579]: I1104 02:31:55.418336 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/5590efb1-88b8-41ad-b0ee-37b2ff9442de-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-sgx95\" (UID: \"5590efb1-88b8-41ad-b0ee-37b2ff9442de\") " pod="openshift-multus/multus-additional-cni-plugins-sgx95" Nov 04 02:31:55.419236 ip-10-0-134-125 kubenswrapper[2579]: I1104 02:31:55.418350 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/ffbb2d83-5a93-4a5b-bce3-e40afb8f58ea-run-ovn\") pod \"ovnkube-node-vwcx8\" (UID: \"ffbb2d83-5a93-4a5b-bce3-e40afb8f58ea\") " pod="openshift-ovn-kubernetes/ovnkube-node-vwcx8" Nov 04 02:31:55.419236 ip-10-0-134-125 kubenswrapper[2579]: I1104 02:31:55.418182 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/3a7ab1db-ad88-40d9-a7e9-b511bbd4584c-dbus\") pod \"global-pull-secret-syncer-krvqk\" (UID: \"3a7ab1db-ad88-40d9-a7e9-b511bbd4584c\") " pod="kube-system/global-pull-secret-syncer-krvqk" Nov 04 02:31:55.419236 ip-10-0-134-125 kubenswrapper[2579]: I1104 02:31:55.418402 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/ffbb2d83-5a93-4a5b-bce3-e40afb8f58ea-host-run-netns\") pod \"ovnkube-node-vwcx8\" (UID: \"ffbb2d83-5a93-4a5b-bce3-e40afb8f58ea\") " pod="openshift-ovn-kubernetes/ovnkube-node-vwcx8" Nov 04 02:31:55.419236 ip-10-0-134-125 kubenswrapper[2579]: I1104 02:31:55.418406 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/f58e4ac8-c132-4716-b6c0-8d8ddb9c19e5-registration-dir\") pod \"aws-ebs-csi-driver-node-6mvxj\" (UID: \"f58e4ac8-c132-4716-b6c0-8d8ddb9c19e5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6mvxj" Nov 04 02:31:55.419236 ip-10-0-134-125 kubenswrapper[2579]: I1104 02:31:55.418420 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/3984ce7b-96c8-464a-aa14-28487cc5b373-multus-daemon-config\") pod \"multus-tglvr\" (UID: \"3984ce7b-96c8-464a-aa14-28487cc5b373\") " pod="openshift-multus/multus-tglvr" Nov 04 02:31:55.419236 ip-10-0-134-125 kubenswrapper[2579]: I1104 02:31:55.418423 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tp4jl\" (UniqueName: \"kubernetes.io/projected/ffbb2d83-5a93-4a5b-bce3-e40afb8f58ea-kube-api-access-tp4jl\") pod \"ovnkube-node-vwcx8\" (UID: \"ffbb2d83-5a93-4a5b-bce3-e40afb8f58ea\") " pod="openshift-ovn-kubernetes/ovnkube-node-vwcx8" Nov 04 02:31:55.419236 ip-10-0-134-125 kubenswrapper[2579]: I1104 02:31:55.418459 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/ffbb2d83-5a93-4a5b-bce3-e40afb8f58ea-host-run-netns\") pod \"ovnkube-node-vwcx8\" (UID: \"ffbb2d83-5a93-4a5b-bce3-e40afb8f58ea\") " pod="openshift-ovn-kubernetes/ovnkube-node-vwcx8" Nov 04 02:31:55.419236 ip-10-0-134-125 kubenswrapper[2579]: I1104 02:31:55.418472 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/3984ce7b-96c8-464a-aa14-28487cc5b373-cni-binary-copy\") pod \"multus-tglvr\" (UID: \"3984ce7b-96c8-464a-aa14-28487cc5b373\") " pod="openshift-multus/multus-tglvr" Nov 04 02:31:55.419727 ip-10-0-134-125 kubenswrapper[2579]: I1104 02:31:55.418509 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/ffbb2d83-5a93-4a5b-bce3-e40afb8f58ea-var-lib-openvswitch\") pod \"ovnkube-node-vwcx8\" (UID: \"ffbb2d83-5a93-4a5b-bce3-e40afb8f58ea\") " pod="openshift-ovn-kubernetes/ovnkube-node-vwcx8" Nov 04 02:31:55.419727 ip-10-0-134-125 kubenswrapper[2579]: I1104 02:31:55.418540 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-stzcf\" (UniqueName: \"kubernetes.io/projected/524e33f8-723b-4868-a4f5-5d7eb92a5fe3-kube-api-access-stzcf\") pod \"iptables-alerter-sx4w7\" (UID: \"524e33f8-723b-4868-a4f5-5d7eb92a5fe3\") " pod="openshift-network-operator/iptables-alerter-sx4w7" Nov 04 02:31:55.419727 ip-10-0-134-125 kubenswrapper[2579]: I1104 02:31:55.418583 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/f58e4ac8-c132-4716-b6c0-8d8ddb9c19e5-kubelet-dir\") pod \"aws-ebs-csi-driver-node-6mvxj\" (UID: \"f58e4ac8-c132-4716-b6c0-8d8ddb9c19e5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6mvxj" Nov 04 02:31:55.419727 ip-10-0-134-125 kubenswrapper[2579]: I1104 02:31:55.418609 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/7cb29949-bfe4-47a2-bc17-115a8d4d895d-serviceca\") pod \"node-ca-2h96g\" (UID: \"7cb29949-bfe4-47a2-bc17-115a8d4d895d\") " pod="openshift-image-registry/node-ca-2h96g" Nov 04 02:31:55.419727 ip-10-0-134-125 kubenswrapper[2579]: I1104 02:31:55.418611 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ffbb2d83-5a93-4a5b-bce3-e40afb8f58ea-host-run-ovn-kubernetes\") pod \"ovnkube-node-vwcx8\" (UID: \"ffbb2d83-5a93-4a5b-bce3-e40afb8f58ea\") " pod="openshift-ovn-kubernetes/ovnkube-node-vwcx8" Nov 04 02:31:55.419727 ip-10-0-134-125 kubenswrapper[2579]: I1104 02:31:55.418697 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/5590efb1-88b8-41ad-b0ee-37b2ff9442de-tuning-conf-dir\") pod \"multus-additional-cni-plugins-sgx95\" (UID: \"5590efb1-88b8-41ad-b0ee-37b2ff9442de\") " pod="openshift-multus/multus-additional-cni-plugins-sgx95" Nov 04 02:31:55.419727 ip-10-0-134-125 kubenswrapper[2579]: I1104 02:31:55.418725 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/3984ce7b-96c8-464a-aa14-28487cc5b373-cnibin\") pod \"multus-tglvr\" (UID: \"3984ce7b-96c8-464a-aa14-28487cc5b373\") " pod="openshift-multus/multus-tglvr" Nov 04 02:31:55.419727 ip-10-0-134-125 kubenswrapper[2579]: I1104 02:31:55.418726 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/5b77e30b-cb0c-49c1-80b9-330a138dbfb7-konnectivity-ca\") pod \"konnectivity-agent-wcp7z\" (UID: \"5b77e30b-cb0c-49c1-80b9-330a138dbfb7\") " pod="kube-system/konnectivity-agent-wcp7z" Nov 04 02:31:55.419727 ip-10-0-134-125 kubenswrapper[2579]: I1104 02:31:55.418750 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-plfvd\" (UniqueName: \"kubernetes.io/projected/3984ce7b-96c8-464a-aa14-28487cc5b373-kube-api-access-plfvd\") pod \"multus-tglvr\" (UID: \"3984ce7b-96c8-464a-aa14-28487cc5b373\") " pod="openshift-multus/multus-tglvr" Nov 04 02:31:55.419727 ip-10-0-134-125 kubenswrapper[2579]: I1104 02:31:55.418773 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/524e33f8-723b-4868-a4f5-5d7eb92a5fe3-iptables-alerter-script\") pod \"iptables-alerter-sx4w7\" (UID: \"524e33f8-723b-4868-a4f5-5d7eb92a5fe3\") " pod="openshift-network-operator/iptables-alerter-sx4w7" Nov 04 02:31:55.419727 ip-10-0-134-125 kubenswrapper[2579]: I1104 02:31:55.418796 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/cb1915a0-5505-4f32-8f79-2f638465e404-etc-modprobe-d\") pod \"tuned-ncfkk\" (UID: \"cb1915a0-5505-4f32-8f79-2f638465e404\") " pod="openshift-cluster-node-tuning-operator/tuned-ncfkk" Nov 04 02:31:55.419727 ip-10-0-134-125 kubenswrapper[2579]: I1104 02:31:55.418818 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8b030f29-dfa6-42b1-81a7-07b539ad9aa9-tmp-dir\") pod \"node-resolver-kzzwk\" (UID: \"8b030f29-dfa6-42b1-81a7-07b539ad9aa9\") " pod="openshift-dns/node-resolver-kzzwk" Nov 04 02:31:55.419727 ip-10-0-134-125 kubenswrapper[2579]: I1104 02:31:55.418840 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/ffbb2d83-5a93-4a5b-bce3-e40afb8f58ea-systemd-units\") pod \"ovnkube-node-vwcx8\" (UID: \"ffbb2d83-5a93-4a5b-bce3-e40afb8f58ea\") " pod="openshift-ovn-kubernetes/ovnkube-node-vwcx8" Nov 04 02:31:55.419727 ip-10-0-134-125 kubenswrapper[2579]: I1104 02:31:55.418863 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/ffbb2d83-5a93-4a5b-bce3-e40afb8f58ea-env-overrides\") pod \"ovnkube-node-vwcx8\" (UID: \"ffbb2d83-5a93-4a5b-bce3-e40afb8f58ea\") " pod="openshift-ovn-kubernetes/ovnkube-node-vwcx8" Nov 04 02:31:55.419727 ip-10-0-134-125 kubenswrapper[2579]: I1104 02:31:55.418889 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/3984ce7b-96c8-464a-aa14-28487cc5b373-multus-cni-dir\") pod \"multus-tglvr\" (UID: \"3984ce7b-96c8-464a-aa14-28487cc5b373\") " pod="openshift-multus/multus-tglvr" Nov 04 02:31:55.419727 ip-10-0-134-125 kubenswrapper[2579]: I1104 02:31:55.418905 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/5590efb1-88b8-41ad-b0ee-37b2ff9442de-tuning-conf-dir\") pod \"multus-additional-cni-plugins-sgx95\" (UID: \"5590efb1-88b8-41ad-b0ee-37b2ff9442de\") " pod="openshift-multus/multus-additional-cni-plugins-sgx95" Nov 04 02:31:55.419727 ip-10-0-134-125 kubenswrapper[2579]: I1104 02:31:55.418914 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/ffbb2d83-5a93-4a5b-bce3-e40afb8f58ea-host-kubelet\") pod \"ovnkube-node-vwcx8\" (UID: \"ffbb2d83-5a93-4a5b-bce3-e40afb8f58ea\") " pod="openshift-ovn-kubernetes/ovnkube-node-vwcx8" Nov 04 02:31:55.420497 ip-10-0-134-125 kubenswrapper[2579]: I1104 02:31:55.418939 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/ffbb2d83-5a93-4a5b-bce3-e40afb8f58ea-host-slash\") pod \"ovnkube-node-vwcx8\" (UID: \"ffbb2d83-5a93-4a5b-bce3-e40afb8f58ea\") " pod="openshift-ovn-kubernetes/ovnkube-node-vwcx8" Nov 04 02:31:55.420497 ip-10-0-134-125 kubenswrapper[2579]: I1104 02:31:55.418961 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/f58e4ac8-c132-4716-b6c0-8d8ddb9c19e5-kubelet-dir\") pod \"aws-ebs-csi-driver-node-6mvxj\" (UID: \"f58e4ac8-c132-4716-b6c0-8d8ddb9c19e5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6mvxj" Nov 04 02:31:55.420497 ip-10-0-134-125 kubenswrapper[2579]: I1104 02:31:55.418971 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/ffbb2d83-5a93-4a5b-bce3-e40afb8f58ea-run-openvswitch\") pod \"ovnkube-node-vwcx8\" (UID: \"ffbb2d83-5a93-4a5b-bce3-e40afb8f58ea\") " pod="openshift-ovn-kubernetes/ovnkube-node-vwcx8" Nov 04 02:31:55.420497 ip-10-0-134-125 kubenswrapper[2579]: I1104 02:31:55.419000 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/3984ce7b-96c8-464a-aa14-28487cc5b373-etc-kubernetes\") pod \"multus-tglvr\" (UID: \"3984ce7b-96c8-464a-aa14-28487cc5b373\") " pod="openshift-multus/multus-tglvr" Nov 04 02:31:55.420497 ip-10-0-134-125 kubenswrapper[2579]: I1104 02:31:55.419012 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/f58e4ac8-c132-4716-b6c0-8d8ddb9c19e5-sys-fs\") pod \"aws-ebs-csi-driver-node-6mvxj\" (UID: \"f58e4ac8-c132-4716-b6c0-8d8ddb9c19e5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6mvxj" Nov 04 02:31:55.420497 ip-10-0-134-125 kubenswrapper[2579]: I1104 02:31:55.419025 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/524e33f8-723b-4868-a4f5-5d7eb92a5fe3-host-slash\") pod \"iptables-alerter-sx4w7\" (UID: \"524e33f8-723b-4868-a4f5-5d7eb92a5fe3\") " pod="openshift-network-operator/iptables-alerter-sx4w7" Nov 04 02:31:55.420497 ip-10-0-134-125 kubenswrapper[2579]: I1104 02:31:55.419051 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/f58e4ac8-c132-4716-b6c0-8d8ddb9c19e5-device-dir\") pod \"aws-ebs-csi-driver-node-6mvxj\" (UID: \"f58e4ac8-c132-4716-b6c0-8d8ddb9c19e5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6mvxj" Nov 04 02:31:55.420497 ip-10-0-134-125 kubenswrapper[2579]: I1104 02:31:55.419076 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/3984ce7b-96c8-464a-aa14-28487cc5b373-cni-binary-copy\") pod \"multus-tglvr\" (UID: \"3984ce7b-96c8-464a-aa14-28487cc5b373\") " pod="openshift-multus/multus-tglvr" Nov 04 02:31:55.420497 ip-10-0-134-125 kubenswrapper[2579]: I1104 02:31:55.419111 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/ffbb2d83-5a93-4a5b-bce3-e40afb8f58ea-host-slash\") pod \"ovnkube-node-vwcx8\" (UID: \"ffbb2d83-5a93-4a5b-bce3-e40afb8f58ea\") " pod="openshift-ovn-kubernetes/ovnkube-node-vwcx8" Nov 04 02:31:55.420497 ip-10-0-134-125 kubenswrapper[2579]: I1104 02:31:55.419119 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/524e33f8-723b-4868-a4f5-5d7eb92a5fe3-host-slash\") pod \"iptables-alerter-sx4w7\" (UID: \"524e33f8-723b-4868-a4f5-5d7eb92a5fe3\") " pod="openshift-network-operator/iptables-alerter-sx4w7" Nov 04 02:31:55.420497 ip-10-0-134-125 kubenswrapper[2579]: I1104 02:31:55.419130 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/ffbb2d83-5a93-4a5b-bce3-e40afb8f58ea-run-openvswitch\") pod \"ovnkube-node-vwcx8\" (UID: \"ffbb2d83-5a93-4a5b-bce3-e40afb8f58ea\") " pod="openshift-ovn-kubernetes/ovnkube-node-vwcx8" Nov 04 02:31:55.420497 ip-10-0-134-125 kubenswrapper[2579]: I1104 02:31:55.419178 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/cb1915a0-5505-4f32-8f79-2f638465e404-etc-modprobe-d\") pod \"tuned-ncfkk\" (UID: \"cb1915a0-5505-4f32-8f79-2f638465e404\") " pod="openshift-cluster-node-tuning-operator/tuned-ncfkk" Nov 04 02:31:55.420497 ip-10-0-134-125 kubenswrapper[2579]: I1104 02:31:55.419348 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/7cb29949-bfe4-47a2-bc17-115a8d4d895d-serviceca\") pod \"node-ca-2h96g\" (UID: \"7cb29949-bfe4-47a2-bc17-115a8d4d895d\") " pod="openshift-image-registry/node-ca-2h96g" Nov 04 02:31:55.420497 ip-10-0-134-125 kubenswrapper[2579]: I1104 02:31:55.419368 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8b030f29-dfa6-42b1-81a7-07b539ad9aa9-tmp-dir\") pod \"node-resolver-kzzwk\" (UID: \"8b030f29-dfa6-42b1-81a7-07b539ad9aa9\") " pod="openshift-dns/node-resolver-kzzwk" Nov 04 02:31:55.420497 ip-10-0-134-125 kubenswrapper[2579]: I1104 02:31:55.419408 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/ffbb2d83-5a93-4a5b-bce3-e40afb8f58ea-systemd-units\") pod \"ovnkube-node-vwcx8\" (UID: \"ffbb2d83-5a93-4a5b-bce3-e40afb8f58ea\") " pod="openshift-ovn-kubernetes/ovnkube-node-vwcx8" Nov 04 02:31:55.420497 ip-10-0-134-125 kubenswrapper[2579]: I1104 02:31:55.419416 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/ffbb2d83-5a93-4a5b-bce3-e40afb8f58ea-host-kubelet\") pod \"ovnkube-node-vwcx8\" (UID: \"ffbb2d83-5a93-4a5b-bce3-e40afb8f58ea\") " pod="openshift-ovn-kubernetes/ovnkube-node-vwcx8" Nov 04 02:31:55.420497 ip-10-0-134-125 kubenswrapper[2579]: I1104 02:31:55.419451 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/f58e4ac8-c132-4716-b6c0-8d8ddb9c19e5-device-dir\") pod \"aws-ebs-csi-driver-node-6mvxj\" (UID: \"f58e4ac8-c132-4716-b6c0-8d8ddb9c19e5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6mvxj" Nov 04 02:31:55.420497 ip-10-0-134-125 kubenswrapper[2579]: I1104 02:31:55.419453 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/3984ce7b-96c8-464a-aa14-28487cc5b373-etc-kubernetes\") pod \"multus-tglvr\" (UID: \"3984ce7b-96c8-464a-aa14-28487cc5b373\") " pod="openshift-multus/multus-tglvr" Nov 04 02:31:55.421282 ip-10-0-134-125 kubenswrapper[2579]: I1104 02:31:55.419490 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/3984ce7b-96c8-464a-aa14-28487cc5b373-cnibin\") pod \"multus-tglvr\" (UID: \"3984ce7b-96c8-464a-aa14-28487cc5b373\") " pod="openshift-multus/multus-tglvr" Nov 04 02:31:55.421282 ip-10-0-134-125 kubenswrapper[2579]: I1104 02:31:55.419512 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/3984ce7b-96c8-464a-aa14-28487cc5b373-multus-cni-dir\") pod \"multus-tglvr\" (UID: \"3984ce7b-96c8-464a-aa14-28487cc5b373\") " pod="openshift-multus/multus-tglvr" Nov 04 02:31:55.421282 ip-10-0-134-125 kubenswrapper[2579]: I1104 02:31:55.419530 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/ffbb2d83-5a93-4a5b-bce3-e40afb8f58ea-host-cni-netd\") pod \"ovnkube-node-vwcx8\" (UID: \"ffbb2d83-5a93-4a5b-bce3-e40afb8f58ea\") " pod="openshift-ovn-kubernetes/ovnkube-node-vwcx8" Nov 04 02:31:55.421282 ip-10-0-134-125 kubenswrapper[2579]: I1104 02:31:55.419556 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/ffbb2d83-5a93-4a5b-bce3-e40afb8f58ea-ovnkube-config\") pod \"ovnkube-node-vwcx8\" (UID: \"ffbb2d83-5a93-4a5b-bce3-e40afb8f58ea\") " pod="openshift-ovn-kubernetes/ovnkube-node-vwcx8" Nov 04 02:31:55.421282 ip-10-0-134-125 kubenswrapper[2579]: I1104 02:31:55.419590 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/ffbb2d83-5a93-4a5b-bce3-e40afb8f58ea-host-cni-netd\") pod \"ovnkube-node-vwcx8\" (UID: \"ffbb2d83-5a93-4a5b-bce3-e40afb8f58ea\") " pod="openshift-ovn-kubernetes/ovnkube-node-vwcx8" Nov 04 02:31:55.421282 ip-10-0-134-125 kubenswrapper[2579]: I1104 02:31:55.419615 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/7cb29949-bfe4-47a2-bc17-115a8d4d895d-host\") pod \"node-ca-2h96g\" (UID: \"7cb29949-bfe4-47a2-bc17-115a8d4d895d\") " pod="openshift-image-registry/node-ca-2h96g" Nov 04 02:31:55.421282 ip-10-0-134-125 kubenswrapper[2579]: I1104 02:31:55.419633 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/3a7ab1db-ad88-40d9-a7e9-b511bbd4584c-original-pull-secret\") pod \"global-pull-secret-syncer-krvqk\" (UID: \"3a7ab1db-ad88-40d9-a7e9-b511bbd4584c\") " pod="kube-system/global-pull-secret-syncer-krvqk" Nov 04 02:31:55.421282 ip-10-0-134-125 kubenswrapper[2579]: I1104 02:31:55.419692 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/7cb29949-bfe4-47a2-bc17-115a8d4d895d-host\") pod \"node-ca-2h96g\" (UID: \"7cb29949-bfe4-47a2-bc17-115a8d4d895d\") " pod="openshift-image-registry/node-ca-2h96g" Nov 04 02:31:55.421282 ip-10-0-134-125 kubenswrapper[2579]: I1104 02:31:55.419704 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/cb1915a0-5505-4f32-8f79-2f638465e404-var-lib-kubelet\") pod \"tuned-ncfkk\" (UID: \"cb1915a0-5505-4f32-8f79-2f638465e404\") " pod="openshift-cluster-node-tuning-operator/tuned-ncfkk" Nov 04 02:31:55.421282 ip-10-0-134-125 kubenswrapper[2579]: I1104 02:31:55.419739 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/f58e4ac8-c132-4716-b6c0-8d8ddb9c19e5-etc-selinux\") pod \"aws-ebs-csi-driver-node-6mvxj\" (UID: \"f58e4ac8-c132-4716-b6c0-8d8ddb9c19e5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6mvxj" Nov 04 02:31:55.421282 ip-10-0-134-125 kubenswrapper[2579]: I1104 02:31:55.419743 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/524e33f8-723b-4868-a4f5-5d7eb92a5fe3-iptables-alerter-script\") pod \"iptables-alerter-sx4w7\" (UID: \"524e33f8-723b-4868-a4f5-5d7eb92a5fe3\") " pod="openshift-network-operator/iptables-alerter-sx4w7" Nov 04 02:31:55.421282 ip-10-0-134-125 kubenswrapper[2579]: I1104 02:31:55.419749 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/ffbb2d83-5a93-4a5b-bce3-e40afb8f58ea-env-overrides\") pod \"ovnkube-node-vwcx8\" (UID: \"ffbb2d83-5a93-4a5b-bce3-e40afb8f58ea\") " pod="openshift-ovn-kubernetes/ovnkube-node-vwcx8" Nov 04 02:31:55.421282 ip-10-0-134-125 kubenswrapper[2579]: I1104 02:31:55.419767 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/cb1915a0-5505-4f32-8f79-2f638465e404-var-lib-kubelet\") pod \"tuned-ncfkk\" (UID: \"cb1915a0-5505-4f32-8f79-2f638465e404\") " pod="openshift-cluster-node-tuning-operator/tuned-ncfkk" Nov 04 02:31:55.421282 ip-10-0-134-125 kubenswrapper[2579]: E1104 02:31:55.419790 2579 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Nov 04 02:31:55.421282 ip-10-0-134-125 kubenswrapper[2579]: I1104 02:31:55.419801 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/5590efb1-88b8-41ad-b0ee-37b2ff9442de-cnibin\") pod \"multus-additional-cni-plugins-sgx95\" (UID: \"5590efb1-88b8-41ad-b0ee-37b2ff9442de\") " pod="openshift-multus/multus-additional-cni-plugins-sgx95" Nov 04 02:31:55.421282 ip-10-0-134-125 kubenswrapper[2579]: I1104 02:31:55.419838 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dddmg\" (UniqueName: \"kubernetes.io/projected/6e27a76f-fbdb-4ce6-9df6-659cab9ad06c-kube-api-access-dddmg\") pod \"network-check-target-zd8z9\" (UID: \"6e27a76f-fbdb-4ce6-9df6-659cab9ad06c\") " pod="openshift-network-diagnostics/network-check-target-zd8z9" Nov 04 02:31:55.421282 ip-10-0-134-125 kubenswrapper[2579]: I1104 02:31:55.419844 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/5590efb1-88b8-41ad-b0ee-37b2ff9442de-cnibin\") pod \"multus-additional-cni-plugins-sgx95\" (UID: \"5590efb1-88b8-41ad-b0ee-37b2ff9442de\") " pod="openshift-multus/multus-additional-cni-plugins-sgx95" Nov 04 02:31:55.421282 ip-10-0-134-125 kubenswrapper[2579]: I1104 02:31:55.419855 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/f58e4ac8-c132-4716-b6c0-8d8ddb9c19e5-etc-selinux\") pod \"aws-ebs-csi-driver-node-6mvxj\" (UID: \"f58e4ac8-c132-4716-b6c0-8d8ddb9c19e5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6mvxj" Nov 04 02:31:55.422165 ip-10-0-134-125 kubenswrapper[2579]: I1104 02:31:55.419863 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/3984ce7b-96c8-464a-aa14-28487cc5b373-hostroot\") pod \"multus-tglvr\" (UID: \"3984ce7b-96c8-464a-aa14-28487cc5b373\") " pod="openshift-multus/multus-tglvr" Nov 04 02:31:55.422165 ip-10-0-134-125 kubenswrapper[2579]: I1104 02:31:55.419887 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/cb1915a0-5505-4f32-8f79-2f638465e404-etc-sysconfig\") pod \"tuned-ncfkk\" (UID: \"cb1915a0-5505-4f32-8f79-2f638465e404\") " pod="openshift-cluster-node-tuning-operator/tuned-ncfkk" Nov 04 02:31:55.422165 ip-10-0-134-125 kubenswrapper[2579]: I1104 02:31:55.419897 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/3984ce7b-96c8-464a-aa14-28487cc5b373-hostroot\") pod \"multus-tglvr\" (UID: \"3984ce7b-96c8-464a-aa14-28487cc5b373\") " pod="openshift-multus/multus-tglvr" Nov 04 02:31:55.422165 ip-10-0-134-125 kubenswrapper[2579]: E1104 02:31:55.419960 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3a7ab1db-ad88-40d9-a7e9-b511bbd4584c-original-pull-secret podName:3a7ab1db-ad88-40d9-a7e9-b511bbd4584c nodeName:}" failed. No retries permitted until 2026-11-04 02:31:55.919938211 +0000 UTC m=+2.275045294 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/3a7ab1db-ad88-40d9-a7e9-b511bbd4584c-original-pull-secret") pod "global-pull-secret-syncer-krvqk" (UID: "3a7ab1db-ad88-40d9-a7e9-b511bbd4584c") : object "kube-system"/"original-pull-secret" not registered Nov 04 02:31:55.422165 ip-10-0-134-125 kubenswrapper[2579]: I1104 02:31:55.419978 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/ffbb2d83-5a93-4a5b-bce3-e40afb8f58ea-log-socket\") pod \"ovnkube-node-vwcx8\" (UID: \"ffbb2d83-5a93-4a5b-bce3-e40afb8f58ea\") " pod="openshift-ovn-kubernetes/ovnkube-node-vwcx8" Nov 04 02:31:55.422165 ip-10-0-134-125 kubenswrapper[2579]: I1104 02:31:55.420012 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/cb1915a0-5505-4f32-8f79-2f638465e404-etc-sysconfig\") pod \"tuned-ncfkk\" (UID: \"cb1915a0-5505-4f32-8f79-2f638465e404\") " pod="openshift-cluster-node-tuning-operator/tuned-ncfkk" Nov 04 02:31:55.422165 ip-10-0-134-125 kubenswrapper[2579]: I1104 02:31:55.420088 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/3984ce7b-96c8-464a-aa14-28487cc5b373-multus-socket-dir-parent\") pod \"multus-tglvr\" (UID: \"3984ce7b-96c8-464a-aa14-28487cc5b373\") " pod="openshift-multus/multus-tglvr" Nov 04 02:31:55.422165 ip-10-0-134-125 kubenswrapper[2579]: I1104 02:31:55.420127 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/ffbb2d83-5a93-4a5b-bce3-e40afb8f58ea-log-socket\") pod \"ovnkube-node-vwcx8\" (UID: \"ffbb2d83-5a93-4a5b-bce3-e40afb8f58ea\") " pod="openshift-ovn-kubernetes/ovnkube-node-vwcx8" Nov 04 02:31:55.422165 ip-10-0-134-125 kubenswrapper[2579]: I1104 02:31:55.420164 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/3a7ab1db-ad88-40d9-a7e9-b511bbd4584c-kubelet-config\") pod \"global-pull-secret-syncer-krvqk\" (UID: \"3a7ab1db-ad88-40d9-a7e9-b511bbd4584c\") " pod="kube-system/global-pull-secret-syncer-krvqk" Nov 04 02:31:55.422165 ip-10-0-134-125 kubenswrapper[2579]: I1104 02:31:55.420176 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/ffbb2d83-5a93-4a5b-bce3-e40afb8f58ea-ovnkube-config\") pod \"ovnkube-node-vwcx8\" (UID: \"ffbb2d83-5a93-4a5b-bce3-e40afb8f58ea\") " pod="openshift-ovn-kubernetes/ovnkube-node-vwcx8" Nov 04 02:31:55.422165 ip-10-0-134-125 kubenswrapper[2579]: I1104 02:31:55.420187 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/3984ce7b-96c8-464a-aa14-28487cc5b373-multus-socket-dir-parent\") pod \"multus-tglvr\" (UID: \"3984ce7b-96c8-464a-aa14-28487cc5b373\") " pod="openshift-multus/multus-tglvr" Nov 04 02:31:55.422165 ip-10-0-134-125 kubenswrapper[2579]: I1104 02:31:55.420251 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/cb1915a0-5505-4f32-8f79-2f638465e404-etc-systemd\") pod \"tuned-ncfkk\" (UID: \"cb1915a0-5505-4f32-8f79-2f638465e404\") " pod="openshift-cluster-node-tuning-operator/tuned-ncfkk" Nov 04 02:31:55.422165 ip-10-0-134-125 kubenswrapper[2579]: I1104 02:31:55.420247 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/3a7ab1db-ad88-40d9-a7e9-b511bbd4584c-kubelet-config\") pod \"global-pull-secret-syncer-krvqk\" (UID: \"3a7ab1db-ad88-40d9-a7e9-b511bbd4584c\") " pod="kube-system/global-pull-secret-syncer-krvqk" Nov 04 02:31:55.422165 ip-10-0-134-125 kubenswrapper[2579]: I1104 02:31:55.420287 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/5590efb1-88b8-41ad-b0ee-37b2ff9442de-cni-binary-copy\") pod \"multus-additional-cni-plugins-sgx95\" (UID: \"5590efb1-88b8-41ad-b0ee-37b2ff9442de\") " pod="openshift-multus/multus-additional-cni-plugins-sgx95" Nov 04 02:31:55.422165 ip-10-0-134-125 kubenswrapper[2579]: I1104 02:31:55.420313 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/cb1915a0-5505-4f32-8f79-2f638465e404-etc-systemd\") pod \"tuned-ncfkk\" (UID: \"cb1915a0-5505-4f32-8f79-2f638465e404\") " pod="openshift-cluster-node-tuning-operator/tuned-ncfkk" Nov 04 02:31:55.422165 ip-10-0-134-125 kubenswrapper[2579]: I1104 02:31:55.420369 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/ffbb2d83-5a93-4a5b-bce3-e40afb8f58ea-ovnkube-script-lib\") pod \"ovnkube-node-vwcx8\" (UID: \"ffbb2d83-5a93-4a5b-bce3-e40afb8f58ea\") " pod="openshift-ovn-kubernetes/ovnkube-node-vwcx8" Nov 04 02:31:55.422165 ip-10-0-134-125 kubenswrapper[2579]: I1104 02:31:55.420458 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/3984ce7b-96c8-464a-aa14-28487cc5b373-host-run-multus-certs\") pod \"multus-tglvr\" (UID: \"3984ce7b-96c8-464a-aa14-28487cc5b373\") " pod="openshift-multus/multus-tglvr" Nov 04 02:31:55.422631 ip-10-0-134-125 kubenswrapper[2579]: I1104 02:31:55.420498 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/3984ce7b-96c8-464a-aa14-28487cc5b373-host-run-multus-certs\") pod \"multus-tglvr\" (UID: \"3984ce7b-96c8-464a-aa14-28487cc5b373\") " pod="openshift-multus/multus-tglvr" Nov 04 02:31:55.422631 ip-10-0-134-125 kubenswrapper[2579]: I1104 02:31:55.420498 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/5b77e30b-cb0c-49c1-80b9-330a138dbfb7-agent-certs\") pod \"konnectivity-agent-wcp7z\" (UID: \"5b77e30b-cb0c-49c1-80b9-330a138dbfb7\") " pod="kube-system/konnectivity-agent-wcp7z" Nov 04 02:31:55.422631 ip-10-0-134-125 kubenswrapper[2579]: I1104 02:31:55.420558 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/f58e4ac8-c132-4716-b6c0-8d8ddb9c19e5-socket-dir\") pod \"aws-ebs-csi-driver-node-6mvxj\" (UID: \"f58e4ac8-c132-4716-b6c0-8d8ddb9c19e5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6mvxj" Nov 04 02:31:55.422631 ip-10-0-134-125 kubenswrapper[2579]: I1104 02:31:55.420585 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8ll77\" (UniqueName: \"kubernetes.io/projected/8b030f29-dfa6-42b1-81a7-07b539ad9aa9-kube-api-access-8ll77\") pod \"node-resolver-kzzwk\" (UID: \"8b030f29-dfa6-42b1-81a7-07b539ad9aa9\") " pod="openshift-dns/node-resolver-kzzwk" Nov 04 02:31:55.422631 ip-10-0-134-125 kubenswrapper[2579]: I1104 02:31:55.420610 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/cb1915a0-5505-4f32-8f79-2f638465e404-run\") pod \"tuned-ncfkk\" (UID: \"cb1915a0-5505-4f32-8f79-2f638465e404\") " pod="openshift-cluster-node-tuning-operator/tuned-ncfkk" Nov 04 02:31:55.422631 ip-10-0-134-125 kubenswrapper[2579]: I1104 02:31:55.420637 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/cb1915a0-5505-4f32-8f79-2f638465e404-lib-modules\") pod \"tuned-ncfkk\" (UID: \"cb1915a0-5505-4f32-8f79-2f638465e404\") " pod="openshift-cluster-node-tuning-operator/tuned-ncfkk" Nov 04 02:31:55.422631 ip-10-0-134-125 kubenswrapper[2579]: I1104 02:31:55.420660 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/cb1915a0-5505-4f32-8f79-2f638465e404-tmp\") pod \"tuned-ncfkk\" (UID: \"cb1915a0-5505-4f32-8f79-2f638465e404\") " pod="openshift-cluster-node-tuning-operator/tuned-ncfkk" Nov 04 02:31:55.422631 ip-10-0-134-125 kubenswrapper[2579]: I1104 02:31:55.420705 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/5590efb1-88b8-41ad-b0ee-37b2ff9442de-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-sgx95\" (UID: \"5590efb1-88b8-41ad-b0ee-37b2ff9442de\") " pod="openshift-multus/multus-additional-cni-plugins-sgx95" Nov 04 02:31:55.422631 ip-10-0-134-125 kubenswrapper[2579]: I1104 02:31:55.420869 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/cb1915a0-5505-4f32-8f79-2f638465e404-run\") pod \"tuned-ncfkk\" (UID: \"cb1915a0-5505-4f32-8f79-2f638465e404\") " pod="openshift-cluster-node-tuning-operator/tuned-ncfkk" Nov 04 02:31:55.422631 ip-10-0-134-125 kubenswrapper[2579]: I1104 02:31:55.420945 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/cb1915a0-5505-4f32-8f79-2f638465e404-lib-modules\") pod \"tuned-ncfkk\" (UID: \"cb1915a0-5505-4f32-8f79-2f638465e404\") " pod="openshift-cluster-node-tuning-operator/tuned-ncfkk" Nov 04 02:31:55.422631 ip-10-0-134-125 kubenswrapper[2579]: I1104 02:31:55.421417 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/ffbb2d83-5a93-4a5b-bce3-e40afb8f58ea-ovn-node-metrics-cert\") pod \"ovnkube-node-vwcx8\" (UID: \"ffbb2d83-5a93-4a5b-bce3-e40afb8f58ea\") " pod="openshift-ovn-kubernetes/ovnkube-node-vwcx8" Nov 04 02:31:55.422631 ip-10-0-134-125 kubenswrapper[2579]: I1104 02:31:55.421779 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/f58e4ac8-c132-4716-b6c0-8d8ddb9c19e5-socket-dir\") pod \"aws-ebs-csi-driver-node-6mvxj\" (UID: \"f58e4ac8-c132-4716-b6c0-8d8ddb9c19e5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6mvxj" Nov 04 02:31:55.422631 ip-10-0-134-125 kubenswrapper[2579]: I1104 02:31:55.421832 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/ffbb2d83-5a93-4a5b-bce3-e40afb8f58ea-ovnkube-script-lib\") pod \"ovnkube-node-vwcx8\" (UID: \"ffbb2d83-5a93-4a5b-bce3-e40afb8f58ea\") " pod="openshift-ovn-kubernetes/ovnkube-node-vwcx8" Nov 04 02:31:55.422631 ip-10-0-134-125 kubenswrapper[2579]: I1104 02:31:55.421825 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/5590efb1-88b8-41ad-b0ee-37b2ff9442de-cni-binary-copy\") pod \"multus-additional-cni-plugins-sgx95\" (UID: \"5590efb1-88b8-41ad-b0ee-37b2ff9442de\") " pod="openshift-multus/multus-additional-cni-plugins-sgx95" Nov 04 02:31:55.422631 ip-10-0-134-125 kubenswrapper[2579]: I1104 02:31:55.421972 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/cb1915a0-5505-4f32-8f79-2f638465e404-etc-tuned\") pod \"tuned-ncfkk\" (UID: \"cb1915a0-5505-4f32-8f79-2f638465e404\") " pod="openshift-cluster-node-tuning-operator/tuned-ncfkk" Nov 04 02:31:55.422631 ip-10-0-134-125 kubenswrapper[2579]: I1104 02:31:55.422127 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/5590efb1-88b8-41ad-b0ee-37b2ff9442de-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-sgx95\" (UID: \"5590efb1-88b8-41ad-b0ee-37b2ff9442de\") " pod="openshift-multus/multus-additional-cni-plugins-sgx95" Nov 04 02:31:55.423224 ip-10-0-134-125 kubenswrapper[2579]: I1104 02:31:55.423209 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/5b77e30b-cb0c-49c1-80b9-330a138dbfb7-agent-certs\") pod \"konnectivity-agent-wcp7z\" (UID: \"5b77e30b-cb0c-49c1-80b9-330a138dbfb7\") " pod="kube-system/konnectivity-agent-wcp7z" Nov 04 02:31:55.423805 ip-10-0-134-125 kubenswrapper[2579]: I1104 02:31:55.423739 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/cb1915a0-5505-4f32-8f79-2f638465e404-tmp\") pod \"tuned-ncfkk\" (UID: \"cb1915a0-5505-4f32-8f79-2f638465e404\") " pod="openshift-cluster-node-tuning-operator/tuned-ncfkk" Nov 04 02:31:55.477997 ip-10-0-134-125 kubenswrapper[2579]: E1104 02:31:55.477815 2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Nov 04 02:31:55.477997 ip-10-0-134-125 kubenswrapper[2579]: E1104 02:31:55.477843 2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Nov 04 02:31:55.477997 ip-10-0-134-125 kubenswrapper[2579]: E1104 02:31:55.477857 2579 projected.go:196] Error preparing data for projected volume kube-api-access-dddmg for pod openshift-network-diagnostics/network-check-target-zd8z9: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 04 02:31:55.477997 ip-10-0-134-125 kubenswrapper[2579]: E1104 02:31:55.477923 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/6e27a76f-fbdb-4ce6-9df6-659cab9ad06c-kube-api-access-dddmg podName:6e27a76f-fbdb-4ce6-9df6-659cab9ad06c nodeName:}" failed. No retries permitted until 2026-11-04 02:31:55.977902755 +0000 UTC m=+2.333009854 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-dddmg" (UniqueName: "kubernetes.io/projected/6e27a76f-fbdb-4ce6-9df6-659cab9ad06c-kube-api-access-dddmg") pod "network-check-target-zd8z9" (UID: "6e27a76f-fbdb-4ce6-9df6-659cab9ad06c") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 04 02:31:55.478941 ip-10-0-134-125 kubenswrapper[2579]: I1104 02:31:55.478909 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n6mc7\" (UniqueName: \"kubernetes.io/projected/cb1915a0-5505-4f32-8f79-2f638465e404-kube-api-access-n6mc7\") pod \"tuned-ncfkk\" (UID: \"cb1915a0-5505-4f32-8f79-2f638465e404\") " pod="openshift-cluster-node-tuning-operator/tuned-ncfkk" Nov 04 02:31:55.478941 ip-10-0-134-125 kubenswrapper[2579]: I1104 02:31:55.478912 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zpbsm\" (UniqueName: \"kubernetes.io/projected/b37e5c97-77c6-4bb4-a9e4-e1ae27484457-kube-api-access-zpbsm\") pod \"network-metrics-daemon-bdcds\" (UID: \"b37e5c97-77c6-4bb4-a9e4-e1ae27484457\") " pod="openshift-multus/network-metrics-daemon-bdcds" Nov 04 02:31:55.479138 ip-10-0-134-125 kubenswrapper[2579]: I1104 02:31:55.479097 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-946v4\" (UniqueName: \"kubernetes.io/projected/7cb29949-bfe4-47a2-bc17-115a8d4d895d-kube-api-access-946v4\") pod \"node-ca-2h96g\" (UID: \"7cb29949-bfe4-47a2-bc17-115a8d4d895d\") " pod="openshift-image-registry/node-ca-2h96g" Nov 04 02:31:55.479489 ip-10-0-134-125 kubenswrapper[2579]: I1104 02:31:55.479432 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b4snb\" (UniqueName: \"kubernetes.io/projected/f58e4ac8-c132-4716-b6c0-8d8ddb9c19e5-kube-api-access-b4snb\") pod \"aws-ebs-csi-driver-node-6mvxj\" (UID: \"f58e4ac8-c132-4716-b6c0-8d8ddb9c19e5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6mvxj" Nov 04 02:31:55.479594 ip-10-0-134-125 kubenswrapper[2579]: I1104 02:31:55.479574 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tp4jl\" (UniqueName: \"kubernetes.io/projected/ffbb2d83-5a93-4a5b-bce3-e40afb8f58ea-kube-api-access-tp4jl\") pod \"ovnkube-node-vwcx8\" (UID: \"ffbb2d83-5a93-4a5b-bce3-e40afb8f58ea\") " pod="openshift-ovn-kubernetes/ovnkube-node-vwcx8" Nov 04 02:31:55.479828 ip-10-0-134-125 kubenswrapper[2579]: I1104 02:31:55.479810 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-plfvd\" (UniqueName: \"kubernetes.io/projected/3984ce7b-96c8-464a-aa14-28487cc5b373-kube-api-access-plfvd\") pod \"multus-tglvr\" (UID: \"3984ce7b-96c8-464a-aa14-28487cc5b373\") " pod="openshift-multus/multus-tglvr" Nov 04 02:31:55.480094 ip-10-0-134-125 kubenswrapper[2579]: I1104 02:31:55.480075 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-stzcf\" (UniqueName: \"kubernetes.io/projected/524e33f8-723b-4868-a4f5-5d7eb92a5fe3-kube-api-access-stzcf\") pod \"iptables-alerter-sx4w7\" (UID: \"524e33f8-723b-4868-a4f5-5d7eb92a5fe3\") " pod="openshift-network-operator/iptables-alerter-sx4w7" Nov 04 02:31:55.480245 ip-10-0-134-125 kubenswrapper[2579]: I1104 02:31:55.480225 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4rpxv\" (UniqueName: \"kubernetes.io/projected/5590efb1-88b8-41ad-b0ee-37b2ff9442de-kube-api-access-4rpxv\") pod \"multus-additional-cni-plugins-sgx95\" (UID: \"5590efb1-88b8-41ad-b0ee-37b2ff9442de\") " pod="openshift-multus/multus-additional-cni-plugins-sgx95" Nov 04 02:31:55.481796 ip-10-0-134-125 kubenswrapper[2579]: I1104 02:31:55.481775 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8ll77\" (UniqueName: \"kubernetes.io/projected/8b030f29-dfa6-42b1-81a7-07b539ad9aa9-kube-api-access-8ll77\") pod \"node-resolver-kzzwk\" (UID: \"8b030f29-dfa6-42b1-81a7-07b539ad9aa9\") " pod="openshift-dns/node-resolver-kzzwk" Nov 04 02:31:55.597990 ip-10-0-134-125 kubenswrapper[2579]: I1104 02:31:55.597954 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-kzzwk" Nov 04 02:31:55.616869 ip-10-0-134-125 kubenswrapper[2579]: I1104 02:31:55.616847 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-tglvr" Nov 04 02:31:55.628177 ip-10-0-134-125 kubenswrapper[2579]: I1104 02:31:55.628152 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-ncfkk" Nov 04 02:31:55.645249 ip-10-0-134-125 kubenswrapper[2579]: I1104 02:31:55.645225 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-2h96g" Nov 04 02:31:55.670873 ip-10-0-134-125 kubenswrapper[2579]: I1104 02:31:55.670839 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-sx4w7" Nov 04 02:31:55.673044 ip-10-0-134-125 kubenswrapper[2579]: I1104 02:31:55.673015 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-sgx95" Nov 04 02:31:55.685313 ip-10-0-134-125 kubenswrapper[2579]: W1104 02:31:55.685280 2579 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc7038986364b9e4776a5785da4dd2f01.slice/crio-e2d730f5206d875c1b306c4525077cc3151c5f42596a1e751c4a0de261d0d629 WatchSource:0}: Error finding container e2d730f5206d875c1b306c4525077cc3151c5f42596a1e751c4a0de261d0d629: Status 404 returned error can't find the container with id e2d730f5206d875c1b306c4525077cc3151c5f42596a1e751c4a0de261d0d629 Nov 04 02:31:55.685758 ip-10-0-134-125 kubenswrapper[2579]: W1104 02:31:55.685728 2579 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod524e33f8_723b_4868_a4f5_5d7eb92a5fe3.slice/crio-40c8e62d5261acd2a0d99afde27dd20c1b7bcb27efa2c4e4823b219a202588d6 WatchSource:0}: Error finding container 40c8e62d5261acd2a0d99afde27dd20c1b7bcb27efa2c4e4823b219a202588d6: Status 404 returned error can't find the container with id 40c8e62d5261acd2a0d99afde27dd20c1b7bcb27efa2c4e4823b219a202588d6 Nov 04 02:31:55.686713 ip-10-0-134-125 kubenswrapper[2579]: W1104 02:31:55.686687 2579 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podee7b213d76bf8a5295d150c84b46451e.slice/crio-e3f6ee0a8c70b4d6cb30ed7c29e4392e9078922367521917c71e40e38a5db2c3 WatchSource:0}: Error finding container e3f6ee0a8c70b4d6cb30ed7c29e4392e9078922367521917c71e40e38a5db2c3: Status 404 returned error can't find the container with id e3f6ee0a8c70b4d6cb30ed7c29e4392e9078922367521917c71e40e38a5db2c3 Nov 04 02:31:55.688279 ip-10-0-134-125 kubenswrapper[2579]: W1104 02:31:55.688254 2579 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7cb29949_bfe4_47a2_bc17_115a8d4d895d.slice/crio-687e7943a5e73801885e494b71552b0607afdb68a04c24bb344b9c0621fe2080 WatchSource:0}: Error finding container 687e7943a5e73801885e494b71552b0607afdb68a04c24bb344b9c0621fe2080: Status 404 returned error can't find the container with id 687e7943a5e73801885e494b71552b0607afdb68a04c24bb344b9c0621fe2080 Nov 04 02:31:55.688760 ip-10-0-134-125 kubenswrapper[2579]: W1104 02:31:55.688732 2579 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5590efb1_88b8_41ad_b0ee_37b2ff9442de.slice/crio-47b9f0efad770fcb435f4488c99815037d5feb96967427da3c63d396c4ec20da WatchSource:0}: Error finding container 47b9f0efad770fcb435f4488c99815037d5feb96967427da3c63d396c4ec20da: Status 404 returned error can't find the container with id 47b9f0efad770fcb435f4488c99815037d5feb96967427da3c63d396c4ec20da Nov 04 02:31:55.691602 ip-10-0-134-125 kubenswrapper[2579]: I1104 02:31:55.691431 2579 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 04 02:31:55.691805 ip-10-0-134-125 kubenswrapper[2579]: I1104 02:31:55.691787 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-wcp7z" Nov 04 02:31:55.698650 ip-10-0-134-125 kubenswrapper[2579]: I1104 02:31:55.698631 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-vwcx8" Nov 04 02:31:55.701815 ip-10-0-134-125 kubenswrapper[2579]: W1104 02:31:55.701793 2579 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5b77e30b_cb0c_49c1_80b9_330a138dbfb7.slice/crio-f8bda2cfe1f2e4c6f7d909b602917ba1869ed772ade09d26192081e4ec879d5c WatchSource:0}: Error finding container f8bda2cfe1f2e4c6f7d909b602917ba1869ed772ade09d26192081e4ec879d5c: Status 404 returned error can't find the container with id f8bda2cfe1f2e4c6f7d909b602917ba1869ed772ade09d26192081e4ec879d5c Nov 04 02:31:55.707618 ip-10-0-134-125 kubenswrapper[2579]: I1104 02:31:55.707597 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6mvxj" Nov 04 02:31:55.709144 ip-10-0-134-125 kubenswrapper[2579]: W1104 02:31:55.709124 2579 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podffbb2d83_5a93_4a5b_bce3_e40afb8f58ea.slice/crio-0a7d3eed97ff0f3d2ca2a3dee83c822cb30b9b1d44f646553cd7922c3a9541b9 WatchSource:0}: Error finding container 0a7d3eed97ff0f3d2ca2a3dee83c822cb30b9b1d44f646553cd7922c3a9541b9: Status 404 returned error can't find the container with id 0a7d3eed97ff0f3d2ca2a3dee83c822cb30b9b1d44f646553cd7922c3a9541b9 Nov 04 02:31:55.717257 ip-10-0-134-125 kubenswrapper[2579]: W1104 02:31:55.717236 2579 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf58e4ac8_c132_4716_b6c0_8d8ddb9c19e5.slice/crio-6d8c7e634cc46f6ff9bcbd86093ccd32f6171bc08ffac03df9cf46af38e95fca WatchSource:0}: Error finding container 6d8c7e634cc46f6ff9bcbd86093ccd32f6171bc08ffac03df9cf46af38e95fca: Status 404 returned error can't find the container with id 6d8c7e634cc46f6ff9bcbd86093ccd32f6171bc08ffac03df9cf46af38e95fca Nov 04 02:31:55.924279 ip-10-0-134-125 kubenswrapper[2579]: I1104 02:31:55.924168 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/3a7ab1db-ad88-40d9-a7e9-b511bbd4584c-original-pull-secret\") pod \"global-pull-secret-syncer-krvqk\" (UID: \"3a7ab1db-ad88-40d9-a7e9-b511bbd4584c\") " pod="kube-system/global-pull-secret-syncer-krvqk" Nov 04 02:31:55.924279 ip-10-0-134-125 kubenswrapper[2579]: I1104 02:31:55.924244 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b37e5c97-77c6-4bb4-a9e4-e1ae27484457-metrics-certs\") pod \"network-metrics-daemon-bdcds\" (UID: \"b37e5c97-77c6-4bb4-a9e4-e1ae27484457\") " pod="openshift-multus/network-metrics-daemon-bdcds" Nov 04 02:31:55.924470 ip-10-0-134-125 kubenswrapper[2579]: E1104 02:31:55.924326 2579 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Nov 04 02:31:55.924470 ip-10-0-134-125 kubenswrapper[2579]: E1104 02:31:55.924363 2579 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Nov 04 02:31:55.924470 ip-10-0-134-125 kubenswrapper[2579]: E1104 02:31:55.924392 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3a7ab1db-ad88-40d9-a7e9-b511bbd4584c-original-pull-secret podName:3a7ab1db-ad88-40d9-a7e9-b511bbd4584c nodeName:}" failed. No retries permitted until 2026-11-04 02:31:56.924376758 +0000 UTC m=+3.279483841 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/3a7ab1db-ad88-40d9-a7e9-b511bbd4584c-original-pull-secret") pod "global-pull-secret-syncer-krvqk" (UID: "3a7ab1db-ad88-40d9-a7e9-b511bbd4584c") : object "kube-system"/"original-pull-secret" not registered Nov 04 02:31:55.924470 ip-10-0-134-125 kubenswrapper[2579]: E1104 02:31:55.924410 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b37e5c97-77c6-4bb4-a9e4-e1ae27484457-metrics-certs podName:b37e5c97-77c6-4bb4-a9e4-e1ae27484457 nodeName:}" failed. No retries permitted until 2026-11-04 02:31:56.924398639 +0000 UTC m=+3.279505722 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/b37e5c97-77c6-4bb4-a9e4-e1ae27484457-metrics-certs") pod "network-metrics-daemon-bdcds" (UID: "b37e5c97-77c6-4bb4-a9e4-e1ae27484457") : object "openshift-multus"/"metrics-daemon-secret" not registered Nov 04 02:31:55.976182 ip-10-0-134-125 kubenswrapper[2579]: I1104 02:31:55.976149 2579 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161" Nov 04 02:31:56.025440 ip-10-0-134-125 kubenswrapper[2579]: I1104 02:31:56.025390 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dddmg\" (UniqueName: \"kubernetes.io/projected/6e27a76f-fbdb-4ce6-9df6-659cab9ad06c-kube-api-access-dddmg\") pod \"network-check-target-zd8z9\" (UID: \"6e27a76f-fbdb-4ce6-9df6-659cab9ad06c\") " pod="openshift-network-diagnostics/network-check-target-zd8z9" Nov 04 02:31:56.025599 ip-10-0-134-125 kubenswrapper[2579]: E1104 02:31:56.025564 2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Nov 04 02:31:56.025599 ip-10-0-134-125 kubenswrapper[2579]: E1104 02:31:56.025579 2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Nov 04 02:31:56.025599 ip-10-0-134-125 kubenswrapper[2579]: E1104 02:31:56.025589 2579 projected.go:196] Error preparing data for projected volume kube-api-access-dddmg for pod openshift-network-diagnostics/network-check-target-zd8z9: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 04 02:31:56.025725 ip-10-0-134-125 kubenswrapper[2579]: E1104 02:31:56.025641 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/6e27a76f-fbdb-4ce6-9df6-659cab9ad06c-kube-api-access-dddmg podName:6e27a76f-fbdb-4ce6-9df6-659cab9ad06c nodeName:}" failed. No retries permitted until 2026-11-04 02:31:57.025627044 +0000 UTC m=+3.380734127 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-dddmg" (UniqueName: "kubernetes.io/projected/6e27a76f-fbdb-4ce6-9df6-659cab9ad06c-kube-api-access-dddmg") pod "network-check-target-zd8z9" (UID: "6e27a76f-fbdb-4ce6-9df6-659cab9ad06c") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 04 02:31:56.153241 ip-10-0-134-125 kubenswrapper[2579]: I1104 02:31:56.153179 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-2h96g" event={"ID":"7cb29949-bfe4-47a2-bc17-115a8d4d895d","Type":"ContainerStarted","Data":"687e7943a5e73801885e494b71552b0607afdb68a04c24bb344b9c0621fe2080"} Nov 04 02:31:56.154228 ip-10-0-134-125 kubenswrapper[2579]: I1104 02:31:56.154207 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6mvxj" event={"ID":"f58e4ac8-c132-4716-b6c0-8d8ddb9c19e5","Type":"ContainerStarted","Data":"6d8c7e634cc46f6ff9bcbd86093ccd32f6171bc08ffac03df9cf46af38e95fca"} Nov 04 02:31:56.155338 ip-10-0-134-125 kubenswrapper[2579]: I1104 02:31:56.155316 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-vwcx8" event={"ID":"ffbb2d83-5a93-4a5b-bce3-e40afb8f58ea","Type":"ContainerStarted","Data":"0a7d3eed97ff0f3d2ca2a3dee83c822cb30b9b1d44f646553cd7922c3a9541b9"} Nov 04 02:31:56.156394 ip-10-0-134-125 kubenswrapper[2579]: I1104 02:31:56.156375 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-wcp7z" event={"ID":"5b77e30b-cb0c-49c1-80b9-330a138dbfb7","Type":"ContainerStarted","Data":"f8bda2cfe1f2e4c6f7d909b602917ba1869ed772ade09d26192081e4ec879d5c"} Nov 04 02:31:56.157480 ip-10-0-134-125 kubenswrapper[2579]: I1104 02:31:56.157461 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-sgx95" event={"ID":"5590efb1-88b8-41ad-b0ee-37b2ff9442de","Type":"ContainerStarted","Data":"47b9f0efad770fcb435f4488c99815037d5feb96967427da3c63d396c4ec20da"} Nov 04 02:31:56.158517 ip-10-0-134-125 kubenswrapper[2579]: I1104 02:31:56.158500 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-sx4w7" event={"ID":"524e33f8-723b-4868-a4f5-5d7eb92a5fe3","Type":"ContainerStarted","Data":"40c8e62d5261acd2a0d99afde27dd20c1b7bcb27efa2c4e4823b219a202588d6"} Nov 04 02:31:56.159554 ip-10-0-134-125 kubenswrapper[2579]: I1104 02:31:56.159534 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-125.us-west-1.compute.internal" event={"ID":"c7038986364b9e4776a5785da4dd2f01","Type":"ContainerStarted","Data":"e2d730f5206d875c1b306c4525077cc3151c5f42596a1e751c4a0de261d0d629"} Nov 04 02:31:56.160518 ip-10-0-134-125 kubenswrapper[2579]: I1104 02:31:56.160500 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-134-125.us-west-1.compute.internal" event={"ID":"ee7b213d76bf8a5295d150c84b46451e","Type":"ContainerStarted","Data":"e3f6ee0a8c70b4d6cb30ed7c29e4392e9078922367521917c71e40e38a5db2c3"} Nov 04 02:31:56.237354 ip-10-0-134-125 kubenswrapper[2579]: I1104 02:31:56.237287 2579 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161" Nov 04 02:31:56.296929 ip-10-0-134-125 kubenswrapper[2579]: I1104 02:31:56.296896 2579 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161" Nov 04 02:31:56.362918 ip-10-0-134-125 kubenswrapper[2579]: I1104 02:31:56.362851 2579 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-11-03 02:26:55 +0000 UTC" deadline="2028-07-15 01:25:58.683898606 +0000 UTC" Nov 04 02:31:56.362918 ip-10-0-134-125 kubenswrapper[2579]: I1104 02:31:56.362907 2579 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14854h54m2.32099558s" Nov 04 02:31:56.498292 ip-10-0-134-125 kubenswrapper[2579]: W1104 02:31:56.498260 2579 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3984ce7b_96c8_464a_aa14_28487cc5b373.slice/crio-616dfef053572a7bd0df5fb2615000eae7a447ca31cea0519cec9e737621ef3e WatchSource:0}: Error finding container 616dfef053572a7bd0df5fb2615000eae7a447ca31cea0519cec9e737621ef3e: Status 404 returned error can't find the container with id 616dfef053572a7bd0df5fb2615000eae7a447ca31cea0519cec9e737621ef3e Nov 04 02:31:56.517742 ip-10-0-134-125 kubenswrapper[2579]: W1104 02:31:56.517715 2579 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcb1915a0_5505_4f32_8f79_2f638465e404.slice/crio-fb0ffaf40f2947cc0200cd4207a7baca6b7aabe2e4bed1249f35ec5abaabf265 WatchSource:0}: Error finding container fb0ffaf40f2947cc0200cd4207a7baca6b7aabe2e4bed1249f35ec5abaabf265: Status 404 returned error can't find the container with id fb0ffaf40f2947cc0200cd4207a7baca6b7aabe2e4bed1249f35ec5abaabf265 Nov 04 02:31:56.634289 ip-10-0-134-125 kubenswrapper[2579]: W1104 02:31:56.634260 2579 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8b030f29_dfa6_42b1_81a7_07b539ad9aa9.slice/crio-bb86d16e06bc56cd7e46a4be8f1338e7c72edac6c81b7c132d32bd3d4feb5415 WatchSource:0}: Error finding container bb86d16e06bc56cd7e46a4be8f1338e7c72edac6c81b7c132d32bd3d4feb5415: Status 404 returned error can't find the container with id bb86d16e06bc56cd7e46a4be8f1338e7c72edac6c81b7c132d32bd3d4feb5415 Nov 04 02:31:56.932087 ip-10-0-134-125 kubenswrapper[2579]: I1104 02:31:56.931999 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b37e5c97-77c6-4bb4-a9e4-e1ae27484457-metrics-certs\") pod \"network-metrics-daemon-bdcds\" (UID: \"b37e5c97-77c6-4bb4-a9e4-e1ae27484457\") " pod="openshift-multus/network-metrics-daemon-bdcds" Nov 04 02:31:56.932087 ip-10-0-134-125 kubenswrapper[2579]: I1104 02:31:56.932062 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/3a7ab1db-ad88-40d9-a7e9-b511bbd4584c-original-pull-secret\") pod \"global-pull-secret-syncer-krvqk\" (UID: \"3a7ab1db-ad88-40d9-a7e9-b511bbd4584c\") " pod="kube-system/global-pull-secret-syncer-krvqk" Nov 04 02:31:56.932344 ip-10-0-134-125 kubenswrapper[2579]: E1104 02:31:56.932218 2579 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Nov 04 02:31:56.932344 ip-10-0-134-125 kubenswrapper[2579]: E1104 02:31:56.932285 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3a7ab1db-ad88-40d9-a7e9-b511bbd4584c-original-pull-secret podName:3a7ab1db-ad88-40d9-a7e9-b511bbd4584c nodeName:}" failed. No retries permitted until 2026-11-04 02:31:58.932267347 +0000 UTC m=+5.287374442 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/3a7ab1db-ad88-40d9-a7e9-b511bbd4584c-original-pull-secret") pod "global-pull-secret-syncer-krvqk" (UID: "3a7ab1db-ad88-40d9-a7e9-b511bbd4584c") : object "kube-system"/"original-pull-secret" not registered Nov 04 02:31:56.932477 ip-10-0-134-125 kubenswrapper[2579]: E1104 02:31:56.932360 2579 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Nov 04 02:31:56.932477 ip-10-0-134-125 kubenswrapper[2579]: E1104 02:31:56.932399 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b37e5c97-77c6-4bb4-a9e4-e1ae27484457-metrics-certs podName:b37e5c97-77c6-4bb4-a9e4-e1ae27484457 nodeName:}" failed. No retries permitted until 2026-11-04 02:31:58.932386857 +0000 UTC m=+5.287493955 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/b37e5c97-77c6-4bb4-a9e4-e1ae27484457-metrics-certs") pod "network-metrics-daemon-bdcds" (UID: "b37e5c97-77c6-4bb4-a9e4-e1ae27484457") : object "openshift-multus"/"metrics-daemon-secret" not registered Nov 04 02:31:57.032786 ip-10-0-134-125 kubenswrapper[2579]: I1104 02:31:57.032745 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dddmg\" (UniqueName: \"kubernetes.io/projected/6e27a76f-fbdb-4ce6-9df6-659cab9ad06c-kube-api-access-dddmg\") pod \"network-check-target-zd8z9\" (UID: \"6e27a76f-fbdb-4ce6-9df6-659cab9ad06c\") " pod="openshift-network-diagnostics/network-check-target-zd8z9" Nov 04 02:31:57.033418 ip-10-0-134-125 kubenswrapper[2579]: E1104 02:31:57.032939 2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Nov 04 02:31:57.033418 ip-10-0-134-125 kubenswrapper[2579]: E1104 02:31:57.032964 2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Nov 04 02:31:57.033418 ip-10-0-134-125 kubenswrapper[2579]: E1104 02:31:57.032978 2579 projected.go:196] Error preparing data for projected volume kube-api-access-dddmg for pod openshift-network-diagnostics/network-check-target-zd8z9: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 04 02:31:57.033418 ip-10-0-134-125 kubenswrapper[2579]: E1104 02:31:57.033062 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/6e27a76f-fbdb-4ce6-9df6-659cab9ad06c-kube-api-access-dddmg podName:6e27a76f-fbdb-4ce6-9df6-659cab9ad06c nodeName:}" failed. No retries permitted until 2026-11-04 02:31:59.033025822 +0000 UTC m=+5.388132923 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-dddmg" (UniqueName: "kubernetes.io/projected/6e27a76f-fbdb-4ce6-9df6-659cab9ad06c-kube-api-access-dddmg") pod "network-check-target-zd8z9" (UID: "6e27a76f-fbdb-4ce6-9df6-659cab9ad06c") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 04 02:31:57.150316 ip-10-0-134-125 kubenswrapper[2579]: I1104 02:31:57.150277 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bdcds" Nov 04 02:31:57.150607 ip-10-0-134-125 kubenswrapper[2579]: E1104 02:31:57.150437 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-bdcds" podUID="b37e5c97-77c6-4bb4-a9e4-e1ae27484457" Nov 04 02:31:57.150607 ip-10-0-134-125 kubenswrapper[2579]: I1104 02:31:57.150537 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-krvqk" Nov 04 02:31:57.150775 ip-10-0-134-125 kubenswrapper[2579]: E1104 02:31:57.150669 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-krvqk" podUID="3a7ab1db-ad88-40d9-a7e9-b511bbd4584c" Nov 04 02:31:57.150775 ip-10-0-134-125 kubenswrapper[2579]: I1104 02:31:57.150705 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zd8z9" Nov 04 02:31:57.150892 ip-10-0-134-125 kubenswrapper[2579]: E1104 02:31:57.150796 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zd8z9" podUID="6e27a76f-fbdb-4ce6-9df6-659cab9ad06c" Nov 04 02:31:57.181339 ip-10-0-134-125 kubenswrapper[2579]: I1104 02:31:57.181298 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-kzzwk" event={"ID":"8b030f29-dfa6-42b1-81a7-07b539ad9aa9","Type":"ContainerStarted","Data":"bb86d16e06bc56cd7e46a4be8f1338e7c72edac6c81b7c132d32bd3d4feb5415"} Nov 04 02:31:57.195067 ip-10-0-134-125 kubenswrapper[2579]: I1104 02:31:57.194953 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-tglvr" event={"ID":"3984ce7b-96c8-464a-aa14-28487cc5b373","Type":"ContainerStarted","Data":"616dfef053572a7bd0df5fb2615000eae7a447ca31cea0519cec9e737621ef3e"} Nov 04 02:31:57.210961 ip-10-0-134-125 kubenswrapper[2579]: I1104 02:31:57.210920 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-ncfkk" event={"ID":"cb1915a0-5505-4f32-8f79-2f638465e404","Type":"ContainerStarted","Data":"fb0ffaf40f2947cc0200cd4207a7baca6b7aabe2e4bed1249f35ec5abaabf265"} Nov 04 02:31:57.363608 ip-10-0-134-125 kubenswrapper[2579]: I1104 02:31:57.363495 2579 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-11-03 02:26:55 +0000 UTC" deadline="2028-05-11 20:07:10.022099292 +0000 UTC" Nov 04 02:31:57.363608 ip-10-0-134-125 kubenswrapper[2579]: I1104 02:31:57.363538 2579 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13313h35m12.658566799s" Nov 04 02:31:58.948770 ip-10-0-134-125 kubenswrapper[2579]: I1104 02:31:58.948708 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b37e5c97-77c6-4bb4-a9e4-e1ae27484457-metrics-certs\") pod \"network-metrics-daemon-bdcds\" (UID: \"b37e5c97-77c6-4bb4-a9e4-e1ae27484457\") " pod="openshift-multus/network-metrics-daemon-bdcds" Nov 04 02:31:58.949368 ip-10-0-134-125 kubenswrapper[2579]: I1104 02:31:58.948802 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/3a7ab1db-ad88-40d9-a7e9-b511bbd4584c-original-pull-secret\") pod \"global-pull-secret-syncer-krvqk\" (UID: \"3a7ab1db-ad88-40d9-a7e9-b511bbd4584c\") " pod="kube-system/global-pull-secret-syncer-krvqk" Nov 04 02:31:58.949368 ip-10-0-134-125 kubenswrapper[2579]: E1104 02:31:58.948947 2579 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Nov 04 02:31:58.949368 ip-10-0-134-125 kubenswrapper[2579]: E1104 02:31:58.949010 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3a7ab1db-ad88-40d9-a7e9-b511bbd4584c-original-pull-secret podName:3a7ab1db-ad88-40d9-a7e9-b511bbd4584c nodeName:}" failed. No retries permitted until 2026-11-04 02:32:02.94899129 +0000 UTC m=+9.304098387 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/3a7ab1db-ad88-40d9-a7e9-b511bbd4584c-original-pull-secret") pod "global-pull-secret-syncer-krvqk" (UID: "3a7ab1db-ad88-40d9-a7e9-b511bbd4584c") : object "kube-system"/"original-pull-secret" not registered Nov 04 02:31:58.949368 ip-10-0-134-125 kubenswrapper[2579]: E1104 02:31:58.949074 2579 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Nov 04 02:31:58.949368 ip-10-0-134-125 kubenswrapper[2579]: E1104 02:31:58.949109 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b37e5c97-77c6-4bb4-a9e4-e1ae27484457-metrics-certs podName:b37e5c97-77c6-4bb4-a9e4-e1ae27484457 nodeName:}" failed. No retries permitted until 2026-11-04 02:32:02.949096836 +0000 UTC m=+9.304203935 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/b37e5c97-77c6-4bb4-a9e4-e1ae27484457-metrics-certs") pod "network-metrics-daemon-bdcds" (UID: "b37e5c97-77c6-4bb4-a9e4-e1ae27484457") : object "openshift-multus"/"metrics-daemon-secret" not registered Nov 04 02:31:59.049998 ip-10-0-134-125 kubenswrapper[2579]: I1104 02:31:59.049954 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dddmg\" (UniqueName: \"kubernetes.io/projected/6e27a76f-fbdb-4ce6-9df6-659cab9ad06c-kube-api-access-dddmg\") pod \"network-check-target-zd8z9\" (UID: \"6e27a76f-fbdb-4ce6-9df6-659cab9ad06c\") " pod="openshift-network-diagnostics/network-check-target-zd8z9" Nov 04 02:31:59.050198 ip-10-0-134-125 kubenswrapper[2579]: E1104 02:31:59.050133 2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Nov 04 02:31:59.050198 ip-10-0-134-125 kubenswrapper[2579]: E1104 02:31:59.050152 2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Nov 04 02:31:59.050198 ip-10-0-134-125 kubenswrapper[2579]: E1104 02:31:59.050164 2579 projected.go:196] Error preparing data for projected volume kube-api-access-dddmg for pod openshift-network-diagnostics/network-check-target-zd8z9: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 04 02:31:59.050377 ip-10-0-134-125 kubenswrapper[2579]: E1104 02:31:59.050223 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/6e27a76f-fbdb-4ce6-9df6-659cab9ad06c-kube-api-access-dddmg podName:6e27a76f-fbdb-4ce6-9df6-659cab9ad06c nodeName:}" failed. No retries permitted until 2026-11-04 02:32:03.050205073 +0000 UTC m=+9.405312159 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-dddmg" (UniqueName: "kubernetes.io/projected/6e27a76f-fbdb-4ce6-9df6-659cab9ad06c-kube-api-access-dddmg") pod "network-check-target-zd8z9" (UID: "6e27a76f-fbdb-4ce6-9df6-659cab9ad06c") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 04 02:31:59.149610 ip-10-0-134-125 kubenswrapper[2579]: I1104 02:31:59.149573 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bdcds" Nov 04 02:31:59.149821 ip-10-0-134-125 kubenswrapper[2579]: E1104 02:31:59.149738 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-bdcds" podUID="b37e5c97-77c6-4bb4-a9e4-e1ae27484457" Nov 04 02:31:59.149908 ip-10-0-134-125 kubenswrapper[2579]: I1104 02:31:59.149827 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-krvqk" Nov 04 02:31:59.149971 ip-10-0-134-125 kubenswrapper[2579]: E1104 02:31:59.149910 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-krvqk" podUID="3a7ab1db-ad88-40d9-a7e9-b511bbd4584c" Nov 04 02:31:59.150031 ip-10-0-134-125 kubenswrapper[2579]: I1104 02:31:59.149977 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zd8z9" Nov 04 02:31:59.150082 ip-10-0-134-125 kubenswrapper[2579]: E1104 02:31:59.150055 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zd8z9" podUID="6e27a76f-fbdb-4ce6-9df6-659cab9ad06c" Nov 04 02:32:00.237258 ip-10-0-134-125 kubenswrapper[2579]: I1104 02:32:00.236855 2579 generic.go:356] "Generic (PLEG): container finished" podID="5590efb1-88b8-41ad-b0ee-37b2ff9442de" containerID="21fb821c31c7d33fdcbecb34e143b49e3552980addd21d256bc02be70fc79d40" exitCode=0 Nov 04 02:32:00.237258 ip-10-0-134-125 kubenswrapper[2579]: I1104 02:32:00.236898 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-sgx95" event={"ID":"5590efb1-88b8-41ad-b0ee-37b2ff9442de","Type":"ContainerDied","Data":"21fb821c31c7d33fdcbecb34e143b49e3552980addd21d256bc02be70fc79d40"} Nov 04 02:32:01.150068 ip-10-0-134-125 kubenswrapper[2579]: I1104 02:32:01.149819 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-krvqk" Nov 04 02:32:01.150068 ip-10-0-134-125 kubenswrapper[2579]: I1104 02:32:01.149854 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bdcds" Nov 04 02:32:01.150068 ip-10-0-134-125 kubenswrapper[2579]: I1104 02:32:01.149821 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zd8z9" Nov 04 02:32:01.150068 ip-10-0-134-125 kubenswrapper[2579]: E1104 02:32:01.149968 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-krvqk" podUID="3a7ab1db-ad88-40d9-a7e9-b511bbd4584c" Nov 04 02:32:01.150068 ip-10-0-134-125 kubenswrapper[2579]: E1104 02:32:01.150004 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zd8z9" podUID="6e27a76f-fbdb-4ce6-9df6-659cab9ad06c" Nov 04 02:32:01.150436 ip-10-0-134-125 kubenswrapper[2579]: E1104 02:32:01.150104 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-bdcds" podUID="b37e5c97-77c6-4bb4-a9e4-e1ae27484457" Nov 04 02:32:01.242458 ip-10-0-134-125 kubenswrapper[2579]: I1104 02:32:01.241796 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-wcp7z" event={"ID":"5b77e30b-cb0c-49c1-80b9-330a138dbfb7","Type":"ContainerStarted","Data":"d19c1ebbf8c3560e7e2f50967a797371a1ccb2590651d286bb1c2517f45ade2b"} Nov 04 02:32:01.246322 ip-10-0-134-125 kubenswrapper[2579]: I1104 02:32:01.245639 2579 generic.go:356] "Generic (PLEG): container finished" podID="c7038986364b9e4776a5785da4dd2f01" containerID="4cf7fe820ecd1cd17891819005268af501eb1d83dc1a8304c2d6bff27e88bc6d" exitCode=0 Nov 04 02:32:01.246322 ip-10-0-134-125 kubenswrapper[2579]: I1104 02:32:01.245708 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-125.us-west-1.compute.internal" event={"ID":"c7038986364b9e4776a5785da4dd2f01","Type":"ContainerDied","Data":"4cf7fe820ecd1cd17891819005268af501eb1d83dc1a8304c2d6bff27e88bc6d"} Nov 04 02:32:01.359021 ip-10-0-134-125 kubenswrapper[2579]: I1104 02:32:01.358535 2579 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/konnectivity-agent-wcp7z" podStartSLOduration=2.596542886 podStartE2EDuration="7.358517023s" totalImagesPullingTime="4.761974137s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 02:31:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 02:32:01.357523091 +0000 UTC m=+7.712630194" watchObservedRunningTime="2026-11-04 02:32:01.358517023 +0000 UTC m=+7.713624127" Nov 04 02:32:02.261216 ip-10-0-134-125 kubenswrapper[2579]: I1104 02:32:02.261137 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-125.us-west-1.compute.internal" event={"ID":"c7038986364b9e4776a5785da4dd2f01","Type":"ContainerStarted","Data":"d3f240aff1ddde7aceaab32efe3b3fb9c5ea9ca055c94b8e5e5da75da0ea7d0b"} Nov 04 02:32:02.270992 ip-10-0-134-125 kubenswrapper[2579]: I1104 02:32:02.270902 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-134-125.us-west-1.compute.internal" event={"ID":"ee7b213d76bf8a5295d150c84b46451e","Type":"ContainerStarted","Data":"8013a1cf8342b8cb45922287bcb35dd722adf8508be17006cefe3b209b699b38"} Nov 04 02:32:02.377498 ip-10-0-134-125 kubenswrapper[2579]: I1104 02:32:02.377208 2579 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-125.us-west-1.compute.internal" podStartSLOduration=8.37719532 podStartE2EDuration="8.37719532s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-04 02:31:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 02:32:02.376905889 +0000 UTC m=+8.732012993" watchObservedRunningTime="2026-11-04 02:32:02.37719532 +0000 UTC m=+8.732302422" Nov 04 02:32:02.985986 ip-10-0-134-125 kubenswrapper[2579]: I1104 02:32:02.985399 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/3a7ab1db-ad88-40d9-a7e9-b511bbd4584c-original-pull-secret\") pod \"global-pull-secret-syncer-krvqk\" (UID: \"3a7ab1db-ad88-40d9-a7e9-b511bbd4584c\") " pod="kube-system/global-pull-secret-syncer-krvqk" Nov 04 02:32:02.985986 ip-10-0-134-125 kubenswrapper[2579]: I1104 02:32:02.985485 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b37e5c97-77c6-4bb4-a9e4-e1ae27484457-metrics-certs\") pod \"network-metrics-daemon-bdcds\" (UID: \"b37e5c97-77c6-4bb4-a9e4-e1ae27484457\") " pod="openshift-multus/network-metrics-daemon-bdcds" Nov 04 02:32:02.985986 ip-10-0-134-125 kubenswrapper[2579]: E1104 02:32:02.985622 2579 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Nov 04 02:32:02.985986 ip-10-0-134-125 kubenswrapper[2579]: E1104 02:32:02.985739 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b37e5c97-77c6-4bb4-a9e4-e1ae27484457-metrics-certs podName:b37e5c97-77c6-4bb4-a9e4-e1ae27484457 nodeName:}" failed. No retries permitted until 2026-11-04 02:32:10.985720856 +0000 UTC m=+17.340827940 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/b37e5c97-77c6-4bb4-a9e4-e1ae27484457-metrics-certs") pod "network-metrics-daemon-bdcds" (UID: "b37e5c97-77c6-4bb4-a9e4-e1ae27484457") : object "openshift-multus"/"metrics-daemon-secret" not registered Nov 04 02:32:02.985986 ip-10-0-134-125 kubenswrapper[2579]: E1104 02:32:02.985833 2579 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Nov 04 02:32:02.985986 ip-10-0-134-125 kubenswrapper[2579]: E1104 02:32:02.985866 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3a7ab1db-ad88-40d9-a7e9-b511bbd4584c-original-pull-secret podName:3a7ab1db-ad88-40d9-a7e9-b511bbd4584c nodeName:}" failed. No retries permitted until 2026-11-04 02:32:10.985855335 +0000 UTC m=+17.340962423 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/3a7ab1db-ad88-40d9-a7e9-b511bbd4584c-original-pull-secret") pod "global-pull-secret-syncer-krvqk" (UID: "3a7ab1db-ad88-40d9-a7e9-b511bbd4584c") : object "kube-system"/"original-pull-secret" not registered Nov 04 02:32:03.086714 ip-10-0-134-125 kubenswrapper[2579]: I1104 02:32:03.086511 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dddmg\" (UniqueName: \"kubernetes.io/projected/6e27a76f-fbdb-4ce6-9df6-659cab9ad06c-kube-api-access-dddmg\") pod \"network-check-target-zd8z9\" (UID: \"6e27a76f-fbdb-4ce6-9df6-659cab9ad06c\") " pod="openshift-network-diagnostics/network-check-target-zd8z9" Nov 04 02:32:03.087114 ip-10-0-134-125 kubenswrapper[2579]: E1104 02:32:03.086926 2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Nov 04 02:32:03.087114 ip-10-0-134-125 kubenswrapper[2579]: E1104 02:32:03.086953 2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Nov 04 02:32:03.087114 ip-10-0-134-125 kubenswrapper[2579]: E1104 02:32:03.086977 2579 projected.go:196] Error preparing data for projected volume kube-api-access-dddmg for pod openshift-network-diagnostics/network-check-target-zd8z9: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 04 02:32:03.087114 ip-10-0-134-125 kubenswrapper[2579]: E1104 02:32:03.087089 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/6e27a76f-fbdb-4ce6-9df6-659cab9ad06c-kube-api-access-dddmg podName:6e27a76f-fbdb-4ce6-9df6-659cab9ad06c nodeName:}" failed. No retries permitted until 2026-11-04 02:32:11.08706728 +0000 UTC m=+17.442174375 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-dddmg" (UniqueName: "kubernetes.io/projected/6e27a76f-fbdb-4ce6-9df6-659cab9ad06c-kube-api-access-dddmg") pod "network-check-target-zd8z9" (UID: "6e27a76f-fbdb-4ce6-9df6-659cab9ad06c") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 04 02:32:03.149352 ip-10-0-134-125 kubenswrapper[2579]: I1104 02:32:03.149250 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-krvqk" Nov 04 02:32:03.149502 ip-10-0-134-125 kubenswrapper[2579]: E1104 02:32:03.149387 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-krvqk" podUID="3a7ab1db-ad88-40d9-a7e9-b511bbd4584c" Nov 04 02:32:03.149502 ip-10-0-134-125 kubenswrapper[2579]: I1104 02:32:03.149465 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zd8z9" Nov 04 02:32:03.149598 ip-10-0-134-125 kubenswrapper[2579]: E1104 02:32:03.149539 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zd8z9" podUID="6e27a76f-fbdb-4ce6-9df6-659cab9ad06c" Nov 04 02:32:03.149637 ip-10-0-134-125 kubenswrapper[2579]: I1104 02:32:03.149605 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bdcds" Nov 04 02:32:03.149730 ip-10-0-134-125 kubenswrapper[2579]: E1104 02:32:03.149707 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-bdcds" podUID="b37e5c97-77c6-4bb4-a9e4-e1ae27484457" Nov 04 02:32:03.274348 ip-10-0-134-125 kubenswrapper[2579]: I1104 02:32:03.274248 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6mvxj" event={"ID":"f58e4ac8-c132-4716-b6c0-8d8ddb9c19e5","Type":"ContainerStarted","Data":"4d825386143c641ef2c30e9861fda169741afa1b9a1c065beb5a7281187304f1"} Nov 04 02:32:03.278317 ip-10-0-134-125 kubenswrapper[2579]: I1104 02:32:03.277799 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-2h96g" event={"ID":"7cb29949-bfe4-47a2-bc17-115a8d4d895d","Type":"ContainerStarted","Data":"49899f1574224b8d016cf7ceb8cc20d4d959c5ca14ddb091adfe28c10768f6ad"} Nov 04 02:32:03.392679 ip-10-0-134-125 kubenswrapper[2579]: I1104 02:32:03.392616 2579 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-134-125.us-west-1.compute.internal" podStartSLOduration=9.392603437 podStartE2EDuration="9.392603437s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-04 02:31:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 02:32:02.492914634 +0000 UTC m=+8.848021738" watchObservedRunningTime="2026-11-04 02:32:03.392603437 +0000 UTC m=+9.747710539" Nov 04 02:32:04.838637 ip-10-0-134-125 kubenswrapper[2579]: I1104 02:32:04.838601 2579 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-wcp7z" Nov 04 02:32:04.839643 ip-10-0-134-125 kubenswrapper[2579]: I1104 02:32:04.839620 2579 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-wcp7z" Nov 04 02:32:04.954623 ip-10-0-134-125 kubenswrapper[2579]: I1104 02:32:04.954569 2579 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-image-registry/node-ca-2h96g" podStartSLOduration=4.435403819 podStartE2EDuration="10.954550177s" totalImagesPullingTime="6.519146358s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-04 02:31:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 02:32:03.392855164 +0000 UTC m=+9.747962264" watchObservedRunningTime="2026-11-04 02:32:04.954550177 +0000 UTC m=+11.309657281" Nov 04 02:32:05.150388 ip-10-0-134-125 kubenswrapper[2579]: I1104 02:32:05.150355 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zd8z9" Nov 04 02:32:05.150492 ip-10-0-134-125 kubenswrapper[2579]: E1104 02:32:05.150449 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zd8z9" podUID="6e27a76f-fbdb-4ce6-9df6-659cab9ad06c" Nov 04 02:32:05.150527 ip-10-0-134-125 kubenswrapper[2579]: I1104 02:32:05.150503 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bdcds" Nov 04 02:32:05.150563 ip-10-0-134-125 kubenswrapper[2579]: E1104 02:32:05.150550 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-bdcds" podUID="b37e5c97-77c6-4bb4-a9e4-e1ae27484457" Nov 04 02:32:05.150596 ip-10-0-134-125 kubenswrapper[2579]: I1104 02:32:05.150591 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-krvqk" Nov 04 02:32:05.150683 ip-10-0-134-125 kubenswrapper[2579]: E1104 02:32:05.150636 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-krvqk" podUID="3a7ab1db-ad88-40d9-a7e9-b511bbd4584c" Nov 04 02:32:05.185695 ip-10-0-134-125 kubenswrapper[2579]: I1104 02:32:05.185648 2579 plugin_watcher.go:201] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Nov 04 02:32:05.296116 ip-10-0-134-125 kubenswrapper[2579]: I1104 02:32:05.296039 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6mvxj" event={"ID":"f58e4ac8-c132-4716-b6c0-8d8ddb9c19e5","Type":"ContainerStarted","Data":"e14b0508f809ae9dfa50a204d3f0c3d7d545cc17e29cfd0981d24999ecc4cf50"} Nov 04 02:32:05.299746 ip-10-0-134-125 kubenswrapper[2579]: I1104 02:32:05.298736 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-tglvr" event={"ID":"3984ce7b-96c8-464a-aa14-28487cc5b373","Type":"ContainerStarted","Data":"f153434f18410b20ff5deffa8e39d19a3f13573e6fd05834780b421124def682"} Nov 04 02:32:05.299746 ip-10-0-134-125 kubenswrapper[2579]: I1104 02:32:05.299040 2579 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-wcp7z" Nov 04 02:32:05.299746 ip-10-0-134-125 kubenswrapper[2579]: I1104 02:32:05.299502 2579 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-wcp7z" Nov 04 02:32:05.417711 ip-10-0-134-125 kubenswrapper[2579]: I1104 02:32:05.417636 2579 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/multus-tglvr" podStartSLOduration=3.002539031 podStartE2EDuration="11.417623311s" totalImagesPullingTime="8.41508428s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-04 02:31:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 02:32:05.41715089 +0000 UTC m=+11.772258197" watchObservedRunningTime="2026-11-04 02:32:05.417623311 +0000 UTC m=+11.772730413" Nov 04 02:32:06.078469 ip-10-0-134-125 kubenswrapper[2579]: I1104 02:32:06.078326 2579 reconciler.go:167] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-11-04T02:32:05.185691225Z","UUID":"5801b8ff-2e7a-40bd-b0c2-538af524e86c","Handler":null,"Name":"","Endpoint":""} Nov 04 02:32:06.080761 ip-10-0-134-125 kubenswrapper[2579]: I1104 02:32:06.080208 2579 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Nov 04 02:32:06.080761 ip-10-0-134-125 kubenswrapper[2579]: I1104 02:32:06.080245 2579 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Nov 04 02:32:06.303492 ip-10-0-134-125 kubenswrapper[2579]: I1104 02:32:06.303457 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-sx4w7" event={"ID":"524e33f8-723b-4868-a4f5-5d7eb92a5fe3","Type":"ContainerStarted","Data":"8c5a81b402d15e28e49f824152402c74376ccafa512585af9abf55d06c7a0d6b"} Nov 04 02:32:06.305386 ip-10-0-134-125 kubenswrapper[2579]: I1104 02:32:06.305353 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-kzzwk" event={"ID":"8b030f29-dfa6-42b1-81a7-07b539ad9aa9","Type":"ContainerStarted","Data":"5554f2ad503770e43abdd784940ab95c3900fb94f65d3ff61a04cdf1ac038eb8"} Nov 04 02:32:06.308164 ip-10-0-134-125 kubenswrapper[2579]: I1104 02:32:06.307729 2579 generic.go:356] "Generic (PLEG): container finished" podID="5590efb1-88b8-41ad-b0ee-37b2ff9442de" containerID="8ac295934f2b3ad233e3ef4ab5980130273940e207feb3bda29db1624fb0d924" exitCode=0 Nov 04 02:32:06.308164 ip-10-0-134-125 kubenswrapper[2579]: I1104 02:32:06.307767 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-sgx95" event={"ID":"5590efb1-88b8-41ad-b0ee-37b2ff9442de","Type":"ContainerDied","Data":"8ac295934f2b3ad233e3ef4ab5980130273940e207feb3bda29db1624fb0d924"} Nov 04 02:32:06.418890 ip-10-0-134-125 kubenswrapper[2579]: I1104 02:32:06.418618 2579 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-sx4w7" podStartSLOduration=3.350801917 podStartE2EDuration="12.41860431s" totalImagesPullingTime="9.067802393s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-04 02:31:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 02:32:06.418384699 +0000 UTC m=+12.773491802" watchObservedRunningTime="2026-11-04 02:32:06.41860431 +0000 UTC m=+12.773711412" Nov 04 02:32:06.533290 ip-10-0-134-125 kubenswrapper[2579]: I1104 02:32:06.533244 2579 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-dns/node-resolver-kzzwk" podStartSLOduration=3.799900594 podStartE2EDuration="12.533228619s" totalImagesPullingTime="8.733328025s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-04 02:31:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 02:32:06.532658631 +0000 UTC m=+12.887765736" watchObservedRunningTime="2026-11-04 02:32:06.533228619 +0000 UTC m=+12.888335718" Nov 04 02:32:07.149715 ip-10-0-134-125 kubenswrapper[2579]: I1104 02:32:07.149604 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-krvqk" Nov 04 02:32:07.149715 ip-10-0-134-125 kubenswrapper[2579]: I1104 02:32:07.149628 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bdcds" Nov 04 02:32:07.150252 ip-10-0-134-125 kubenswrapper[2579]: E1104 02:32:07.149786 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-krvqk" podUID="3a7ab1db-ad88-40d9-a7e9-b511bbd4584c" Nov 04 02:32:07.150252 ip-10-0-134-125 kubenswrapper[2579]: I1104 02:32:07.149835 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zd8z9" Nov 04 02:32:07.150252 ip-10-0-134-125 kubenswrapper[2579]: E1104 02:32:07.149961 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-bdcds" podUID="b37e5c97-77c6-4bb4-a9e4-e1ae27484457" Nov 04 02:32:07.150252 ip-10-0-134-125 kubenswrapper[2579]: E1104 02:32:07.150053 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zd8z9" podUID="6e27a76f-fbdb-4ce6-9df6-659cab9ad06c" Nov 04 02:32:07.312872 ip-10-0-134-125 kubenswrapper[2579]: I1104 02:32:07.312835 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6mvxj" event={"ID":"f58e4ac8-c132-4716-b6c0-8d8ddb9c19e5","Type":"ContainerStarted","Data":"44be4a4d1880f783f395a97d9627bff50601f432d73cda436a4c95f89b55e674"} Nov 04 02:32:07.431714 ip-10-0-134-125 kubenswrapper[2579]: I1104 02:32:07.431633 2579 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6mvxj" podStartSLOduration=2.6007672 podStartE2EDuration="13.431614188s" totalImagesPullingTime="10.830846988s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-04 02:31:54 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 02:32:07.431593548 +0000 UTC m=+13.786700655" watchObservedRunningTime="2026-11-04 02:32:07.431614188 +0000 UTC m=+13.786721292" Nov 04 02:32:08.317305 ip-10-0-134-125 kubenswrapper[2579]: I1104 02:32:08.317259 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-ncfkk" event={"ID":"cb1915a0-5505-4f32-8f79-2f638465e404","Type":"ContainerStarted","Data":"f6f0befd05809d699e93ef73d1caf555209d10f0e65eb7ba722b496501d44c8e"} Nov 04 02:32:08.435280 ip-10-0-134-125 kubenswrapper[2579]: I1104 02:32:08.435223 2579 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-ncfkk" podStartSLOduration=3.619607021 podStartE2EDuration="14.435210009s" totalImagesPullingTime="10.815602988s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-04 02:31:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 02:32:08.434795201 +0000 UTC m=+14.789902305" watchObservedRunningTime="2026-11-04 02:32:08.435210009 +0000 UTC m=+14.790317112" Nov 04 02:32:09.149684 ip-10-0-134-125 kubenswrapper[2579]: I1104 02:32:09.149634 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-krvqk" Nov 04 02:32:09.149857 ip-10-0-134-125 kubenswrapper[2579]: I1104 02:32:09.149723 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zd8z9" Nov 04 02:32:09.149857 ip-10-0-134-125 kubenswrapper[2579]: I1104 02:32:09.149815 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bdcds" Nov 04 02:32:09.149857 ip-10-0-134-125 kubenswrapper[2579]: E1104 02:32:09.149836 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-krvqk" podUID="3a7ab1db-ad88-40d9-a7e9-b511bbd4584c" Nov 04 02:32:09.150033 ip-10-0-134-125 kubenswrapper[2579]: E1104 02:32:09.149927 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-bdcds" podUID="b37e5c97-77c6-4bb4-a9e4-e1ae27484457" Nov 04 02:32:09.150033 ip-10-0-134-125 kubenswrapper[2579]: E1104 02:32:09.150009 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zd8z9" podUID="6e27a76f-fbdb-4ce6-9df6-659cab9ad06c" Nov 04 02:32:09.322244 ip-10-0-134-125 kubenswrapper[2579]: I1104 02:32:09.322153 2579 generic.go:356] "Generic (PLEG): container finished" podID="5590efb1-88b8-41ad-b0ee-37b2ff9442de" containerID="47a4ead167d300937b05838143c2ee79d5c09b04650c4f7dfa4325807791e10e" exitCode=0 Nov 04 02:32:09.323018 ip-10-0-134-125 kubenswrapper[2579]: I1104 02:32:09.322238 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-sgx95" event={"ID":"5590efb1-88b8-41ad-b0ee-37b2ff9442de","Type":"ContainerDied","Data":"47a4ead167d300937b05838143c2ee79d5c09b04650c4f7dfa4325807791e10e"} Nov 04 02:32:11.053562 ip-10-0-134-125 kubenswrapper[2579]: I1104 02:32:11.053317 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/3a7ab1db-ad88-40d9-a7e9-b511bbd4584c-original-pull-secret\") pod \"global-pull-secret-syncer-krvqk\" (UID: \"3a7ab1db-ad88-40d9-a7e9-b511bbd4584c\") " pod="kube-system/global-pull-secret-syncer-krvqk" Nov 04 02:32:11.053562 ip-10-0-134-125 kubenswrapper[2579]: I1104 02:32:11.053534 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b37e5c97-77c6-4bb4-a9e4-e1ae27484457-metrics-certs\") pod \"network-metrics-daemon-bdcds\" (UID: \"b37e5c97-77c6-4bb4-a9e4-e1ae27484457\") " pod="openshift-multus/network-metrics-daemon-bdcds" Nov 04 02:32:11.053562 ip-10-0-134-125 kubenswrapper[2579]: E1104 02:32:11.053473 2579 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Nov 04 02:32:11.054038 ip-10-0-134-125 kubenswrapper[2579]: E1104 02:32:11.053624 2579 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Nov 04 02:32:11.054038 ip-10-0-134-125 kubenswrapper[2579]: E1104 02:32:11.053634 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3a7ab1db-ad88-40d9-a7e9-b511bbd4584c-original-pull-secret podName:3a7ab1db-ad88-40d9-a7e9-b511bbd4584c nodeName:}" failed. No retries permitted until 2026-11-04 02:32:27.053615354 +0000 UTC m=+33.408722437 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/3a7ab1db-ad88-40d9-a7e9-b511bbd4584c-original-pull-secret") pod "global-pull-secret-syncer-krvqk" (UID: "3a7ab1db-ad88-40d9-a7e9-b511bbd4584c") : object "kube-system"/"original-pull-secret" not registered Nov 04 02:32:11.054038 ip-10-0-134-125 kubenswrapper[2579]: E1104 02:32:11.053692 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b37e5c97-77c6-4bb4-a9e4-e1ae27484457-metrics-certs podName:b37e5c97-77c6-4bb4-a9e4-e1ae27484457 nodeName:}" failed. No retries permitted until 2026-11-04 02:32:27.053657984 +0000 UTC m=+33.408765078 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/b37e5c97-77c6-4bb4-a9e4-e1ae27484457-metrics-certs") pod "network-metrics-daemon-bdcds" (UID: "b37e5c97-77c6-4bb4-a9e4-e1ae27484457") : object "openshift-multus"/"metrics-daemon-secret" not registered Nov 04 02:32:11.149323 ip-10-0-134-125 kubenswrapper[2579]: I1104 02:32:11.149296 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-krvqk" Nov 04 02:32:11.149502 ip-10-0-134-125 kubenswrapper[2579]: I1104 02:32:11.149389 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zd8z9" Nov 04 02:32:11.149502 ip-10-0-134-125 kubenswrapper[2579]: E1104 02:32:11.149395 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-krvqk" podUID="3a7ab1db-ad88-40d9-a7e9-b511bbd4584c" Nov 04 02:32:11.149631 ip-10-0-134-125 kubenswrapper[2579]: E1104 02:32:11.149497 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zd8z9" podUID="6e27a76f-fbdb-4ce6-9df6-659cab9ad06c" Nov 04 02:32:11.149631 ip-10-0-134-125 kubenswrapper[2579]: I1104 02:32:11.149548 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bdcds" Nov 04 02:32:11.149738 ip-10-0-134-125 kubenswrapper[2579]: E1104 02:32:11.149642 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-bdcds" podUID="b37e5c97-77c6-4bb4-a9e4-e1ae27484457" Nov 04 02:32:11.154523 ip-10-0-134-125 kubenswrapper[2579]: I1104 02:32:11.154500 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dddmg\" (UniqueName: \"kubernetes.io/projected/6e27a76f-fbdb-4ce6-9df6-659cab9ad06c-kube-api-access-dddmg\") pod \"network-check-target-zd8z9\" (UID: \"6e27a76f-fbdb-4ce6-9df6-659cab9ad06c\") " pod="openshift-network-diagnostics/network-check-target-zd8z9" Nov 04 02:32:11.154636 ip-10-0-134-125 kubenswrapper[2579]: E1104 02:32:11.154626 2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Nov 04 02:32:11.154691 ip-10-0-134-125 kubenswrapper[2579]: E1104 02:32:11.154642 2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Nov 04 02:32:11.154691 ip-10-0-134-125 kubenswrapper[2579]: E1104 02:32:11.154651 2579 projected.go:196] Error preparing data for projected volume kube-api-access-dddmg for pod openshift-network-diagnostics/network-check-target-zd8z9: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 04 02:32:11.156111 ip-10-0-134-125 kubenswrapper[2579]: E1104 02:32:11.154720 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/6e27a76f-fbdb-4ce6-9df6-659cab9ad06c-kube-api-access-dddmg podName:6e27a76f-fbdb-4ce6-9df6-659cab9ad06c nodeName:}" failed. No retries permitted until 2026-11-04 02:32:27.15470304 +0000 UTC m=+33.509810128 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-dddmg" (UniqueName: "kubernetes.io/projected/6e27a76f-fbdb-4ce6-9df6-659cab9ad06c-kube-api-access-dddmg") pod "network-check-target-zd8z9" (UID: "6e27a76f-fbdb-4ce6-9df6-659cab9ad06c") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 04 02:32:11.328952 ip-10-0-134-125 kubenswrapper[2579]: I1104 02:32:11.328915 2579 generic.go:356] "Generic (PLEG): container finished" podID="5590efb1-88b8-41ad-b0ee-37b2ff9442de" containerID="52fa444baef42ca6a81a26cc17ea54dd984a1d9854f060b085428a6d0c43f4ff" exitCode=0 Nov 04 02:32:11.329064 ip-10-0-134-125 kubenswrapper[2579]: I1104 02:32:11.328970 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-sgx95" event={"ID":"5590efb1-88b8-41ad-b0ee-37b2ff9442de","Type":"ContainerDied","Data":"52fa444baef42ca6a81a26cc17ea54dd984a1d9854f060b085428a6d0c43f4ff"} Nov 04 02:32:12.334959 ip-10-0-134-125 kubenswrapper[2579]: I1104 02:32:12.334740 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-vwcx8_ffbb2d83-5a93-4a5b-bce3-e40afb8f58ea/ovn-acl-logging/0.log" Nov 04 02:32:12.335421 ip-10-0-134-125 kubenswrapper[2579]: I1104 02:32:12.335263 2579 generic.go:356] "Generic (PLEG): container finished" podID="ffbb2d83-5a93-4a5b-bce3-e40afb8f58ea" containerID="27e1d382d0a808b2f653af8eda862b4e951c02cee226bb3ed628e1179030bddb" exitCode=1 Nov 04 02:32:12.335421 ip-10-0-134-125 kubenswrapper[2579]: I1104 02:32:12.335305 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-vwcx8" event={"ID":"ffbb2d83-5a93-4a5b-bce3-e40afb8f58ea","Type":"ContainerStarted","Data":"9808a4c130458794eb61e0848ce3377d2c182396cb72a7d33406a90ea3929e5f"} Nov 04 02:32:12.335421 ip-10-0-134-125 kubenswrapper[2579]: I1104 02:32:12.335382 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-vwcx8" event={"ID":"ffbb2d83-5a93-4a5b-bce3-e40afb8f58ea","Type":"ContainerStarted","Data":"3ce8e6c2602637c14e712872fb26eeead02a72d45d77f129f2cfd803ee87e552"} Nov 04 02:32:12.335525 ip-10-0-134-125 kubenswrapper[2579]: I1104 02:32:12.335423 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-vwcx8" event={"ID":"ffbb2d83-5a93-4a5b-bce3-e40afb8f58ea","Type":"ContainerStarted","Data":"bb85614a2f81fd36a8a01d8dabe2f49649f98f20d9d32c4d46eaa4b3813c77a4"} Nov 04 02:32:12.335525 ip-10-0-134-125 kubenswrapper[2579]: I1104 02:32:12.335451 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-vwcx8" event={"ID":"ffbb2d83-5a93-4a5b-bce3-e40afb8f58ea","Type":"ContainerStarted","Data":"7e7c858f3b16a237e260a05b104440feda1976b74315255cd4caed955138c723"} Nov 04 02:32:12.335525 ip-10-0-134-125 kubenswrapper[2579]: I1104 02:32:12.335477 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-vwcx8" event={"ID":"ffbb2d83-5a93-4a5b-bce3-e40afb8f58ea","Type":"ContainerDied","Data":"27e1d382d0a808b2f653af8eda862b4e951c02cee226bb3ed628e1179030bddb"} Nov 04 02:32:12.335525 ip-10-0-134-125 kubenswrapper[2579]: I1104 02:32:12.335505 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-vwcx8" event={"ID":"ffbb2d83-5a93-4a5b-bce3-e40afb8f58ea","Type":"ContainerStarted","Data":"46c187d35f6fab172edd2f0634960007239bfb74dd1a585de08f719e136ba7ee"} Nov 04 02:32:13.149780 ip-10-0-134-125 kubenswrapper[2579]: I1104 02:32:13.149751 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zd8z9" Nov 04 02:32:13.149928 ip-10-0-134-125 kubenswrapper[2579]: I1104 02:32:13.149789 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bdcds" Nov 04 02:32:13.149928 ip-10-0-134-125 kubenswrapper[2579]: E1104 02:32:13.149845 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zd8z9" podUID="6e27a76f-fbdb-4ce6-9df6-659cab9ad06c" Nov 04 02:32:13.149928 ip-10-0-134-125 kubenswrapper[2579]: I1104 02:32:13.149885 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-krvqk" Nov 04 02:32:13.150078 ip-10-0-134-125 kubenswrapper[2579]: E1104 02:32:13.149994 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-krvqk" podUID="3a7ab1db-ad88-40d9-a7e9-b511bbd4584c" Nov 04 02:32:13.150078 ip-10-0-134-125 kubenswrapper[2579]: E1104 02:32:13.150046 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-bdcds" podUID="b37e5c97-77c6-4bb4-a9e4-e1ae27484457" Nov 04 02:32:13.339380 ip-10-0-134-125 kubenswrapper[2579]: I1104 02:32:13.339347 2579 generic.go:356] "Generic (PLEG): container finished" podID="5590efb1-88b8-41ad-b0ee-37b2ff9442de" containerID="5cb7eb23c657a2e50d9f8d7a68926096010528f0b6598f76c429b3485e5eed39" exitCode=0 Nov 04 02:32:13.339796 ip-10-0-134-125 kubenswrapper[2579]: I1104 02:32:13.339394 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-sgx95" event={"ID":"5590efb1-88b8-41ad-b0ee-37b2ff9442de","Type":"ContainerDied","Data":"5cb7eb23c657a2e50d9f8d7a68926096010528f0b6598f76c429b3485e5eed39"} Nov 04 02:32:14.344774 ip-10-0-134-125 kubenswrapper[2579]: I1104 02:32:14.344743 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-vwcx8_ffbb2d83-5a93-4a5b-bce3-e40afb8f58ea/ovn-acl-logging/0.log" Nov 04 02:32:14.345310 ip-10-0-134-125 kubenswrapper[2579]: I1104 02:32:14.345084 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-vwcx8" event={"ID":"ffbb2d83-5a93-4a5b-bce3-e40afb8f58ea","Type":"ContainerStarted","Data":"e620531a47b412881fedef4eee6e3f0a478f5ac90a1cda8800d975a5066c9081"} Nov 04 02:32:14.348031 ip-10-0-134-125 kubenswrapper[2579]: I1104 02:32:14.348002 2579 generic.go:356] "Generic (PLEG): container finished" podID="5590efb1-88b8-41ad-b0ee-37b2ff9442de" containerID="521d111ac8b37148a2e99975bc5c4b1a1b607ddc48fe54f97d58a90d8248e495" exitCode=0 Nov 04 02:32:14.348126 ip-10-0-134-125 kubenswrapper[2579]: I1104 02:32:14.348048 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-sgx95" event={"ID":"5590efb1-88b8-41ad-b0ee-37b2ff9442de","Type":"ContainerDied","Data":"521d111ac8b37148a2e99975bc5c4b1a1b607ddc48fe54f97d58a90d8248e495"} Nov 04 02:32:15.150266 ip-10-0-134-125 kubenswrapper[2579]: I1104 02:32:15.150231 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-krvqk" Nov 04 02:32:15.150266 ip-10-0-134-125 kubenswrapper[2579]: I1104 02:32:15.150263 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zd8z9" Nov 04 02:32:15.150530 ip-10-0-134-125 kubenswrapper[2579]: E1104 02:32:15.150360 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-krvqk" podUID="3a7ab1db-ad88-40d9-a7e9-b511bbd4584c" Nov 04 02:32:15.150530 ip-10-0-134-125 kubenswrapper[2579]: E1104 02:32:15.150390 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zd8z9" podUID="6e27a76f-fbdb-4ce6-9df6-659cab9ad06c" Nov 04 02:32:15.150530 ip-10-0-134-125 kubenswrapper[2579]: I1104 02:32:15.150447 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bdcds" Nov 04 02:32:15.150663 ip-10-0-134-125 kubenswrapper[2579]: E1104 02:32:15.150532 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-bdcds" podUID="b37e5c97-77c6-4bb4-a9e4-e1ae27484457" Nov 04 02:32:15.353467 ip-10-0-134-125 kubenswrapper[2579]: I1104 02:32:15.353427 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-sgx95" event={"ID":"5590efb1-88b8-41ad-b0ee-37b2ff9442de","Type":"ContainerStarted","Data":"2ae9bb1bcf19e76cbe6953158a9ff39b3b3f7288cc39ed43da993f3bac9a9d80"} Nov 04 02:32:15.475353 ip-10-0-134-125 kubenswrapper[2579]: I1104 02:32:15.475249 2579 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-sgx95" podStartSLOduration=7.174248317 podStartE2EDuration="21.475235918s" totalImagesPullingTime="14.300987601s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-04 02:31:54 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 02:32:15.474963455 +0000 UTC m=+21.830070558" watchObservedRunningTime="2026-11-04 02:32:15.475235918 +0000 UTC m=+21.830343024" Nov 04 02:32:16.358779 ip-10-0-134-125 kubenswrapper[2579]: I1104 02:32:16.358748 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-vwcx8_ffbb2d83-5a93-4a5b-bce3-e40afb8f58ea/ovn-acl-logging/0.log" Nov 04 02:32:16.359422 ip-10-0-134-125 kubenswrapper[2579]: I1104 02:32:16.359125 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-vwcx8" event={"ID":"ffbb2d83-5a93-4a5b-bce3-e40afb8f58ea","Type":"ContainerStarted","Data":"e298323a0ae73b26a13dbbe4c7c93121349374d44aa1f9d47782642babee98aa"} Nov 04 02:32:16.359578 ip-10-0-134-125 kubenswrapper[2579]: I1104 02:32:16.359563 2579 scope.go:122] "RemoveContainer" containerID="27e1d382d0a808b2f653af8eda862b4e951c02cee226bb3ed628e1179030bddb" Nov 04 02:32:16.359747 ip-10-0-134-125 kubenswrapper[2579]: I1104 02:32:16.359730 2579 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-vwcx8" Nov 04 02:32:16.376168 ip-10-0-134-125 kubenswrapper[2579]: I1104 02:32:16.376148 2579 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-vwcx8" Nov 04 02:32:17.150319 ip-10-0-134-125 kubenswrapper[2579]: I1104 02:32:17.150289 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-krvqk" Nov 04 02:32:17.150474 ip-10-0-134-125 kubenswrapper[2579]: I1104 02:32:17.150321 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zd8z9" Nov 04 02:32:17.150474 ip-10-0-134-125 kubenswrapper[2579]: E1104 02:32:17.150388 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-krvqk" podUID="3a7ab1db-ad88-40d9-a7e9-b511bbd4584c" Nov 04 02:32:17.150474 ip-10-0-134-125 kubenswrapper[2579]: I1104 02:32:17.150434 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bdcds" Nov 04 02:32:17.150710 ip-10-0-134-125 kubenswrapper[2579]: E1104 02:32:17.150515 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zd8z9" podUID="6e27a76f-fbdb-4ce6-9df6-659cab9ad06c" Nov 04 02:32:17.150710 ip-10-0-134-125 kubenswrapper[2579]: E1104 02:32:17.150593 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-bdcds" podUID="b37e5c97-77c6-4bb4-a9e4-e1ae27484457" Nov 04 02:32:17.365268 ip-10-0-134-125 kubenswrapper[2579]: I1104 02:32:17.365244 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-vwcx8_ffbb2d83-5a93-4a5b-bce3-e40afb8f58ea/ovn-acl-logging/0.log" Nov 04 02:32:17.365634 ip-10-0-134-125 kubenswrapper[2579]: I1104 02:32:17.365619 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-vwcx8" event={"ID":"ffbb2d83-5a93-4a5b-bce3-e40afb8f58ea","Type":"ContainerStarted","Data":"07acc7b1bfbe87de51cb073f9e327378def2a4c62004a3420a7b10f5495c1314"} Nov 04 02:32:17.365862 ip-10-0-134-125 kubenswrapper[2579]: I1104 02:32:17.365835 2579 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-vwcx8" Nov 04 02:32:17.366012 ip-10-0-134-125 kubenswrapper[2579]: I1104 02:32:17.365910 2579 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-vwcx8" Nov 04 02:32:17.381865 ip-10-0-134-125 kubenswrapper[2579]: I1104 02:32:17.381838 2579 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-vwcx8" Nov 04 02:32:17.499812 ip-10-0-134-125 kubenswrapper[2579]: I1104 02:32:17.498040 2579 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-vwcx8" podStartSLOduration=7.914918825 podStartE2EDuration="23.498021235s" totalImagesPullingTime="15.58310241s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-04 02:31:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 02:32:17.496766066 +0000 UTC m=+23.851873169" watchObservedRunningTime="2026-11-04 02:32:17.498021235 +0000 UTC m=+23.853128341" Nov 04 02:32:18.525497 ip-10-0-134-125 kubenswrapper[2579]: I1104 02:32:18.525423 2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-krvqk"] Nov 04 02:32:18.526079 ip-10-0-134-125 kubenswrapper[2579]: I1104 02:32:18.525534 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-krvqk" Nov 04 02:32:18.526079 ip-10-0-134-125 kubenswrapper[2579]: E1104 02:32:18.525620 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-krvqk" podUID="3a7ab1db-ad88-40d9-a7e9-b511bbd4584c" Nov 04 02:32:18.528406 ip-10-0-134-125 kubenswrapper[2579]: I1104 02:32:18.528380 2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-zd8z9"] Nov 04 02:32:18.528530 ip-10-0-134-125 kubenswrapper[2579]: I1104 02:32:18.528483 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zd8z9" Nov 04 02:32:18.528605 ip-10-0-134-125 kubenswrapper[2579]: E1104 02:32:18.528573 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zd8z9" podUID="6e27a76f-fbdb-4ce6-9df6-659cab9ad06c" Nov 04 02:32:18.529067 ip-10-0-134-125 kubenswrapper[2579]: I1104 02:32:18.529044 2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-bdcds"] Nov 04 02:32:18.529183 ip-10-0-134-125 kubenswrapper[2579]: I1104 02:32:18.529159 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bdcds" Nov 04 02:32:18.529286 ip-10-0-134-125 kubenswrapper[2579]: E1104 02:32:18.529264 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-bdcds" podUID="b37e5c97-77c6-4bb4-a9e4-e1ae27484457" Nov 04 02:32:20.149475 ip-10-0-134-125 kubenswrapper[2579]: I1104 02:32:20.149446 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bdcds" Nov 04 02:32:20.149957 ip-10-0-134-125 kubenswrapper[2579]: I1104 02:32:20.149447 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zd8z9" Nov 04 02:32:20.149957 ip-10-0-134-125 kubenswrapper[2579]: E1104 02:32:20.149554 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-bdcds" podUID="b37e5c97-77c6-4bb4-a9e4-e1ae27484457" Nov 04 02:32:20.149957 ip-10-0-134-125 kubenswrapper[2579]: E1104 02:32:20.149654 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zd8z9" podUID="6e27a76f-fbdb-4ce6-9df6-659cab9ad06c" Nov 04 02:32:20.149957 ip-10-0-134-125 kubenswrapper[2579]: I1104 02:32:20.149447 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-krvqk" Nov 04 02:32:20.149957 ip-10-0-134-125 kubenswrapper[2579]: E1104 02:32:20.149792 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-krvqk" podUID="3a7ab1db-ad88-40d9-a7e9-b511bbd4584c" Nov 04 02:32:22.149742 ip-10-0-134-125 kubenswrapper[2579]: I1104 02:32:22.149706 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-krvqk" Nov 04 02:32:22.150249 ip-10-0-134-125 kubenswrapper[2579]: I1104 02:32:22.149706 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bdcds" Nov 04 02:32:22.150249 ip-10-0-134-125 kubenswrapper[2579]: I1104 02:32:22.149841 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zd8z9" Nov 04 02:32:22.150249 ip-10-0-134-125 kubenswrapper[2579]: E1104 02:32:22.149842 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-krvqk" podUID="3a7ab1db-ad88-40d9-a7e9-b511bbd4584c" Nov 04 02:32:22.150249 ip-10-0-134-125 kubenswrapper[2579]: E1104 02:32:22.149930 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-bdcds" podUID="b37e5c97-77c6-4bb4-a9e4-e1ae27484457" Nov 04 02:32:22.150249 ip-10-0-134-125 kubenswrapper[2579]: E1104 02:32:22.149998 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zd8z9" podUID="6e27a76f-fbdb-4ce6-9df6-659cab9ad06c" Nov 04 02:32:23.891563 ip-10-0-134-125 kubenswrapper[2579]: I1104 02:32:23.891484 2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-125.us-west-1.compute.internal" event="NodeReady" Nov 04 02:32:23.891975 ip-10-0-134-125 kubenswrapper[2579]: I1104 02:32:23.891633 2579 kubelet_node_status.go:477] "Fast updating node status as it just became ready" Nov 04 02:32:24.005376 ip-10-0-134-125 kubenswrapper[2579]: I1104 02:32:24.005347 2579 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-wbzfc"] Nov 04 02:32:24.008460 ip-10-0-134-125 kubenswrapper[2579]: I1104 02:32:24.008442 2579 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-mn7dn"] Nov 04 02:32:24.008611 ip-10-0-134-125 kubenswrapper[2579]: I1104 02:32:24.008593 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-wbzfc" Nov 04 02:32:24.011234 ip-10-0-134-125 kubenswrapper[2579]: I1104 02:32:24.011214 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-mn7dn" Nov 04 02:32:24.044445 ip-10-0-134-125 kubenswrapper[2579]: I1104 02:32:24.044415 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/4f76b032-61eb-4796-b082-47c23ff73d18-config-volume\") pod \"dns-default-wbzfc\" (UID: \"4f76b032-61eb-4796-b082-47c23ff73d18\") " pod="openshift-dns/dns-default-wbzfc" Nov 04 02:32:24.044566 ip-10-0-134-125 kubenswrapper[2579]: I1104 02:32:24.044473 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2rpzn\" (UniqueName: \"kubernetes.io/projected/a245d359-742e-489d-9c58-eea4e686286c-kube-api-access-2rpzn\") pod \"ingress-canary-mn7dn\" (UID: \"a245d359-742e-489d-9c58-eea4e686286c\") " pod="openshift-ingress-canary/ingress-canary-mn7dn" Nov 04 02:32:24.044621 ip-10-0-134-125 kubenswrapper[2579]: I1104 02:32:24.044603 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8lqpv\" (UniqueName: \"kubernetes.io/projected/4f76b032-61eb-4796-b082-47c23ff73d18-kube-api-access-8lqpv\") pod \"dns-default-wbzfc\" (UID: \"4f76b032-61eb-4796-b082-47c23ff73d18\") " pod="openshift-dns/dns-default-wbzfc" Nov 04 02:32:24.044680 ip-10-0-134-125 kubenswrapper[2579]: I1104 02:32:24.044651 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4f76b032-61eb-4796-b082-47c23ff73d18-tmp-dir\") pod \"dns-default-wbzfc\" (UID: \"4f76b032-61eb-4796-b082-47c23ff73d18\") " pod="openshift-dns/dns-default-wbzfc" Nov 04 02:32:24.044723 ip-10-0-134-125 kubenswrapper[2579]: I1104 02:32:24.044709 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/a245d359-742e-489d-9c58-eea4e686286c-cert\") pod \"ingress-canary-mn7dn\" (UID: \"a245d359-742e-489d-9c58-eea4e686286c\") " pod="openshift-ingress-canary/ingress-canary-mn7dn" Nov 04 02:32:24.044755 ip-10-0-134-125 kubenswrapper[2579]: I1104 02:32:24.044745 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/4f76b032-61eb-4796-b082-47c23ff73d18-metrics-tls\") pod \"dns-default-wbzfc\" (UID: \"4f76b032-61eb-4796-b082-47c23ff73d18\") " pod="openshift-dns/dns-default-wbzfc" Nov 04 02:32:24.048320 ip-10-0-134-125 kubenswrapper[2579]: I1104 02:32:24.048297 2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-wbzfc"] Nov 04 02:32:24.061003 ip-10-0-134-125 kubenswrapper[2579]: I1104 02:32:24.060981 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Nov 04 02:32:24.061003 ip-10-0-134-125 kubenswrapper[2579]: I1104 02:32:24.060999 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Nov 04 02:32:24.061183 ip-10-0-134-125 kubenswrapper[2579]: I1104 02:32:24.061169 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-6tfxk\"" Nov 04 02:32:24.097679 ip-10-0-134-125 kubenswrapper[2579]: I1104 02:32:24.097652 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Nov 04 02:32:24.097880 ip-10-0-134-125 kubenswrapper[2579]: I1104 02:32:24.097867 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Nov 04 02:32:24.097921 ip-10-0-134-125 kubenswrapper[2579]: I1104 02:32:24.097894 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Nov 04 02:32:24.097953 ip-10-0-134-125 kubenswrapper[2579]: I1104 02:32:24.097923 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"ingress-canary-dockercfg-l8p2p\"" Nov 04 02:32:24.110946 ip-10-0-134-125 kubenswrapper[2579]: I1104 02:32:24.110916 2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-mn7dn"] Nov 04 02:32:24.146044 ip-10-0-134-125 kubenswrapper[2579]: I1104 02:32:24.145827 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/4f76b032-61eb-4796-b082-47c23ff73d18-metrics-tls\") pod \"dns-default-wbzfc\" (UID: \"4f76b032-61eb-4796-b082-47c23ff73d18\") " pod="openshift-dns/dns-default-wbzfc" Nov 04 02:32:24.146044 ip-10-0-134-125 kubenswrapper[2579]: I1104 02:32:24.146019 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/4f76b032-61eb-4796-b082-47c23ff73d18-config-volume\") pod \"dns-default-wbzfc\" (UID: \"4f76b032-61eb-4796-b082-47c23ff73d18\") " pod="openshift-dns/dns-default-wbzfc" Nov 04 02:32:24.146249 ip-10-0-134-125 kubenswrapper[2579]: I1104 02:32:24.146050 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2rpzn\" (UniqueName: \"kubernetes.io/projected/a245d359-742e-489d-9c58-eea4e686286c-kube-api-access-2rpzn\") pod \"ingress-canary-mn7dn\" (UID: \"a245d359-742e-489d-9c58-eea4e686286c\") " pod="openshift-ingress-canary/ingress-canary-mn7dn" Nov 04 02:32:24.146249 ip-10-0-134-125 kubenswrapper[2579]: E1104 02:32:24.145973 2579 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Nov 04 02:32:24.146249 ip-10-0-134-125 kubenswrapper[2579]: I1104 02:32:24.146083 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8lqpv\" (UniqueName: \"kubernetes.io/projected/4f76b032-61eb-4796-b082-47c23ff73d18-kube-api-access-8lqpv\") pod \"dns-default-wbzfc\" (UID: \"4f76b032-61eb-4796-b082-47c23ff73d18\") " pod="openshift-dns/dns-default-wbzfc" Nov 04 02:32:24.146249 ip-10-0-134-125 kubenswrapper[2579]: E1104 02:32:24.146124 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4f76b032-61eb-4796-b082-47c23ff73d18-metrics-tls podName:4f76b032-61eb-4796-b082-47c23ff73d18 nodeName:}" failed. No retries permitted until 2026-11-04 02:32:24.646105006 +0000 UTC m=+31.001212121 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/4f76b032-61eb-4796-b082-47c23ff73d18-metrics-tls") pod "dns-default-wbzfc" (UID: "4f76b032-61eb-4796-b082-47c23ff73d18") : secret "dns-default-metrics-tls" not found Nov 04 02:32:24.146249 ip-10-0-134-125 kubenswrapper[2579]: I1104 02:32:24.146141 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4f76b032-61eb-4796-b082-47c23ff73d18-tmp-dir\") pod \"dns-default-wbzfc\" (UID: \"4f76b032-61eb-4796-b082-47c23ff73d18\") " pod="openshift-dns/dns-default-wbzfc" Nov 04 02:32:24.146249 ip-10-0-134-125 kubenswrapper[2579]: I1104 02:32:24.146168 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/a245d359-742e-489d-9c58-eea4e686286c-cert\") pod \"ingress-canary-mn7dn\" (UID: \"a245d359-742e-489d-9c58-eea4e686286c\") " pod="openshift-ingress-canary/ingress-canary-mn7dn" Nov 04 02:32:24.146631 ip-10-0-134-125 kubenswrapper[2579]: I1104 02:32:24.146613 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4f76b032-61eb-4796-b082-47c23ff73d18-tmp-dir\") pod \"dns-default-wbzfc\" (UID: \"4f76b032-61eb-4796-b082-47c23ff73d18\") " pod="openshift-dns/dns-default-wbzfc" Nov 04 02:32:24.146874 ip-10-0-134-125 kubenswrapper[2579]: I1104 02:32:24.146842 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/4f76b032-61eb-4796-b082-47c23ff73d18-config-volume\") pod \"dns-default-wbzfc\" (UID: \"4f76b032-61eb-4796-b082-47c23ff73d18\") " pod="openshift-dns/dns-default-wbzfc" Nov 04 02:32:24.150145 ip-10-0-134-125 kubenswrapper[2579]: I1104 02:32:24.150119 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-krvqk" Nov 04 02:32:24.150145 ip-10-0-134-125 kubenswrapper[2579]: I1104 02:32:24.150126 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zd8z9" Nov 04 02:32:24.150293 ip-10-0-134-125 kubenswrapper[2579]: I1104 02:32:24.150238 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bdcds" Nov 04 02:32:24.150567 ip-10-0-134-125 kubenswrapper[2579]: I1104 02:32:24.150542 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/a245d359-742e-489d-9c58-eea4e686286c-cert\") pod \"ingress-canary-mn7dn\" (UID: \"a245d359-742e-489d-9c58-eea4e686286c\") " pod="openshift-ingress-canary/ingress-canary-mn7dn" Nov 04 02:32:24.202724 ip-10-0-134-125 kubenswrapper[2579]: I1104 02:32:24.202689 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Nov 04 02:32:24.202848 ip-10-0-134-125 kubenswrapper[2579]: I1104 02:32:24.202817 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Nov 04 02:32:24.202938 ip-10-0-134-125 kubenswrapper[2579]: I1104 02:32:24.202922 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Nov 04 02:32:24.202994 ip-10-0-134-125 kubenswrapper[2579]: I1104 02:32:24.202974 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-8zwfm\"" Nov 04 02:32:24.203036 ip-10-0-134-125 kubenswrapper[2579]: I1104 02:32:24.202988 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Nov 04 02:32:24.203036 ip-10-0-134-125 kubenswrapper[2579]: I1104 02:32:24.202979 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-82bnw\"" Nov 04 02:32:24.203036 ip-10-0-134-125 kubenswrapper[2579]: I1104 02:32:24.202980 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-9fmkz\"" Nov 04 02:32:24.207302 ip-10-0-134-125 kubenswrapper[2579]: I1104 02:32:24.207281 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8lqpv\" (UniqueName: \"kubernetes.io/projected/4f76b032-61eb-4796-b082-47c23ff73d18-kube-api-access-8lqpv\") pod \"dns-default-wbzfc\" (UID: \"4f76b032-61eb-4796-b082-47c23ff73d18\") " pod="openshift-dns/dns-default-wbzfc" Nov 04 02:32:24.207488 ip-10-0-134-125 kubenswrapper[2579]: I1104 02:32:24.207465 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2rpzn\" (UniqueName: \"kubernetes.io/projected/a245d359-742e-489d-9c58-eea4e686286c-kube-api-access-2rpzn\") pod \"ingress-canary-mn7dn\" (UID: \"a245d359-742e-489d-9c58-eea4e686286c\") " pod="openshift-ingress-canary/ingress-canary-mn7dn" Nov 04 02:32:24.329290 ip-10-0-134-125 kubenswrapper[2579]: I1104 02:32:24.329254 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-mn7dn" Nov 04 02:32:24.650962 ip-10-0-134-125 kubenswrapper[2579]: I1104 02:32:24.650922 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/4f76b032-61eb-4796-b082-47c23ff73d18-metrics-tls\") pod \"dns-default-wbzfc\" (UID: \"4f76b032-61eb-4796-b082-47c23ff73d18\") " pod="openshift-dns/dns-default-wbzfc" Nov 04 02:32:24.651139 ip-10-0-134-125 kubenswrapper[2579]: E1104 02:32:24.651035 2579 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Nov 04 02:32:24.651139 ip-10-0-134-125 kubenswrapper[2579]: E1104 02:32:24.651086 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4f76b032-61eb-4796-b082-47c23ff73d18-metrics-tls podName:4f76b032-61eb-4796-b082-47c23ff73d18 nodeName:}" failed. No retries permitted until 2026-11-04 02:32:25.651074379 +0000 UTC m=+32.006181473 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/4f76b032-61eb-4796-b082-47c23ff73d18-metrics-tls") pod "dns-default-wbzfc" (UID: "4f76b032-61eb-4796-b082-47c23ff73d18") : secret "dns-default-metrics-tls" not found Nov 04 02:32:24.710262 ip-10-0-134-125 kubenswrapper[2579]: I1104 02:32:24.710224 2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-mn7dn"] Nov 04 02:32:24.714812 ip-10-0-134-125 kubenswrapper[2579]: W1104 02:32:24.714782 2579 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda245d359_742e_489d_9c58_eea4e686286c.slice/crio-8c8e7de721ec2b40adbe23f38728b8f083e9ec26f592a50f20039593ff57d23b WatchSource:0}: Error finding container 8c8e7de721ec2b40adbe23f38728b8f083e9ec26f592a50f20039593ff57d23b: Status 404 returned error can't find the container with id 8c8e7de721ec2b40adbe23f38728b8f083e9ec26f592a50f20039593ff57d23b Nov 04 02:32:25.385523 ip-10-0-134-125 kubenswrapper[2579]: I1104 02:32:25.385494 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-mn7dn" event={"ID":"a245d359-742e-489d-9c58-eea4e686286c","Type":"ContainerStarted","Data":"8c8e7de721ec2b40adbe23f38728b8f083e9ec26f592a50f20039593ff57d23b"} Nov 04 02:32:25.658318 ip-10-0-134-125 kubenswrapper[2579]: I1104 02:32:25.658233 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/4f76b032-61eb-4796-b082-47c23ff73d18-metrics-tls\") pod \"dns-default-wbzfc\" (UID: \"4f76b032-61eb-4796-b082-47c23ff73d18\") " pod="openshift-dns/dns-default-wbzfc" Nov 04 02:32:25.658458 ip-10-0-134-125 kubenswrapper[2579]: E1104 02:32:25.658359 2579 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Nov 04 02:32:25.658458 ip-10-0-134-125 kubenswrapper[2579]: E1104 02:32:25.658410 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4f76b032-61eb-4796-b082-47c23ff73d18-metrics-tls podName:4f76b032-61eb-4796-b082-47c23ff73d18 nodeName:}" failed. No retries permitted until 2026-11-04 02:32:27.65839914 +0000 UTC m=+34.013506234 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/4f76b032-61eb-4796-b082-47c23ff73d18-metrics-tls") pod "dns-default-wbzfc" (UID: "4f76b032-61eb-4796-b082-47c23ff73d18") : secret "dns-default-metrics-tls" not found Nov 04 02:32:27.072509 ip-10-0-134-125 kubenswrapper[2579]: I1104 02:32:27.072467 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b37e5c97-77c6-4bb4-a9e4-e1ae27484457-metrics-certs\") pod \"network-metrics-daemon-bdcds\" (UID: \"b37e5c97-77c6-4bb4-a9e4-e1ae27484457\") " pod="openshift-multus/network-metrics-daemon-bdcds" Nov 04 02:32:27.073048 ip-10-0-134-125 kubenswrapper[2579]: I1104 02:32:27.072557 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/3a7ab1db-ad88-40d9-a7e9-b511bbd4584c-original-pull-secret\") pod \"global-pull-secret-syncer-krvqk\" (UID: \"3a7ab1db-ad88-40d9-a7e9-b511bbd4584c\") " pod="kube-system/global-pull-secret-syncer-krvqk" Nov 04 02:32:27.073048 ip-10-0-134-125 kubenswrapper[2579]: E1104 02:32:27.072602 2579 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found Nov 04 02:32:27.073048 ip-10-0-134-125 kubenswrapper[2579]: E1104 02:32:27.072691 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b37e5c97-77c6-4bb4-a9e4-e1ae27484457-metrics-certs podName:b37e5c97-77c6-4bb4-a9e4-e1ae27484457 nodeName:}" failed. No retries permitted until 2026-11-04 02:32:59.072651732 +0000 UTC m=+65.427758814 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/b37e5c97-77c6-4bb4-a9e4-e1ae27484457-metrics-certs") pod "network-metrics-daemon-bdcds" (UID: "b37e5c97-77c6-4bb4-a9e4-e1ae27484457") : secret "metrics-daemon-secret" not found Nov 04 02:32:27.075440 ip-10-0-134-125 kubenswrapper[2579]: I1104 02:32:27.075414 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/3a7ab1db-ad88-40d9-a7e9-b511bbd4584c-original-pull-secret\") pod \"global-pull-secret-syncer-krvqk\" (UID: \"3a7ab1db-ad88-40d9-a7e9-b511bbd4584c\") " pod="kube-system/global-pull-secret-syncer-krvqk" Nov 04 02:32:27.167212 ip-10-0-134-125 kubenswrapper[2579]: I1104 02:32:27.167171 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-krvqk" Nov 04 02:32:27.173205 ip-10-0-134-125 kubenswrapper[2579]: I1104 02:32:27.173175 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dddmg\" (UniqueName: \"kubernetes.io/projected/6e27a76f-fbdb-4ce6-9df6-659cab9ad06c-kube-api-access-dddmg\") pod \"network-check-target-zd8z9\" (UID: \"6e27a76f-fbdb-4ce6-9df6-659cab9ad06c\") " pod="openshift-network-diagnostics/network-check-target-zd8z9" Nov 04 02:32:27.175995 ip-10-0-134-125 kubenswrapper[2579]: I1104 02:32:27.175971 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dddmg\" (UniqueName: \"kubernetes.io/projected/6e27a76f-fbdb-4ce6-9df6-659cab9ad06c-kube-api-access-dddmg\") pod \"network-check-target-zd8z9\" (UID: \"6e27a76f-fbdb-4ce6-9df6-659cab9ad06c\") " pod="openshift-network-diagnostics/network-check-target-zd8z9" Nov 04 02:32:27.473076 ip-10-0-134-125 kubenswrapper[2579]: I1104 02:32:27.473035 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zd8z9" Nov 04 02:32:27.537277 ip-10-0-134-125 kubenswrapper[2579]: I1104 02:32:27.537250 2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-krvqk"] Nov 04 02:32:27.540834 ip-10-0-134-125 kubenswrapper[2579]: W1104 02:32:27.540806 2579 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3a7ab1db_ad88_40d9_a7e9_b511bbd4584c.slice/crio-eb581f44938bcc48c524533eb00406852e02b243984608e724360926c929b6c6 WatchSource:0}: Error finding container eb581f44938bcc48c524533eb00406852e02b243984608e724360926c929b6c6: Status 404 returned error can't find the container with id eb581f44938bcc48c524533eb00406852e02b243984608e724360926c929b6c6 Nov 04 02:32:27.678233 ip-10-0-134-125 kubenswrapper[2579]: I1104 02:32:27.678198 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/4f76b032-61eb-4796-b082-47c23ff73d18-metrics-tls\") pod \"dns-default-wbzfc\" (UID: \"4f76b032-61eb-4796-b082-47c23ff73d18\") " pod="openshift-dns/dns-default-wbzfc" Nov 04 02:32:27.678400 ip-10-0-134-125 kubenswrapper[2579]: E1104 02:32:27.678336 2579 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Nov 04 02:32:27.678450 ip-10-0-134-125 kubenswrapper[2579]: E1104 02:32:27.678401 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4f76b032-61eb-4796-b082-47c23ff73d18-metrics-tls podName:4f76b032-61eb-4796-b082-47c23ff73d18 nodeName:}" failed. No retries permitted until 2026-11-04 02:32:31.678386257 +0000 UTC m=+38.033493354 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/4f76b032-61eb-4796-b082-47c23ff73d18-metrics-tls") pod "dns-default-wbzfc" (UID: "4f76b032-61eb-4796-b082-47c23ff73d18") : secret "dns-default-metrics-tls" not found Nov 04 02:32:27.864395 ip-10-0-134-125 kubenswrapper[2579]: I1104 02:32:27.864365 2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-zd8z9"] Nov 04 02:32:27.867851 ip-10-0-134-125 kubenswrapper[2579]: W1104 02:32:27.867818 2579 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6e27a76f_fbdb_4ce6_9df6_659cab9ad06c.slice/crio-469d4eddcf15bea0eb81f4a450042830e9b4e23d956d0f061ab35341b066dcd5 WatchSource:0}: Error finding container 469d4eddcf15bea0eb81f4a450042830e9b4e23d956d0f061ab35341b066dcd5: Status 404 returned error can't find the container with id 469d4eddcf15bea0eb81f4a450042830e9b4e23d956d0f061ab35341b066dcd5 Nov 04 02:32:28.393535 ip-10-0-134-125 kubenswrapper[2579]: I1104 02:32:28.393453 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-krvqk" event={"ID":"3a7ab1db-ad88-40d9-a7e9-b511bbd4584c","Type":"ContainerStarted","Data":"eb581f44938bcc48c524533eb00406852e02b243984608e724360926c929b6c6"} Nov 04 02:32:28.394844 ip-10-0-134-125 kubenswrapper[2579]: I1104 02:32:28.394812 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-zd8z9" event={"ID":"6e27a76f-fbdb-4ce6-9df6-659cab9ad06c","Type":"ContainerStarted","Data":"469d4eddcf15bea0eb81f4a450042830e9b4e23d956d0f061ab35341b066dcd5"} Nov 04 02:32:30.069416 ip-10-0-134-125 kubenswrapper[2579]: I1104 02:32:30.069338 2579 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-5f2vb"] Nov 04 02:32:30.087567 ip-10-0-134-125 kubenswrapper[2579]: I1104 02:32:30.087531 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-5f2vb" Nov 04 02:32:30.140235 ip-10-0-134-125 kubenswrapper[2579]: I1104 02:32:30.140198 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Nov 04 02:32:30.190128 ip-10-0-134-125 kubenswrapper[2579]: I1104 02:32:30.190090 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Nov 04 02:32:30.190300 ip-10-0-134-125 kubenswrapper[2579]: I1104 02:32:30.190223 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-wt2vt\"" Nov 04 02:32:30.191140 ip-10-0-134-125 kubenswrapper[2579]: I1104 02:32:30.191117 2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-5f2vb"] Nov 04 02:32:30.198683 ip-10-0-134-125 kubenswrapper[2579]: I1104 02:32:30.198641 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/4acab1fc-c591-475a-a15f-790be8131547-crio-socket\") pod \"insights-runtime-extractor-5f2vb\" (UID: \"4acab1fc-c591-475a-a15f-790be8131547\") " pod="openshift-insights/insights-runtime-extractor-5f2vb" Nov 04 02:32:30.198795 ip-10-0-134-125 kubenswrapper[2579]: I1104 02:32:30.198756 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bkf4l\" (UniqueName: \"kubernetes.io/projected/4acab1fc-c591-475a-a15f-790be8131547-kube-api-access-bkf4l\") pod \"insights-runtime-extractor-5f2vb\" (UID: \"4acab1fc-c591-475a-a15f-790be8131547\") " pod="openshift-insights/insights-runtime-extractor-5f2vb" Nov 04 02:32:30.198795 ip-10-0-134-125 kubenswrapper[2579]: I1104 02:32:30.198787 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/4acab1fc-c591-475a-a15f-790be8131547-data-volume\") pod \"insights-runtime-extractor-5f2vb\" (UID: \"4acab1fc-c591-475a-a15f-790be8131547\") " pod="openshift-insights/insights-runtime-extractor-5f2vb" Nov 04 02:32:30.198904 ip-10-0-134-125 kubenswrapper[2579]: I1104 02:32:30.198811 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/4acab1fc-c591-475a-a15f-790be8131547-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-5f2vb\" (UID: \"4acab1fc-c591-475a-a15f-790be8131547\") " pod="openshift-insights/insights-runtime-extractor-5f2vb" Nov 04 02:32:30.198904 ip-10-0-134-125 kubenswrapper[2579]: I1104 02:32:30.198843 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/4acab1fc-c591-475a-a15f-790be8131547-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-5f2vb\" (UID: \"4acab1fc-c591-475a-a15f-790be8131547\") " pod="openshift-insights/insights-runtime-extractor-5f2vb" Nov 04 02:32:30.240639 ip-10-0-134-125 kubenswrapper[2579]: I1104 02:32:30.240599 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Nov 04 02:32:30.240801 ip-10-0-134-125 kubenswrapper[2579]: I1104 02:32:30.240681 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Nov 04 02:32:30.299966 ip-10-0-134-125 kubenswrapper[2579]: I1104 02:32:30.299928 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/4acab1fc-c591-475a-a15f-790be8131547-data-volume\") pod \"insights-runtime-extractor-5f2vb\" (UID: \"4acab1fc-c591-475a-a15f-790be8131547\") " pod="openshift-insights/insights-runtime-extractor-5f2vb" Nov 04 02:32:30.299966 ip-10-0-134-125 kubenswrapper[2579]: I1104 02:32:30.299972 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/4acab1fc-c591-475a-a15f-790be8131547-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-5f2vb\" (UID: \"4acab1fc-c591-475a-a15f-790be8131547\") " pod="openshift-insights/insights-runtime-extractor-5f2vb" Nov 04 02:32:30.300224 ip-10-0-134-125 kubenswrapper[2579]: I1104 02:32:30.300017 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/4acab1fc-c591-475a-a15f-790be8131547-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-5f2vb\" (UID: \"4acab1fc-c591-475a-a15f-790be8131547\") " pod="openshift-insights/insights-runtime-extractor-5f2vb" Nov 04 02:32:30.300224 ip-10-0-134-125 kubenswrapper[2579]: I1104 02:32:30.300054 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/4acab1fc-c591-475a-a15f-790be8131547-crio-socket\") pod \"insights-runtime-extractor-5f2vb\" (UID: \"4acab1fc-c591-475a-a15f-790be8131547\") " pod="openshift-insights/insights-runtime-extractor-5f2vb" Nov 04 02:32:30.300224 ip-10-0-134-125 kubenswrapper[2579]: I1104 02:32:30.300185 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bkf4l\" (UniqueName: \"kubernetes.io/projected/4acab1fc-c591-475a-a15f-790be8131547-kube-api-access-bkf4l\") pod \"insights-runtime-extractor-5f2vb\" (UID: \"4acab1fc-c591-475a-a15f-790be8131547\") " pod="openshift-insights/insights-runtime-extractor-5f2vb" Nov 04 02:32:30.301041 ip-10-0-134-125 kubenswrapper[2579]: I1104 02:32:30.300976 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/4acab1fc-c591-475a-a15f-790be8131547-data-volume\") pod \"insights-runtime-extractor-5f2vb\" (UID: \"4acab1fc-c591-475a-a15f-790be8131547\") " pod="openshift-insights/insights-runtime-extractor-5f2vb" Nov 04 02:32:30.301179 ip-10-0-134-125 kubenswrapper[2579]: E1104 02:32:30.301125 2579 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Nov 04 02:32:30.301246 ip-10-0-134-125 kubenswrapper[2579]: E1104 02:32:30.301220 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/4acab1fc-c591-475a-a15f-790be8131547-kube-rbac-proxy-cm podName:4acab1fc-c591-475a-a15f-790be8131547 nodeName:}" failed. No retries permitted until 2026-11-04 02:32:30.801183859 +0000 UTC m=+37.156290956 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/4acab1fc-c591-475a-a15f-790be8131547-kube-rbac-proxy-cm") pod "insights-runtime-extractor-5f2vb" (UID: "4acab1fc-c591-475a-a15f-790be8131547") : configmap "kube-rbac-proxy" not found Nov 04 02:32:30.301494 ip-10-0-134-125 kubenswrapper[2579]: E1104 02:32:30.301479 2579 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Nov 04 02:32:30.301638 ip-10-0-134-125 kubenswrapper[2579]: E1104 02:32:30.301524 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4acab1fc-c591-475a-a15f-790be8131547-insights-runtime-extractor-tls podName:4acab1fc-c591-475a-a15f-790be8131547 nodeName:}" failed. No retries permitted until 2026-11-04 02:32:30.80151194 +0000 UTC m=+37.156619034 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/4acab1fc-c591-475a-a15f-790be8131547-insights-runtime-extractor-tls") pod "insights-runtime-extractor-5f2vb" (UID: "4acab1fc-c591-475a-a15f-790be8131547") : secret "insights-runtime-extractor-tls" not found Nov 04 02:32:30.301728 ip-10-0-134-125 kubenswrapper[2579]: I1104 02:32:30.301707 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/4acab1fc-c591-475a-a15f-790be8131547-crio-socket\") pod \"insights-runtime-extractor-5f2vb\" (UID: \"4acab1fc-c591-475a-a15f-790be8131547\") " pod="openshift-insights/insights-runtime-extractor-5f2vb" Nov 04 02:32:30.321023 ip-10-0-134-125 kubenswrapper[2579]: I1104 02:32:30.320955 2579 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/periodic-gathering-hxwj2-xcshl"] Nov 04 02:32:30.333298 ip-10-0-134-125 kubenswrapper[2579]: I1104 02:32:30.333272 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-hxwj2-xcshl" Nov 04 02:32:30.362356 ip-10-0-134-125 kubenswrapper[2579]: I1104 02:32:30.362328 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bkf4l\" (UniqueName: \"kubernetes.io/projected/4acab1fc-c591-475a-a15f-790be8131547-kube-api-access-bkf4l\") pod \"insights-runtime-extractor-5f2vb\" (UID: \"4acab1fc-c591-475a-a15f-790be8131547\") " pod="openshift-insights/insights-runtime-extractor-5f2vb" Nov 04 02:32:30.384975 ip-10-0-134-125 kubenswrapper[2579]: I1104 02:32:30.384910 2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/periodic-gathering-hxwj2-xcshl"] Nov 04 02:32:30.386409 ip-10-0-134-125 kubenswrapper[2579]: I1104 02:32:30.386374 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"service-ca-bundle\"" Nov 04 02:32:30.386716 ip-10-0-134-125 kubenswrapper[2579]: I1104 02:32:30.386696 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"operator-dockercfg-h7xc9\"" Nov 04 02:32:30.400488 ip-10-0-134-125 kubenswrapper[2579]: I1104 02:32:30.400442 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h288h\" (UniqueName: \"kubernetes.io/projected/5a5c4b25-7a35-4290-ad1e-04b4a3617877-kube-api-access-h288h\") pod \"periodic-gathering-hxwj2-xcshl\" (UID: \"5a5c4b25-7a35-4290-ad1e-04b4a3617877\") " pod="openshift-insights/periodic-gathering-hxwj2-xcshl" Nov 04 02:32:30.400633 ip-10-0-134-125 kubenswrapper[2579]: I1104 02:32:30.400508 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5a5c4b25-7a35-4290-ad1e-04b4a3617877-service-ca-bundle\") pod \"periodic-gathering-hxwj2-xcshl\" (UID: \"5a5c4b25-7a35-4290-ad1e-04b4a3617877\") " pod="openshift-insights/periodic-gathering-hxwj2-xcshl" Nov 04 02:32:30.400723 ip-10-0-134-125 kubenswrapper[2579]: I1104 02:32:30.400688 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/5a5c4b25-7a35-4290-ad1e-04b4a3617877-archives-path\") pod \"periodic-gathering-hxwj2-xcshl\" (UID: \"5a5c4b25-7a35-4290-ad1e-04b4a3617877\") " pod="openshift-insights/periodic-gathering-hxwj2-xcshl" Nov 04 02:32:30.400767 ip-10-0-134-125 kubenswrapper[2579]: I1104 02:32:30.400742 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5a5c4b25-7a35-4290-ad1e-04b4a3617877-trusted-ca-bundle\") pod \"periodic-gathering-hxwj2-xcshl\" (UID: \"5a5c4b25-7a35-4290-ad1e-04b4a3617877\") " pod="openshift-insights/periodic-gathering-hxwj2-xcshl" Nov 04 02:32:30.401342 ip-10-0-134-125 kubenswrapper[2579]: I1104 02:32:30.401308 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-mn7dn" event={"ID":"a245d359-742e-489d-9c58-eea4e686286c","Type":"ContainerStarted","Data":"152bd9bcb54a5a2cd86afb683c769fc3af8ad9387c526b205d05b7bff684c81f"} Nov 04 02:32:30.447899 ip-10-0-134-125 kubenswrapper[2579]: I1104 02:32:30.447842 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"trusted-ca-bundle\"" Nov 04 02:32:30.502815 ip-10-0-134-125 kubenswrapper[2579]: I1104 02:32:30.501827 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/5a5c4b25-7a35-4290-ad1e-04b4a3617877-archives-path\") pod \"periodic-gathering-hxwj2-xcshl\" (UID: \"5a5c4b25-7a35-4290-ad1e-04b4a3617877\") " pod="openshift-insights/periodic-gathering-hxwj2-xcshl" Nov 04 02:32:30.502815 ip-10-0-134-125 kubenswrapper[2579]: I1104 02:32:30.501925 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5a5c4b25-7a35-4290-ad1e-04b4a3617877-trusted-ca-bundle\") pod \"periodic-gathering-hxwj2-xcshl\" (UID: \"5a5c4b25-7a35-4290-ad1e-04b4a3617877\") " pod="openshift-insights/periodic-gathering-hxwj2-xcshl" Nov 04 02:32:30.502815 ip-10-0-134-125 kubenswrapper[2579]: I1104 02:32:30.501957 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h288h\" (UniqueName: \"kubernetes.io/projected/5a5c4b25-7a35-4290-ad1e-04b4a3617877-kube-api-access-h288h\") pod \"periodic-gathering-hxwj2-xcshl\" (UID: \"5a5c4b25-7a35-4290-ad1e-04b4a3617877\") " pod="openshift-insights/periodic-gathering-hxwj2-xcshl" Nov 04 02:32:30.502815 ip-10-0-134-125 kubenswrapper[2579]: I1104 02:32:30.502003 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5a5c4b25-7a35-4290-ad1e-04b4a3617877-service-ca-bundle\") pod \"periodic-gathering-hxwj2-xcshl\" (UID: \"5a5c4b25-7a35-4290-ad1e-04b4a3617877\") " pod="openshift-insights/periodic-gathering-hxwj2-xcshl" Nov 04 02:32:30.502815 ip-10-0-134-125 kubenswrapper[2579]: I1104 02:32:30.502764 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5a5c4b25-7a35-4290-ad1e-04b4a3617877-service-ca-bundle\") pod \"periodic-gathering-hxwj2-xcshl\" (UID: \"5a5c4b25-7a35-4290-ad1e-04b4a3617877\") " pod="openshift-insights/periodic-gathering-hxwj2-xcshl" Nov 04 02:32:30.503188 ip-10-0-134-125 kubenswrapper[2579]: I1104 02:32:30.502997 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/5a5c4b25-7a35-4290-ad1e-04b4a3617877-archives-path\") pod \"periodic-gathering-hxwj2-xcshl\" (UID: \"5a5c4b25-7a35-4290-ad1e-04b4a3617877\") " pod="openshift-insights/periodic-gathering-hxwj2-xcshl" Nov 04 02:32:30.503458 ip-10-0-134-125 kubenswrapper[2579]: I1104 02:32:30.503438 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5a5c4b25-7a35-4290-ad1e-04b4a3617877-trusted-ca-bundle\") pod \"periodic-gathering-hxwj2-xcshl\" (UID: \"5a5c4b25-7a35-4290-ad1e-04b4a3617877\") " pod="openshift-insights/periodic-gathering-hxwj2-xcshl" Nov 04 02:32:30.564987 ip-10-0-134-125 kubenswrapper[2579]: I1104 02:32:30.564948 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h288h\" (UniqueName: \"kubernetes.io/projected/5a5c4b25-7a35-4290-ad1e-04b4a3617877-kube-api-access-h288h\") pod \"periodic-gathering-hxwj2-xcshl\" (UID: \"5a5c4b25-7a35-4290-ad1e-04b4a3617877\") " pod="openshift-insights/periodic-gathering-hxwj2-xcshl" Nov 04 02:32:30.572887 ip-10-0-134-125 kubenswrapper[2579]: I1104 02:32:30.572785 2579 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-mn7dn" podStartSLOduration=2.6229728 podStartE2EDuration="7.572764767s" totalImagesPullingTime="4.949791967s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 02:32:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 02:32:30.572013498 +0000 UTC m=+36.927120605" watchObservedRunningTime="2026-11-04 02:32:30.572764767 +0000 UTC m=+36.927871869" Nov 04 02:32:30.656907 ip-10-0-134-125 kubenswrapper[2579]: I1104 02:32:30.656857 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-hxwj2-xcshl" Nov 04 02:32:30.804397 ip-10-0-134-125 kubenswrapper[2579]: I1104 02:32:30.804321 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/4acab1fc-c591-475a-a15f-790be8131547-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-5f2vb\" (UID: \"4acab1fc-c591-475a-a15f-790be8131547\") " pod="openshift-insights/insights-runtime-extractor-5f2vb" Nov 04 02:32:30.804397 ip-10-0-134-125 kubenswrapper[2579]: I1104 02:32:30.804375 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/4acab1fc-c591-475a-a15f-790be8131547-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-5f2vb\" (UID: \"4acab1fc-c591-475a-a15f-790be8131547\") " pod="openshift-insights/insights-runtime-extractor-5f2vb" Nov 04 02:32:30.804660 ip-10-0-134-125 kubenswrapper[2579]: E1104 02:32:30.804500 2579 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Nov 04 02:32:30.804660 ip-10-0-134-125 kubenswrapper[2579]: E1104 02:32:30.804574 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4acab1fc-c591-475a-a15f-790be8131547-insights-runtime-extractor-tls podName:4acab1fc-c591-475a-a15f-790be8131547 nodeName:}" failed. No retries permitted until 2026-11-04 02:32:31.804559452 +0000 UTC m=+38.159666552 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/4acab1fc-c591-475a-a15f-790be8131547-insights-runtime-extractor-tls") pod "insights-runtime-extractor-5f2vb" (UID: "4acab1fc-c591-475a-a15f-790be8131547") : secret "insights-runtime-extractor-tls" not found Nov 04 02:32:30.804932 ip-10-0-134-125 kubenswrapper[2579]: E1104 02:32:30.804916 2579 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Nov 04 02:32:30.805041 ip-10-0-134-125 kubenswrapper[2579]: E1104 02:32:30.804970 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/4acab1fc-c591-475a-a15f-790be8131547-kube-rbac-proxy-cm podName:4acab1fc-c591-475a-a15f-790be8131547 nodeName:}" failed. No retries permitted until 2026-11-04 02:32:31.804954953 +0000 UTC m=+38.160062049 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/4acab1fc-c591-475a-a15f-790be8131547-kube-rbac-proxy-cm") pod "insights-runtime-extractor-5f2vb" (UID: "4acab1fc-c591-475a-a15f-790be8131547") : configmap "kube-rbac-proxy" not found Nov 04 02:32:31.080978 ip-10-0-134-125 kubenswrapper[2579]: I1104 02:32:31.080937 2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/periodic-gathering-hxwj2-xcshl"] Nov 04 02:32:31.088122 ip-10-0-134-125 kubenswrapper[2579]: W1104 02:32:31.088092 2579 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5a5c4b25_7a35_4290_ad1e_04b4a3617877.slice/crio-7f29a771e6ee4b7e85aa9aa1134769a08873c5be8f3c263500189b2051fd67b8 WatchSource:0}: Error finding container 7f29a771e6ee4b7e85aa9aa1134769a08873c5be8f3c263500189b2051fd67b8: Status 404 returned error can't find the container with id 7f29a771e6ee4b7e85aa9aa1134769a08873c5be8f3c263500189b2051fd67b8 Nov 04 02:32:31.406255 ip-10-0-134-125 kubenswrapper[2579]: I1104 02:32:31.406207 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-hxwj2-xcshl" event={"ID":"5a5c4b25-7a35-4290-ad1e-04b4a3617877","Type":"ContainerStarted","Data":"7f29a771e6ee4b7e85aa9aa1134769a08873c5be8f3c263500189b2051fd67b8"} Nov 04 02:32:31.712965 ip-10-0-134-125 kubenswrapper[2579]: I1104 02:32:31.712890 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/4f76b032-61eb-4796-b082-47c23ff73d18-metrics-tls\") pod \"dns-default-wbzfc\" (UID: \"4f76b032-61eb-4796-b082-47c23ff73d18\") " pod="openshift-dns/dns-default-wbzfc" Nov 04 02:32:31.713114 ip-10-0-134-125 kubenswrapper[2579]: E1104 02:32:31.713069 2579 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Nov 04 02:32:31.713168 ip-10-0-134-125 kubenswrapper[2579]: E1104 02:32:31.713152 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4f76b032-61eb-4796-b082-47c23ff73d18-metrics-tls podName:4f76b032-61eb-4796-b082-47c23ff73d18 nodeName:}" failed. No retries permitted until 2026-11-04 02:32:39.713131835 +0000 UTC m=+46.068238933 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/4f76b032-61eb-4796-b082-47c23ff73d18-metrics-tls") pod "dns-default-wbzfc" (UID: "4f76b032-61eb-4796-b082-47c23ff73d18") : secret "dns-default-metrics-tls" not found Nov 04 02:32:31.813312 ip-10-0-134-125 kubenswrapper[2579]: I1104 02:32:31.813271 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/4acab1fc-c591-475a-a15f-790be8131547-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-5f2vb\" (UID: \"4acab1fc-c591-475a-a15f-790be8131547\") " pod="openshift-insights/insights-runtime-extractor-5f2vb" Nov 04 02:32:31.813491 ip-10-0-134-125 kubenswrapper[2579]: I1104 02:32:31.813390 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/4acab1fc-c591-475a-a15f-790be8131547-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-5f2vb\" (UID: \"4acab1fc-c591-475a-a15f-790be8131547\") " pod="openshift-insights/insights-runtime-extractor-5f2vb" Nov 04 02:32:31.813491 ip-10-0-134-125 kubenswrapper[2579]: E1104 02:32:31.813451 2579 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Nov 04 02:32:31.813600 ip-10-0-134-125 kubenswrapper[2579]: E1104 02:32:31.813530 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4acab1fc-c591-475a-a15f-790be8131547-insights-runtime-extractor-tls podName:4acab1fc-c591-475a-a15f-790be8131547 nodeName:}" failed. No retries permitted until 2026-11-04 02:32:33.813513819 +0000 UTC m=+40.168620919 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/4acab1fc-c591-475a-a15f-790be8131547-insights-runtime-extractor-tls") pod "insights-runtime-extractor-5f2vb" (UID: "4acab1fc-c591-475a-a15f-790be8131547") : secret "insights-runtime-extractor-tls" not found Nov 04 02:32:31.814024 ip-10-0-134-125 kubenswrapper[2579]: I1104 02:32:31.814001 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/4acab1fc-c591-475a-a15f-790be8131547-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-5f2vb\" (UID: \"4acab1fc-c591-475a-a15f-790be8131547\") " pod="openshift-insights/insights-runtime-extractor-5f2vb" Nov 04 02:32:32.410606 ip-10-0-134-125 kubenswrapper[2579]: I1104 02:32:32.410561 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-zd8z9" event={"ID":"6e27a76f-fbdb-4ce6-9df6-659cab9ad06c","Type":"ContainerStarted","Data":"59106576a6de837bc701aa035bc1a07d5ffe7387825e05c6e007b92de6e11b3f"} Nov 04 02:32:32.411021 ip-10-0-134-125 kubenswrapper[2579]: I1104 02:32:32.410643 2579 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-zd8z9" Nov 04 02:32:32.526702 ip-10-0-134-125 kubenswrapper[2579]: I1104 02:32:32.526638 2579 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-zd8z9" podStartSLOduration=34.429227484 podStartE2EDuration="38.526622403s" totalImagesPullingTime="4.097394919s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 02:31:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 02:32:32.525860274 +0000 UTC m=+38.880967382" watchObservedRunningTime="2026-11-04 02:32:32.526622403 +0000 UTC m=+38.881729533" Nov 04 02:32:33.415080 ip-10-0-134-125 kubenswrapper[2579]: I1104 02:32:33.414994 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-krvqk" event={"ID":"3a7ab1db-ad88-40d9-a7e9-b511bbd4584c","Type":"ContainerStarted","Data":"c74118c420743fba218629825b1380c856b9f61b0f8feda4dbb2b584eb62e759"} Nov 04 02:32:33.484036 ip-10-0-134-125 systemd[1]: Stopping Kubernetes Kubelet... Nov 04 02:32:33.497138 ip-10-0-134-125 systemd[1]: kubelet.service: Deactivated successfully. Nov 04 02:32:33.497361 ip-10-0-134-125 systemd[1]: Stopped Kubernetes Kubelet. Nov 04 02:32:33.497453 ip-10-0-134-125 systemd[1]: kubelet.service: Consumed 2.364s CPU time. Nov 04 02:32:33.513419 ip-10-0-134-125 systemd[1]: Starting Kubernetes Kubelet... Nov 04 02:32:33.613206 ip-10-0-134-125 kubenswrapper[4405]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Nov 04 02:32:33.613206 ip-10-0-134-125 kubenswrapper[4405]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Nov 04 02:32:33.613206 ip-10-0-134-125 kubenswrapper[4405]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Nov 04 02:32:33.613206 ip-10-0-134-125 kubenswrapper[4405]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Nov 04 02:32:33.615346 ip-10-0-134-125 kubenswrapper[4405]: W1104 02:32:33.615265 4405 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 04 02:32:33.615346 ip-10-0-134-125 kubenswrapper[4405]: W1104 02:32:33.615347 4405 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 04 02:32:33.615425 ip-10-0-134-125 kubenswrapper[4405]: W1104 02:32:33.615351 4405 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 04 02:32:33.615425 ip-10-0-134-125 kubenswrapper[4405]: W1104 02:32:33.615354 4405 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 04 02:32:33.615425 ip-10-0-134-125 kubenswrapper[4405]: W1104 02:32:33.615357 4405 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 04 02:32:33.615425 ip-10-0-134-125 kubenswrapper[4405]: W1104 02:32:33.615360 4405 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 04 02:32:33.615425 ip-10-0-134-125 kubenswrapper[4405]: W1104 02:32:33.615362 4405 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 04 02:32:33.615425 ip-10-0-134-125 kubenswrapper[4405]: W1104 02:32:33.615366 4405 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 04 02:32:33.615425 ip-10-0-134-125 kubenswrapper[4405]: W1104 02:32:33.615369 4405 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 04 02:32:33.615425 ip-10-0-134-125 kubenswrapper[4405]: W1104 02:32:33.615371 4405 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 04 02:32:33.615425 ip-10-0-134-125 kubenswrapper[4405]: W1104 02:32:33.615374 4405 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 04 02:32:33.615425 ip-10-0-134-125 kubenswrapper[4405]: W1104 02:32:33.615377 4405 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 04 02:32:33.615425 ip-10-0-134-125 kubenswrapper[4405]: W1104 02:32:33.615379 4405 feature_gate.go:386] unrecognized feature gate: EVPN Nov 04 02:32:33.615425 ip-10-0-134-125 kubenswrapper[4405]: W1104 02:32:33.615382 4405 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 04 02:32:33.615425 ip-10-0-134-125 kubenswrapper[4405]: W1104 02:32:33.615384 4405 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 04 02:32:33.615425 ip-10-0-134-125 kubenswrapper[4405]: W1104 02:32:33.615387 4405 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 04 02:32:33.615425 ip-10-0-134-125 kubenswrapper[4405]: W1104 02:32:33.615389 4405 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 04 02:32:33.615425 ip-10-0-134-125 kubenswrapper[4405]: W1104 02:32:33.615392 4405 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 04 02:32:33.615425 ip-10-0-134-125 kubenswrapper[4405]: W1104 02:32:33.615395 4405 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 04 02:32:33.615425 ip-10-0-134-125 kubenswrapper[4405]: W1104 02:32:33.615398 4405 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 04 02:32:33.615425 ip-10-0-134-125 kubenswrapper[4405]: W1104 02:32:33.615400 4405 feature_gate.go:386] unrecognized feature gate: Example Nov 04 02:32:33.615425 ip-10-0-134-125 kubenswrapper[4405]: W1104 02:32:33.615403 4405 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 04 02:32:33.615979 ip-10-0-134-125 kubenswrapper[4405]: W1104 02:32:33.615406 4405 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 04 02:32:33.615979 ip-10-0-134-125 kubenswrapper[4405]: W1104 02:32:33.615416 4405 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 04 02:32:33.615979 ip-10-0-134-125 kubenswrapper[4405]: W1104 02:32:33.615419 4405 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 04 02:32:33.615979 ip-10-0-134-125 kubenswrapper[4405]: W1104 02:32:33.615422 4405 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 04 02:32:33.615979 ip-10-0-134-125 kubenswrapper[4405]: W1104 02:32:33.615424 4405 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 04 02:32:33.615979 ip-10-0-134-125 kubenswrapper[4405]: W1104 02:32:33.615427 4405 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 04 02:32:33.615979 ip-10-0-134-125 kubenswrapper[4405]: W1104 02:32:33.615430 4405 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 04 02:32:33.615979 ip-10-0-134-125 kubenswrapper[4405]: W1104 02:32:33.615432 4405 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 04 02:32:33.615979 ip-10-0-134-125 kubenswrapper[4405]: W1104 02:32:33.615436 4405 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 04 02:32:33.615979 ip-10-0-134-125 kubenswrapper[4405]: W1104 02:32:33.615438 4405 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 04 02:32:33.615979 ip-10-0-134-125 kubenswrapper[4405]: W1104 02:32:33.615441 4405 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 04 02:32:33.615979 ip-10-0-134-125 kubenswrapper[4405]: W1104 02:32:33.615444 4405 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 04 02:32:33.615979 ip-10-0-134-125 kubenswrapper[4405]: W1104 02:32:33.615448 4405 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 04 02:32:33.615979 ip-10-0-134-125 kubenswrapper[4405]: W1104 02:32:33.615452 4405 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 04 02:32:33.615979 ip-10-0-134-125 kubenswrapper[4405]: W1104 02:32:33.615456 4405 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 04 02:32:33.615979 ip-10-0-134-125 kubenswrapper[4405]: W1104 02:32:33.615459 4405 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 04 02:32:33.615979 ip-10-0-134-125 kubenswrapper[4405]: W1104 02:32:33.615461 4405 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 04 02:32:33.615979 ip-10-0-134-125 kubenswrapper[4405]: W1104 02:32:33.615464 4405 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 04 02:32:33.615979 ip-10-0-134-125 kubenswrapper[4405]: W1104 02:32:33.615466 4405 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 04 02:32:33.616488 ip-10-0-134-125 kubenswrapper[4405]: W1104 02:32:33.615469 4405 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 04 02:32:33.616488 ip-10-0-134-125 kubenswrapper[4405]: W1104 02:32:33.615473 4405 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 04 02:32:33.616488 ip-10-0-134-125 kubenswrapper[4405]: W1104 02:32:33.615476 4405 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 04 02:32:33.616488 ip-10-0-134-125 kubenswrapper[4405]: W1104 02:32:33.615478 4405 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 04 02:32:33.616488 ip-10-0-134-125 kubenswrapper[4405]: W1104 02:32:33.615481 4405 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 04 02:32:33.616488 ip-10-0-134-125 kubenswrapper[4405]: W1104 02:32:33.615484 4405 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 04 02:32:33.616488 ip-10-0-134-125 kubenswrapper[4405]: W1104 02:32:33.615487 4405 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 04 02:32:33.616488 ip-10-0-134-125 kubenswrapper[4405]: W1104 02:32:33.615490 4405 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 04 02:32:33.616488 ip-10-0-134-125 kubenswrapper[4405]: W1104 02:32:33.615493 4405 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 04 02:32:33.616488 ip-10-0-134-125 kubenswrapper[4405]: W1104 02:32:33.615497 4405 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 04 02:32:33.616488 ip-10-0-134-125 kubenswrapper[4405]: W1104 02:32:33.615500 4405 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 04 02:32:33.616488 ip-10-0-134-125 kubenswrapper[4405]: W1104 02:32:33.615504 4405 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 04 02:32:33.616488 ip-10-0-134-125 kubenswrapper[4405]: W1104 02:32:33.615507 4405 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 04 02:32:33.616488 ip-10-0-134-125 kubenswrapper[4405]: W1104 02:32:33.615509 4405 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 04 02:32:33.616488 ip-10-0-134-125 kubenswrapper[4405]: W1104 02:32:33.615513 4405 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 04 02:32:33.616488 ip-10-0-134-125 kubenswrapper[4405]: W1104 02:32:33.615515 4405 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 04 02:32:33.616488 ip-10-0-134-125 kubenswrapper[4405]: W1104 02:32:33.615518 4405 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 04 02:32:33.616488 ip-10-0-134-125 kubenswrapper[4405]: W1104 02:32:33.615520 4405 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 04 02:32:33.616488 ip-10-0-134-125 kubenswrapper[4405]: W1104 02:32:33.615523 4405 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 04 02:32:33.616488 ip-10-0-134-125 kubenswrapper[4405]: W1104 02:32:33.615525 4405 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 04 02:32:33.617039 ip-10-0-134-125 kubenswrapper[4405]: W1104 02:32:33.615528 4405 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 04 02:32:33.617039 ip-10-0-134-125 kubenswrapper[4405]: W1104 02:32:33.615532 4405 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 04 02:32:33.617039 ip-10-0-134-125 kubenswrapper[4405]: W1104 02:32:33.615535 4405 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 04 02:32:33.617039 ip-10-0-134-125 kubenswrapper[4405]: W1104 02:32:33.615537 4405 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 04 02:32:33.617039 ip-10-0-134-125 kubenswrapper[4405]: W1104 02:32:33.615540 4405 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 04 02:32:33.617039 ip-10-0-134-125 kubenswrapper[4405]: W1104 02:32:33.615542 4405 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 04 02:32:33.617039 ip-10-0-134-125 kubenswrapper[4405]: W1104 02:32:33.615545 4405 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 04 02:32:33.617039 ip-10-0-134-125 kubenswrapper[4405]: W1104 02:32:33.615548 4405 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 04 02:32:33.617039 ip-10-0-134-125 kubenswrapper[4405]: W1104 02:32:33.615550 4405 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 04 02:32:33.617039 ip-10-0-134-125 kubenswrapper[4405]: W1104 02:32:33.615553 4405 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 04 02:32:33.617039 ip-10-0-134-125 kubenswrapper[4405]: W1104 02:32:33.615555 4405 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 04 02:32:33.617039 ip-10-0-134-125 kubenswrapper[4405]: W1104 02:32:33.615558 4405 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 04 02:32:33.617039 ip-10-0-134-125 kubenswrapper[4405]: W1104 02:32:33.615561 4405 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 04 02:32:33.617039 ip-10-0-134-125 kubenswrapper[4405]: W1104 02:32:33.615563 4405 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 04 02:32:33.617039 ip-10-0-134-125 kubenswrapper[4405]: W1104 02:32:33.615567 4405 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 04 02:32:33.617039 ip-10-0-134-125 kubenswrapper[4405]: W1104 02:32:33.615569 4405 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 04 02:32:33.617039 ip-10-0-134-125 kubenswrapper[4405]: W1104 02:32:33.615572 4405 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 04 02:32:33.617039 ip-10-0-134-125 kubenswrapper[4405]: W1104 02:32:33.615575 4405 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 04 02:32:33.617039 ip-10-0-134-125 kubenswrapper[4405]: W1104 02:32:33.615578 4405 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 04 02:32:33.617543 ip-10-0-134-125 kubenswrapper[4405]: W1104 02:32:33.615580 4405 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 04 02:32:33.617543 ip-10-0-134-125 kubenswrapper[4405]: W1104 02:32:33.615583 4405 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 04 02:32:33.617543 ip-10-0-134-125 kubenswrapper[4405]: W1104 02:32:33.615585 4405 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 04 02:32:33.617543 ip-10-0-134-125 kubenswrapper[4405]: W1104 02:32:33.615588 4405 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 04 02:32:33.617543 ip-10-0-134-125 kubenswrapper[4405]: W1104 02:32:33.615591 4405 feature_gate.go:386] unrecognized feature gate: Example2 Nov 04 02:32:33.617543 ip-10-0-134-125 kubenswrapper[4405]: W1104 02:32:33.615594 4405 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 04 02:32:33.617543 ip-10-0-134-125 kubenswrapper[4405]: W1104 02:32:33.615596 4405 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 04 02:32:33.617543 ip-10-0-134-125 kubenswrapper[4405]: W1104 02:32:33.615599 4405 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 04 02:32:33.617543 ip-10-0-134-125 kubenswrapper[4405]: W1104 02:32:33.615602 4405 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 04 02:32:33.617543 ip-10-0-134-125 kubenswrapper[4405]: W1104 02:32:33.615604 4405 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 04 02:32:33.617543 ip-10-0-134-125 kubenswrapper[4405]: W1104 02:32:33.615607 4405 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 04 02:32:33.617543 ip-10-0-134-125 kubenswrapper[4405]: W1104 02:32:33.615609 4405 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 04 02:32:33.617543 ip-10-0-134-125 kubenswrapper[4405]: W1104 02:32:33.615612 4405 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 04 02:32:33.617543 ip-10-0-134-125 kubenswrapper[4405]: W1104 02:32:33.615615 4405 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 04 02:32:33.617543 ip-10-0-134-125 kubenswrapper[4405]: W1104 02:32:33.615618 4405 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 04 02:32:33.617543 ip-10-0-134-125 kubenswrapper[4405]: W1104 02:32:33.615621 4405 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 04 02:32:33.617543 ip-10-0-134-125 kubenswrapper[4405]: W1104 02:32:33.615623 4405 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 04 02:32:33.617543 ip-10-0-134-125 kubenswrapper[4405]: W1104 02:32:33.615626 4405 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 04 02:32:33.617543 ip-10-0-134-125 kubenswrapper[4405]: W1104 02:32:33.615629 4405 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 04 02:32:33.617543 ip-10-0-134-125 kubenswrapper[4405]: W1104 02:32:33.615631 4405 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 04 02:32:33.618093 ip-10-0-134-125 kubenswrapper[4405]: W1104 02:32:33.615633 4405 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 04 02:32:33.618093 ip-10-0-134-125 kubenswrapper[4405]: W1104 02:32:33.615636 4405 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 04 02:32:33.618093 ip-10-0-134-125 kubenswrapper[4405]: W1104 02:32:33.615638 4405 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 04 02:32:33.618093 ip-10-0-134-125 kubenswrapper[4405]: W1104 02:32:33.615641 4405 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 04 02:32:33.618093 ip-10-0-134-125 kubenswrapper[4405]: W1104 02:32:33.615644 4405 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 04 02:32:33.618093 ip-10-0-134-125 kubenswrapper[4405]: W1104 02:32:33.615646 4405 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 04 02:32:33.618093 ip-10-0-134-125 kubenswrapper[4405]: W1104 02:32:33.616661 4405 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 04 02:32:33.618093 ip-10-0-134-125 kubenswrapper[4405]: W1104 02:32:33.616683 4405 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 04 02:32:33.618093 ip-10-0-134-125 kubenswrapper[4405]: W1104 02:32:33.616688 4405 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 04 02:32:33.618093 ip-10-0-134-125 kubenswrapper[4405]: W1104 02:32:33.616691 4405 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 04 02:32:33.618093 ip-10-0-134-125 kubenswrapper[4405]: W1104 02:32:33.616693 4405 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 04 02:32:33.618093 ip-10-0-134-125 kubenswrapper[4405]: W1104 02:32:33.616696 4405 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 04 02:32:33.618093 ip-10-0-134-125 kubenswrapper[4405]: W1104 02:32:33.616699 4405 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 04 02:32:33.618093 ip-10-0-134-125 kubenswrapper[4405]: W1104 02:32:33.616701 4405 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 04 02:32:33.618093 ip-10-0-134-125 kubenswrapper[4405]: W1104 02:32:33.616705 4405 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 04 02:32:33.618093 ip-10-0-134-125 kubenswrapper[4405]: W1104 02:32:33.616707 4405 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 04 02:32:33.618093 ip-10-0-134-125 kubenswrapper[4405]: W1104 02:32:33.616710 4405 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 04 02:32:33.618093 ip-10-0-134-125 kubenswrapper[4405]: W1104 02:32:33.616713 4405 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 04 02:32:33.618093 ip-10-0-134-125 kubenswrapper[4405]: W1104 02:32:33.616716 4405 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 04 02:32:33.618093 ip-10-0-134-125 kubenswrapper[4405]: W1104 02:32:33.616718 4405 feature_gate.go:386] unrecognized feature gate: Example Nov 04 02:32:33.618637 ip-10-0-134-125 kubenswrapper[4405]: W1104 02:32:33.616721 4405 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 04 02:32:33.618637 ip-10-0-134-125 kubenswrapper[4405]: W1104 02:32:33.616724 4405 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 04 02:32:33.618637 ip-10-0-134-125 kubenswrapper[4405]: W1104 02:32:33.616726 4405 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 04 02:32:33.618637 ip-10-0-134-125 kubenswrapper[4405]: W1104 02:32:33.616729 4405 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 04 02:32:33.618637 ip-10-0-134-125 kubenswrapper[4405]: W1104 02:32:33.616732 4405 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 04 02:32:33.618637 ip-10-0-134-125 kubenswrapper[4405]: W1104 02:32:33.616734 4405 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 04 02:32:33.618637 ip-10-0-134-125 kubenswrapper[4405]: W1104 02:32:33.616737 4405 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 04 02:32:33.618637 ip-10-0-134-125 kubenswrapper[4405]: W1104 02:32:33.616740 4405 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 04 02:32:33.618637 ip-10-0-134-125 kubenswrapper[4405]: W1104 02:32:33.616742 4405 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 04 02:32:33.618637 ip-10-0-134-125 kubenswrapper[4405]: W1104 02:32:33.616745 4405 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 04 02:32:33.618637 ip-10-0-134-125 kubenswrapper[4405]: W1104 02:32:33.616748 4405 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 04 02:32:33.618637 ip-10-0-134-125 kubenswrapper[4405]: W1104 02:32:33.616750 4405 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 04 02:32:33.618637 ip-10-0-134-125 kubenswrapper[4405]: W1104 02:32:33.616753 4405 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 04 02:32:33.618637 ip-10-0-134-125 kubenswrapper[4405]: W1104 02:32:33.616756 4405 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 04 02:32:33.618637 ip-10-0-134-125 kubenswrapper[4405]: W1104 02:32:33.616758 4405 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 04 02:32:33.618637 ip-10-0-134-125 kubenswrapper[4405]: W1104 02:32:33.616761 4405 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 04 02:32:33.618637 ip-10-0-134-125 kubenswrapper[4405]: W1104 02:32:33.616764 4405 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 04 02:32:33.618637 ip-10-0-134-125 kubenswrapper[4405]: W1104 02:32:33.616766 4405 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 04 02:32:33.618637 ip-10-0-134-125 kubenswrapper[4405]: W1104 02:32:33.616769 4405 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 04 02:32:33.618637 ip-10-0-134-125 kubenswrapper[4405]: W1104 02:32:33.616771 4405 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 04 02:32:33.619166 ip-10-0-134-125 kubenswrapper[4405]: W1104 02:32:33.616774 4405 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 04 02:32:33.619166 ip-10-0-134-125 kubenswrapper[4405]: W1104 02:32:33.616777 4405 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 04 02:32:33.619166 ip-10-0-134-125 kubenswrapper[4405]: W1104 02:32:33.616780 4405 feature_gate.go:386] unrecognized feature gate: EVPN Nov 04 02:32:33.619166 ip-10-0-134-125 kubenswrapper[4405]: W1104 02:32:33.616782 4405 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 04 02:32:33.619166 ip-10-0-134-125 kubenswrapper[4405]: W1104 02:32:33.616785 4405 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 04 02:32:33.619166 ip-10-0-134-125 kubenswrapper[4405]: W1104 02:32:33.616788 4405 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 04 02:32:33.619166 ip-10-0-134-125 kubenswrapper[4405]: W1104 02:32:33.616790 4405 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 04 02:32:33.619166 ip-10-0-134-125 kubenswrapper[4405]: W1104 02:32:33.616793 4405 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 04 02:32:33.619166 ip-10-0-134-125 kubenswrapper[4405]: W1104 02:32:33.616796 4405 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 04 02:32:33.619166 ip-10-0-134-125 kubenswrapper[4405]: W1104 02:32:33.616799 4405 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 04 02:32:33.619166 ip-10-0-134-125 kubenswrapper[4405]: W1104 02:32:33.616802 4405 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 04 02:32:33.619166 ip-10-0-134-125 kubenswrapper[4405]: W1104 02:32:33.616804 4405 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 04 02:32:33.619166 ip-10-0-134-125 kubenswrapper[4405]: W1104 02:32:33.616807 4405 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 04 02:32:33.619166 ip-10-0-134-125 kubenswrapper[4405]: W1104 02:32:33.616810 4405 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 04 02:32:33.619166 ip-10-0-134-125 kubenswrapper[4405]: W1104 02:32:33.616812 4405 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 04 02:32:33.619166 ip-10-0-134-125 kubenswrapper[4405]: W1104 02:32:33.616815 4405 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 04 02:32:33.619166 ip-10-0-134-125 kubenswrapper[4405]: W1104 02:32:33.616817 4405 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 04 02:32:33.619166 ip-10-0-134-125 kubenswrapper[4405]: W1104 02:32:33.616820 4405 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 04 02:32:33.619166 ip-10-0-134-125 kubenswrapper[4405]: W1104 02:32:33.616822 4405 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 04 02:32:33.619681 ip-10-0-134-125 kubenswrapper[4405]: W1104 02:32:33.616825 4405 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 04 02:32:33.619681 ip-10-0-134-125 kubenswrapper[4405]: W1104 02:32:33.616828 4405 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 04 02:32:33.619681 ip-10-0-134-125 kubenswrapper[4405]: W1104 02:32:33.616830 4405 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 04 02:32:33.619681 ip-10-0-134-125 kubenswrapper[4405]: W1104 02:32:33.616833 4405 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 04 02:32:33.619681 ip-10-0-134-125 kubenswrapper[4405]: W1104 02:32:33.616835 4405 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 04 02:32:33.619681 ip-10-0-134-125 kubenswrapper[4405]: W1104 02:32:33.616838 4405 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 04 02:32:33.619681 ip-10-0-134-125 kubenswrapper[4405]: W1104 02:32:33.616841 4405 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 04 02:32:33.619681 ip-10-0-134-125 kubenswrapper[4405]: W1104 02:32:33.616843 4405 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 04 02:32:33.619681 ip-10-0-134-125 kubenswrapper[4405]: W1104 02:32:33.616846 4405 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 04 02:32:33.619681 ip-10-0-134-125 kubenswrapper[4405]: W1104 02:32:33.616849 4405 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 04 02:32:33.619681 ip-10-0-134-125 kubenswrapper[4405]: W1104 02:32:33.616851 4405 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 04 02:32:33.619681 ip-10-0-134-125 kubenswrapper[4405]: W1104 02:32:33.616856 4405 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 04 02:32:33.619681 ip-10-0-134-125 kubenswrapper[4405]: W1104 02:32:33.616858 4405 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 04 02:32:33.619681 ip-10-0-134-125 kubenswrapper[4405]: W1104 02:32:33.616861 4405 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 04 02:32:33.619681 ip-10-0-134-125 kubenswrapper[4405]: W1104 02:32:33.616863 4405 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 04 02:32:33.619681 ip-10-0-134-125 kubenswrapper[4405]: W1104 02:32:33.616866 4405 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 04 02:32:33.619681 ip-10-0-134-125 kubenswrapper[4405]: W1104 02:32:33.616869 4405 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 04 02:32:33.619681 ip-10-0-134-125 kubenswrapper[4405]: W1104 02:32:33.616871 4405 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 04 02:32:33.619681 ip-10-0-134-125 kubenswrapper[4405]: W1104 02:32:33.616873 4405 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 04 02:32:33.620190 ip-10-0-134-125 kubenswrapper[4405]: W1104 02:32:33.616877 4405 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 04 02:32:33.620190 ip-10-0-134-125 kubenswrapper[4405]: W1104 02:32:33.616881 4405 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 04 02:32:33.620190 ip-10-0-134-125 kubenswrapper[4405]: W1104 02:32:33.616884 4405 feature_gate.go:386] unrecognized feature gate: Example2 Nov 04 02:32:33.620190 ip-10-0-134-125 kubenswrapper[4405]: W1104 02:32:33.616886 4405 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 04 02:32:33.620190 ip-10-0-134-125 kubenswrapper[4405]: W1104 02:32:33.616891 4405 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 04 02:32:33.620190 ip-10-0-134-125 kubenswrapper[4405]: W1104 02:32:33.616895 4405 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 04 02:32:33.620190 ip-10-0-134-125 kubenswrapper[4405]: W1104 02:32:33.616898 4405 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 04 02:32:33.620190 ip-10-0-134-125 kubenswrapper[4405]: W1104 02:32:33.616901 4405 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 04 02:32:33.620190 ip-10-0-134-125 kubenswrapper[4405]: W1104 02:32:33.616904 4405 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 04 02:32:33.620190 ip-10-0-134-125 kubenswrapper[4405]: W1104 02:32:33.616907 4405 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 04 02:32:33.620190 ip-10-0-134-125 kubenswrapper[4405]: W1104 02:32:33.616909 4405 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 04 02:32:33.620190 ip-10-0-134-125 kubenswrapper[4405]: W1104 02:32:33.616912 4405 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 04 02:32:33.620190 ip-10-0-134-125 kubenswrapper[4405]: W1104 02:32:33.616915 4405 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 04 02:32:33.620190 ip-10-0-134-125 kubenswrapper[4405]: W1104 02:32:33.616917 4405 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 04 02:32:33.620190 ip-10-0-134-125 kubenswrapper[4405]: W1104 02:32:33.616920 4405 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 04 02:32:33.620190 ip-10-0-134-125 kubenswrapper[4405]: W1104 02:32:33.616923 4405 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 04 02:32:33.620190 ip-10-0-134-125 kubenswrapper[4405]: W1104 02:32:33.616925 4405 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 04 02:32:33.620190 ip-10-0-134-125 kubenswrapper[4405]: W1104 02:32:33.616927 4405 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 04 02:32:33.620190 ip-10-0-134-125 kubenswrapper[4405]: W1104 02:32:33.616930 4405 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 04 02:32:33.620727 ip-10-0-134-125 kubenswrapper[4405]: W1104 02:32:33.616936 4405 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 04 02:32:33.620727 ip-10-0-134-125 kubenswrapper[4405]: W1104 02:32:33.616938 4405 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 04 02:32:33.620727 ip-10-0-134-125 kubenswrapper[4405]: W1104 02:32:33.616941 4405 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 04 02:32:33.620727 ip-10-0-134-125 kubenswrapper[4405]: W1104 02:32:33.616944 4405 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 04 02:32:33.620727 ip-10-0-134-125 kubenswrapper[4405]: W1104 02:32:33.616946 4405 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 04 02:32:33.620727 ip-10-0-134-125 kubenswrapper[4405]: W1104 02:32:33.616949 4405 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 04 02:32:33.620727 ip-10-0-134-125 kubenswrapper[4405]: W1104 02:32:33.616953 4405 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 04 02:32:33.620727 ip-10-0-134-125 kubenswrapper[4405]: W1104 02:32:33.616956 4405 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 04 02:32:33.620727 ip-10-0-134-125 kubenswrapper[4405]: W1104 02:32:33.616958 4405 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 04 02:32:33.620727 ip-10-0-134-125 kubenswrapper[4405]: W1104 02:32:33.616961 4405 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 04 02:32:33.620727 ip-10-0-134-125 kubenswrapper[4405]: W1104 02:32:33.616964 4405 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 04 02:32:33.620727 ip-10-0-134-125 kubenswrapper[4405]: W1104 02:32:33.616967 4405 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 04 02:32:33.620727 ip-10-0-134-125 kubenswrapper[4405]: W1104 02:32:33.616969 4405 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 04 02:32:33.620727 ip-10-0-134-125 kubenswrapper[4405]: W1104 02:32:33.616972 4405 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 04 02:32:33.620727 ip-10-0-134-125 kubenswrapper[4405]: W1104 02:32:33.616974 4405 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 04 02:32:33.620727 ip-10-0-134-125 kubenswrapper[4405]: W1104 02:32:33.617478 4405 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 04 02:32:33.620727 ip-10-0-134-125 kubenswrapper[4405]: W1104 02:32:33.617484 4405 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 04 02:32:33.620727 ip-10-0-134-125 kubenswrapper[4405]: W1104 02:32:33.617487 4405 feature_gate.go:386] unrecognized feature gate: Example2 Nov 04 02:32:33.620727 ip-10-0-134-125 kubenswrapper[4405]: W1104 02:32:33.617491 4405 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 04 02:32:33.620727 ip-10-0-134-125 kubenswrapper[4405]: W1104 02:32:33.617494 4405 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 04 02:32:33.620727 ip-10-0-134-125 kubenswrapper[4405]: W1104 02:32:33.617497 4405 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 04 02:32:33.621328 ip-10-0-134-125 kubenswrapper[4405]: W1104 02:32:33.617500 4405 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 04 02:32:33.621328 ip-10-0-134-125 kubenswrapper[4405]: W1104 02:32:33.617503 4405 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 04 02:32:33.621328 ip-10-0-134-125 kubenswrapper[4405]: W1104 02:32:33.617506 4405 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 04 02:32:33.621328 ip-10-0-134-125 kubenswrapper[4405]: W1104 02:32:33.617508 4405 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 04 02:32:33.621328 ip-10-0-134-125 kubenswrapper[4405]: W1104 02:32:33.617511 4405 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 04 02:32:33.621328 ip-10-0-134-125 kubenswrapper[4405]: W1104 02:32:33.617514 4405 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 04 02:32:33.621328 ip-10-0-134-125 kubenswrapper[4405]: W1104 02:32:33.617517 4405 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 04 02:32:33.621328 ip-10-0-134-125 kubenswrapper[4405]: W1104 02:32:33.617519 4405 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 04 02:32:33.621328 ip-10-0-134-125 kubenswrapper[4405]: W1104 02:32:33.617522 4405 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 04 02:32:33.621328 ip-10-0-134-125 kubenswrapper[4405]: W1104 02:32:33.617524 4405 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 04 02:32:33.621328 ip-10-0-134-125 kubenswrapper[4405]: W1104 02:32:33.617527 4405 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 04 02:32:33.621328 ip-10-0-134-125 kubenswrapper[4405]: W1104 02:32:33.617532 4405 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 04 02:32:33.621328 ip-10-0-134-125 kubenswrapper[4405]: W1104 02:32:33.617534 4405 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 04 02:32:33.621328 ip-10-0-134-125 kubenswrapper[4405]: W1104 02:32:33.617537 4405 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 04 02:32:33.621328 ip-10-0-134-125 kubenswrapper[4405]: W1104 02:32:33.617539 4405 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 04 02:32:33.621328 ip-10-0-134-125 kubenswrapper[4405]: W1104 02:32:33.617542 4405 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 04 02:32:33.621328 ip-10-0-134-125 kubenswrapper[4405]: W1104 02:32:33.617560 4405 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 04 02:32:33.621328 ip-10-0-134-125 kubenswrapper[4405]: W1104 02:32:33.617565 4405 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 04 02:32:33.621328 ip-10-0-134-125 kubenswrapper[4405]: W1104 02:32:33.617569 4405 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 04 02:32:33.622129 ip-10-0-134-125 kubenswrapper[4405]: W1104 02:32:33.617572 4405 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 04 02:32:33.622129 ip-10-0-134-125 kubenswrapper[4405]: W1104 02:32:33.617575 4405 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 04 02:32:33.622129 ip-10-0-134-125 kubenswrapper[4405]: W1104 02:32:33.617578 4405 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 04 02:32:33.622129 ip-10-0-134-125 kubenswrapper[4405]: W1104 02:32:33.617581 4405 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 04 02:32:33.622129 ip-10-0-134-125 kubenswrapper[4405]: W1104 02:32:33.617584 4405 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 04 02:32:33.622129 ip-10-0-134-125 kubenswrapper[4405]: W1104 02:32:33.617587 4405 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 04 02:32:33.622129 ip-10-0-134-125 kubenswrapper[4405]: W1104 02:32:33.617589 4405 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 04 02:32:33.622129 ip-10-0-134-125 kubenswrapper[4405]: W1104 02:32:33.617592 4405 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 04 02:32:33.622129 ip-10-0-134-125 kubenswrapper[4405]: W1104 02:32:33.617595 4405 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 04 02:32:33.622129 ip-10-0-134-125 kubenswrapper[4405]: W1104 02:32:33.617598 4405 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 04 02:32:33.622129 ip-10-0-134-125 kubenswrapper[4405]: W1104 02:32:33.617600 4405 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 04 02:32:33.622129 ip-10-0-134-125 kubenswrapper[4405]: W1104 02:32:33.617604 4405 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 04 02:32:33.622129 ip-10-0-134-125 kubenswrapper[4405]: W1104 02:32:33.617606 4405 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 04 02:32:33.622129 ip-10-0-134-125 kubenswrapper[4405]: W1104 02:32:33.617609 4405 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 04 02:32:33.622129 ip-10-0-134-125 kubenswrapper[4405]: W1104 02:32:33.617611 4405 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 04 02:32:33.622129 ip-10-0-134-125 kubenswrapper[4405]: W1104 02:32:33.617614 4405 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 04 02:32:33.622129 ip-10-0-134-125 kubenswrapper[4405]: W1104 02:32:33.617616 4405 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 04 02:32:33.622129 ip-10-0-134-125 kubenswrapper[4405]: W1104 02:32:33.617619 4405 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 04 02:32:33.622129 ip-10-0-134-125 kubenswrapper[4405]: W1104 02:32:33.617621 4405 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 04 02:32:33.622995 ip-10-0-134-125 kubenswrapper[4405]: W1104 02:32:33.617624 4405 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 04 02:32:33.622995 ip-10-0-134-125 kubenswrapper[4405]: W1104 02:32:33.617627 4405 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 04 02:32:33.622995 ip-10-0-134-125 kubenswrapper[4405]: W1104 02:32:33.617629 4405 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 04 02:32:33.622995 ip-10-0-134-125 kubenswrapper[4405]: W1104 02:32:33.617632 4405 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 04 02:32:33.622995 ip-10-0-134-125 kubenswrapper[4405]: W1104 02:32:33.617634 4405 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 04 02:32:33.622995 ip-10-0-134-125 kubenswrapper[4405]: W1104 02:32:33.617637 4405 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 04 02:32:33.622995 ip-10-0-134-125 kubenswrapper[4405]: W1104 02:32:33.617640 4405 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 04 02:32:33.622995 ip-10-0-134-125 kubenswrapper[4405]: W1104 02:32:33.617642 4405 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 04 02:32:33.622995 ip-10-0-134-125 kubenswrapper[4405]: W1104 02:32:33.617645 4405 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 04 02:32:33.622995 ip-10-0-134-125 kubenswrapper[4405]: W1104 02:32:33.617647 4405 feature_gate.go:386] unrecognized feature gate: EVPN Nov 04 02:32:33.622995 ip-10-0-134-125 kubenswrapper[4405]: W1104 02:32:33.617650 4405 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 04 02:32:33.622995 ip-10-0-134-125 kubenswrapper[4405]: W1104 02:32:33.617652 4405 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 04 02:32:33.622995 ip-10-0-134-125 kubenswrapper[4405]: W1104 02:32:33.617655 4405 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 04 02:32:33.622995 ip-10-0-134-125 kubenswrapper[4405]: W1104 02:32:33.617658 4405 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 04 02:32:33.622995 ip-10-0-134-125 kubenswrapper[4405]: W1104 02:32:33.617660 4405 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 04 02:32:33.622995 ip-10-0-134-125 kubenswrapper[4405]: W1104 02:32:33.617676 4405 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 04 02:32:33.622995 ip-10-0-134-125 kubenswrapper[4405]: W1104 02:32:33.617680 4405 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 04 02:32:33.622995 ip-10-0-134-125 kubenswrapper[4405]: W1104 02:32:33.617682 4405 feature_gate.go:386] unrecognized feature gate: Example Nov 04 02:32:33.622995 ip-10-0-134-125 kubenswrapper[4405]: W1104 02:32:33.617685 4405 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 04 02:32:33.622995 ip-10-0-134-125 kubenswrapper[4405]: W1104 02:32:33.617689 4405 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 04 02:32:33.623931 ip-10-0-134-125 kubenswrapper[4405]: W1104 02:32:33.617692 4405 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 04 02:32:33.623931 ip-10-0-134-125 kubenswrapper[4405]: W1104 02:32:33.617695 4405 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 04 02:32:33.623931 ip-10-0-134-125 kubenswrapper[4405]: W1104 02:32:33.617697 4405 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 04 02:32:33.623931 ip-10-0-134-125 kubenswrapper[4405]: W1104 02:32:33.617700 4405 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 04 02:32:33.623931 ip-10-0-134-125 kubenswrapper[4405]: W1104 02:32:33.617702 4405 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 04 02:32:33.623931 ip-10-0-134-125 kubenswrapper[4405]: W1104 02:32:33.617705 4405 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 04 02:32:33.623931 ip-10-0-134-125 kubenswrapper[4405]: W1104 02:32:33.617707 4405 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 04 02:32:33.623931 ip-10-0-134-125 kubenswrapper[4405]: W1104 02:32:33.617710 4405 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 04 02:32:33.623931 ip-10-0-134-125 kubenswrapper[4405]: W1104 02:32:33.617713 4405 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 04 02:32:33.623931 ip-10-0-134-125 kubenswrapper[4405]: W1104 02:32:33.617715 4405 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 04 02:32:33.623931 ip-10-0-134-125 kubenswrapper[4405]: W1104 02:32:33.617718 4405 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 04 02:32:33.623931 ip-10-0-134-125 kubenswrapper[4405]: W1104 02:32:33.617720 4405 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 04 02:32:33.623931 ip-10-0-134-125 kubenswrapper[4405]: W1104 02:32:33.617723 4405 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 04 02:32:33.623931 ip-10-0-134-125 kubenswrapper[4405]: W1104 02:32:33.617726 4405 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 04 02:32:33.623931 ip-10-0-134-125 kubenswrapper[4405]: W1104 02:32:33.617729 4405 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 04 02:32:33.623931 ip-10-0-134-125 kubenswrapper[4405]: W1104 02:32:33.617731 4405 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 04 02:32:33.623931 ip-10-0-134-125 kubenswrapper[4405]: W1104 02:32:33.617734 4405 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 04 02:32:33.623931 ip-10-0-134-125 kubenswrapper[4405]: W1104 02:32:33.617737 4405 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 04 02:32:33.623931 ip-10-0-134-125 kubenswrapper[4405]: W1104 02:32:33.617739 4405 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 04 02:32:33.624814 ip-10-0-134-125 kubenswrapper[4405]: W1104 02:32:33.617742 4405 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 04 02:32:33.624814 ip-10-0-134-125 kubenswrapper[4405]: W1104 02:32:33.617744 4405 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 04 02:32:33.624814 ip-10-0-134-125 kubenswrapper[4405]: W1104 02:32:33.617749 4405 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 04 02:32:33.624814 ip-10-0-134-125 kubenswrapper[4405]: W1104 02:32:33.617752 4405 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 04 02:32:33.624814 ip-10-0-134-125 kubenswrapper[4405]: W1104 02:32:33.617755 4405 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 04 02:32:33.624814 ip-10-0-134-125 kubenswrapper[4405]: W1104 02:32:33.617758 4405 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 04 02:32:33.624814 ip-10-0-134-125 kubenswrapper[4405]: W1104 02:32:33.617760 4405 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 04 02:32:33.624814 ip-10-0-134-125 kubenswrapper[4405]: W1104 02:32:33.617763 4405 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 04 02:32:33.624814 ip-10-0-134-125 kubenswrapper[4405]: W1104 02:32:33.617766 4405 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 04 02:32:33.624814 ip-10-0-134-125 kubenswrapper[4405]: W1104 02:32:33.617769 4405 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 04 02:32:33.624814 ip-10-0-134-125 kubenswrapper[4405]: W1104 02:32:33.617772 4405 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 04 02:32:33.624814 ip-10-0-134-125 kubenswrapper[4405]: W1104 02:32:33.617774 4405 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 04 02:32:33.624814 ip-10-0-134-125 kubenswrapper[4405]: W1104 02:32:33.617778 4405 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 04 02:32:33.624814 ip-10-0-134-125 kubenswrapper[4405]: W1104 02:32:33.617781 4405 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 04 02:32:33.624814 ip-10-0-134-125 kubenswrapper[4405]: W1104 02:32:33.617783 4405 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 04 02:32:33.624814 ip-10-0-134-125 kubenswrapper[4405]: W1104 02:32:33.617786 4405 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 04 02:32:33.624814 ip-10-0-134-125 kubenswrapper[4405]: W1104 02:32:33.617789 4405 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 04 02:32:33.624814 ip-10-0-134-125 kubenswrapper[4405]: W1104 02:32:33.617791 4405 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 04 02:32:33.624814 ip-10-0-134-125 kubenswrapper[4405]: W1104 02:32:33.617794 4405 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 04 02:32:33.625709 ip-10-0-134-125 kubenswrapper[4405]: W1104 02:32:33.617796 4405 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 04 02:32:33.625709 ip-10-0-134-125 kubenswrapper[4405]: W1104 02:32:33.617799 4405 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 04 02:32:33.625709 ip-10-0-134-125 kubenswrapper[4405]: W1104 02:32:33.617802 4405 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 04 02:32:33.625709 ip-10-0-134-125 kubenswrapper[4405]: W1104 02:32:33.617804 4405 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 04 02:32:33.625709 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:32:33.617894 4405 flags.go:64] FLAG: --address="0.0.0.0" Nov 04 02:32:33.625709 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:32:33.617901 4405 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Nov 04 02:32:33.625709 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:32:33.617907 4405 flags.go:64] FLAG: --anonymous-auth="true" Nov 04 02:32:33.625709 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:32:33.617911 4405 flags.go:64] FLAG: --application-metrics-count-limit="100" Nov 04 02:32:33.625709 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:32:33.617916 4405 flags.go:64] FLAG: --authentication-token-webhook="false" Nov 04 02:32:33.625709 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:32:33.617919 4405 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Nov 04 02:32:33.625709 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:32:33.617924 4405 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Nov 04 02:32:33.625709 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:32:33.617928 4405 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Nov 04 02:32:33.625709 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:32:33.617931 4405 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Nov 04 02:32:33.625709 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:32:33.617934 4405 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Nov 04 02:32:33.625709 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:32:33.617938 4405 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Nov 04 02:32:33.625709 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:32:33.617941 4405 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Nov 04 02:32:33.625709 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:32:33.617944 4405 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Nov 04 02:32:33.625709 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:32:33.617947 4405 flags.go:64] FLAG: --cgroup-root="" Nov 04 02:32:33.625709 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:32:33.617950 4405 flags.go:64] FLAG: --cgroups-per-qos="true" Nov 04 02:32:33.625709 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:32:33.617953 4405 flags.go:64] FLAG: --client-ca-file="" Nov 04 02:32:33.625709 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:32:33.617956 4405 flags.go:64] FLAG: --cloud-provider="external" Nov 04 02:32:33.625709 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:32:33.617959 4405 flags.go:64] FLAG: --cluster-dns="[]" Nov 04 02:32:33.625709 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:32:33.617964 4405 flags.go:64] FLAG: --cluster-domain="" Nov 04 02:32:33.625709 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:32:33.617967 4405 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Nov 04 02:32:33.626849 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:32:33.617970 4405 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Nov 04 02:32:33.626849 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:32:33.617975 4405 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Nov 04 02:32:33.626849 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:32:33.617978 4405 flags.go:64] FLAG: --container-log-max-files="5" Nov 04 02:32:33.626849 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:32:33.617982 4405 flags.go:64] FLAG: --container-log-max-size="10Mi" Nov 04 02:32:33.626849 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:32:33.617985 4405 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Nov 04 02:32:33.626849 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:32:33.617988 4405 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Nov 04 02:32:33.626849 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:32:33.617991 4405 flags.go:64] FLAG: --containerd-namespace="k8s.io" Nov 04 02:32:33.626849 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:32:33.617994 4405 flags.go:64] FLAG: --contention-profiling="false" Nov 04 02:32:33.626849 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:32:33.617999 4405 flags.go:64] FLAG: --cpu-cfs-quota="true" Nov 04 02:32:33.626849 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:32:33.618002 4405 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Nov 04 02:32:33.626849 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:32:33.618005 4405 flags.go:64] FLAG: --cpu-manager-policy="none" Nov 04 02:32:33.626849 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:32:33.618008 4405 flags.go:64] FLAG: --cpu-manager-policy-options="" Nov 04 02:32:33.626849 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:32:33.618012 4405 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Nov 04 02:32:33.626849 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:32:33.618016 4405 flags.go:64] FLAG: --enable-controller-attach-detach="true" Nov 04 02:32:33.626849 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:32:33.618018 4405 flags.go:64] FLAG: --enable-debugging-handlers="true" Nov 04 02:32:33.626849 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:32:33.618021 4405 flags.go:64] FLAG: --enable-load-reader="false" Nov 04 02:32:33.626849 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:32:33.618024 4405 flags.go:64] FLAG: --enable-server="true" Nov 04 02:32:33.626849 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:32:33.618027 4405 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Nov 04 02:32:33.626849 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:32:33.618031 4405 flags.go:64] FLAG: --event-burst="100" Nov 04 02:32:33.626849 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:32:33.618035 4405 flags.go:64] FLAG: --event-qps="50" Nov 04 02:32:33.626849 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:32:33.618037 4405 flags.go:64] FLAG: --event-storage-age-limit="default=0" Nov 04 02:32:33.626849 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:32:33.618040 4405 flags.go:64] FLAG: --event-storage-event-limit="default=0" Nov 04 02:32:33.626849 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:32:33.618043 4405 flags.go:64] FLAG: --eviction-hard="" Nov 04 02:32:33.626849 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:32:33.618048 4405 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Nov 04 02:32:33.627958 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:32:33.618051 4405 flags.go:64] FLAG: --eviction-minimum-reclaim="" Nov 04 02:32:33.627958 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:32:33.618054 4405 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Nov 04 02:32:33.627958 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:32:33.618057 4405 flags.go:64] FLAG: --eviction-soft="" Nov 04 02:32:33.627958 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:32:33.618060 4405 flags.go:64] FLAG: --eviction-soft-grace-period="" Nov 04 02:32:33.627958 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:32:33.618062 4405 flags.go:64] FLAG: --exit-on-lock-contention="false" Nov 04 02:32:33.627958 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:32:33.618065 4405 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Nov 04 02:32:33.627958 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:32:33.618068 4405 flags.go:64] FLAG: --experimental-mounter-path="" Nov 04 02:32:33.627958 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:32:33.618071 4405 flags.go:64] FLAG: --fail-cgroupv1="true" Nov 04 02:32:33.627958 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:32:33.618074 4405 flags.go:64] FLAG: --fail-swap-on="true" Nov 04 02:32:33.627958 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:32:33.618079 4405 flags.go:64] FLAG: --feature-gates="" Nov 04 02:32:33.627958 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:32:33.618083 4405 flags.go:64] FLAG: --file-check-frequency="20s" Nov 04 02:32:33.627958 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:32:33.618086 4405 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Nov 04 02:32:33.627958 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:32:33.618089 4405 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Nov 04 02:32:33.627958 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:32:33.618092 4405 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Nov 04 02:32:33.627958 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:32:33.618095 4405 flags.go:64] FLAG: --healthz-port="10248" Nov 04 02:32:33.627958 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:32:33.618099 4405 flags.go:64] FLAG: --help="false" Nov 04 02:32:33.627958 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:32:33.618103 4405 flags.go:64] FLAG: --hostname-override="ip-10-0-134-125.us-west-1.compute.internal" Nov 04 02:32:33.627958 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:32:33.618106 4405 flags.go:64] FLAG: --housekeeping-interval="10s" Nov 04 02:32:33.627958 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:32:33.618109 4405 flags.go:64] FLAG: --http-check-frequency="20s" Nov 04 02:32:33.627958 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:32:33.618112 4405 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Nov 04 02:32:33.627958 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:32:33.618115 4405 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Nov 04 02:32:33.627958 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:32:33.618119 4405 flags.go:64] FLAG: --image-gc-high-threshold="85" Nov 04 02:32:33.627958 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:32:33.618122 4405 flags.go:64] FLAG: --image-gc-low-threshold="80" Nov 04 02:32:33.629035 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:32:33.618124 4405 flags.go:64] FLAG: --image-service-endpoint="" Nov 04 02:32:33.629035 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:32:33.618127 4405 flags.go:64] FLAG: --kernel-memcg-notification="false" Nov 04 02:32:33.629035 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:32:33.618130 4405 flags.go:64] FLAG: --kube-api-burst="100" Nov 04 02:32:33.629035 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:32:33.618133 4405 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Nov 04 02:32:33.629035 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:32:33.618136 4405 flags.go:64] FLAG: --kube-api-qps="50" Nov 04 02:32:33.629035 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:32:33.618139 4405 flags.go:64] FLAG: --kube-reserved="" Nov 04 02:32:33.629035 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:32:33.618142 4405 flags.go:64] FLAG: --kube-reserved-cgroup="" Nov 04 02:32:33.629035 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:32:33.618145 4405 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Nov 04 02:32:33.629035 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:32:33.618148 4405 flags.go:64] FLAG: --kubelet-cgroups="" Nov 04 02:32:33.629035 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:32:33.618151 4405 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Nov 04 02:32:33.629035 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:32:33.618154 4405 flags.go:64] FLAG: --lock-file="" Nov 04 02:32:33.629035 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:32:33.618157 4405 flags.go:64] FLAG: --log-cadvisor-usage="false" Nov 04 02:32:33.629035 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:32:33.618160 4405 flags.go:64] FLAG: --log-flush-frequency="5s" Nov 04 02:32:33.629035 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:32:33.618162 4405 flags.go:64] FLAG: --log-json-info-buffer-size="0" Nov 04 02:32:33.629035 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:32:33.618168 4405 flags.go:64] FLAG: --log-json-split-stream="false" Nov 04 02:32:33.629035 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:32:33.618171 4405 flags.go:64] FLAG: --log-text-info-buffer-size="0" Nov 04 02:32:33.629035 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:32:33.618174 4405 flags.go:64] FLAG: --log-text-split-stream="false" Nov 04 02:32:33.629035 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:32:33.618177 4405 flags.go:64] FLAG: --logging-format="text" Nov 04 02:32:33.629035 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:32:33.618194 4405 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Nov 04 02:32:33.629035 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:32:33.618200 4405 flags.go:64] FLAG: --make-iptables-util-chains="true" Nov 04 02:32:33.629035 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:32:33.618205 4405 flags.go:64] FLAG: --manifest-url="" Nov 04 02:32:33.629035 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:32:33.618210 4405 flags.go:64] FLAG: --manifest-url-header="" Nov 04 02:32:33.629035 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:32:33.618217 4405 flags.go:64] FLAG: --max-housekeeping-interval="15s" Nov 04 02:32:33.629035 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:32:33.618221 4405 flags.go:64] FLAG: --max-open-files="1000000" Nov 04 02:32:33.629035 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:32:33.618225 4405 flags.go:64] FLAG: --max-pods="110" Nov 04 02:32:33.630204 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:32:33.618229 4405 flags.go:64] FLAG: --maximum-dead-containers="-1" Nov 04 02:32:33.630204 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:32:33.618233 4405 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Nov 04 02:32:33.630204 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:32:33.618236 4405 flags.go:64] FLAG: --memory-manager-policy="None" Nov 04 02:32:33.630204 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:32:33.618239 4405 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Nov 04 02:32:33.630204 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:32:33.618242 4405 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Nov 04 02:32:33.630204 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:32:33.618244 4405 flags.go:64] FLAG: --node-ip="0.0.0.0" Nov 04 02:32:33.630204 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:32:33.618247 4405 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Nov 04 02:32:33.630204 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:32:33.618254 4405 flags.go:64] FLAG: --node-status-max-images="50" Nov 04 02:32:33.630204 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:32:33.618257 4405 flags.go:64] FLAG: --node-status-update-frequency="10s" Nov 04 02:32:33.630204 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:32:33.618260 4405 flags.go:64] FLAG: --oom-score-adj="-999" Nov 04 02:32:33.630204 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:32:33.618263 4405 flags.go:64] FLAG: --pod-cidr="" Nov 04 02:32:33.630204 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:32:33.618266 4405 flags.go:64] FLAG: --pod-manifest-path="" Nov 04 02:32:33.630204 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:32:33.618269 4405 flags.go:64] FLAG: --pod-max-pids="-1" Nov 04 02:32:33.630204 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:32:33.618272 4405 flags.go:64] FLAG: --pods-per-core="0" Nov 04 02:32:33.630204 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:32:33.618275 4405 flags.go:64] FLAG: --port="10250" Nov 04 02:32:33.630204 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:32:33.618278 4405 flags.go:64] FLAG: --protect-kernel-defaults="false" Nov 04 02:32:33.630204 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:32:33.618281 4405 flags.go:64] FLAG: --provider-id="aws:///us-west-1a/i-0e1ce850fea14055b" Nov 04 02:32:33.630204 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:32:33.618284 4405 flags.go:64] FLAG: --qos-reserved="" Nov 04 02:32:33.630204 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:32:33.618287 4405 flags.go:64] FLAG: --read-only-port="10255" Nov 04 02:32:33.630204 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:32:33.618290 4405 flags.go:64] FLAG: --register-node="true" Nov 04 02:32:33.630204 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:32:33.618293 4405 flags.go:64] FLAG: --register-with-taints="" Nov 04 02:32:33.630204 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:32:33.618297 4405 flags.go:64] FLAG: --registry-burst="10" Nov 04 02:32:33.630204 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:32:33.618299 4405 flags.go:64] FLAG: --registry-qps="5" Nov 04 02:32:33.630204 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:32:33.618302 4405 flags.go:64] FLAG: --reserved-cpus="" Nov 04 02:32:33.630204 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:32:33.618305 4405 flags.go:64] FLAG: --reserved-memory="" Nov 04 02:32:33.630204 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:32:33.618310 4405 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Nov 04 02:32:33.631035 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:32:33.618313 4405 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Nov 04 02:32:33.631035 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:32:33.618317 4405 flags.go:64] FLAG: --rotate-certificates="false" Nov 04 02:32:33.631035 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:32:33.618320 4405 flags.go:64] FLAG: --rotate-server-certificates="false" Nov 04 02:32:33.631035 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:32:33.618323 4405 flags.go:64] FLAG: --runonce="false" Nov 04 02:32:33.631035 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:32:33.618325 4405 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Nov 04 02:32:33.631035 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:32:33.618328 4405 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Nov 04 02:32:33.631035 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:32:33.618332 4405 flags.go:64] FLAG: --seccomp-default="false" Nov 04 02:32:33.631035 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:32:33.618335 4405 flags.go:64] FLAG: --serialize-image-pulls="true" Nov 04 02:32:33.631035 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:32:33.618338 4405 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Nov 04 02:32:33.631035 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:32:33.618341 4405 flags.go:64] FLAG: --storage-driver-db="cadvisor" Nov 04 02:32:33.631035 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:32:33.618344 4405 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Nov 04 02:32:33.631035 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:32:33.618347 4405 flags.go:64] FLAG: --storage-driver-password="root" Nov 04 02:32:33.631035 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:32:33.618350 4405 flags.go:64] FLAG: --storage-driver-secure="false" Nov 04 02:32:33.631035 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:32:33.618353 4405 flags.go:64] FLAG: --storage-driver-table="stats" Nov 04 02:32:33.631035 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:32:33.618356 4405 flags.go:64] FLAG: --storage-driver-user="root" Nov 04 02:32:33.631035 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:32:33.618359 4405 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Nov 04 02:32:33.631035 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:32:33.618362 4405 flags.go:64] FLAG: --sync-frequency="1m0s" Nov 04 02:32:33.631035 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:32:33.618365 4405 flags.go:64] FLAG: --system-cgroups="" Nov 04 02:32:33.631035 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:32:33.618368 4405 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Nov 04 02:32:33.631035 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:32:33.618373 4405 flags.go:64] FLAG: --system-reserved-cgroup="" Nov 04 02:32:33.631035 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:32:33.618376 4405 flags.go:64] FLAG: --tls-cert-file="" Nov 04 02:32:33.631035 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:32:33.618379 4405 flags.go:64] FLAG: --tls-cipher-suites="[]" Nov 04 02:32:33.631035 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:32:33.618384 4405 flags.go:64] FLAG: --tls-min-version="" Nov 04 02:32:33.631035 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:32:33.618387 4405 flags.go:64] FLAG: --tls-private-key-file="" Nov 04 02:32:33.631035 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:32:33.618390 4405 flags.go:64] FLAG: --topology-manager-policy="none" Nov 04 02:32:33.631846 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:32:33.618392 4405 flags.go:64] FLAG: --topology-manager-policy-options="" Nov 04 02:32:33.631846 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:32:33.618395 4405 flags.go:64] FLAG: --topology-manager-scope="container" Nov 04 02:32:33.631846 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:32:33.618398 4405 flags.go:64] FLAG: --v="2" Nov 04 02:32:33.631846 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:32:33.618402 4405 flags.go:64] FLAG: --version="false" Nov 04 02:32:33.631846 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:32:33.618406 4405 flags.go:64] FLAG: --vmodule="" Nov 04 02:32:33.631846 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:32:33.618410 4405 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Nov 04 02:32:33.631846 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:32:33.618415 4405 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Nov 04 02:32:33.631846 ip-10-0-134-125 kubenswrapper[4405]: W1104 02:32:33.618568 4405 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 04 02:32:33.631846 ip-10-0-134-125 kubenswrapper[4405]: W1104 02:32:33.618572 4405 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 04 02:32:33.631846 ip-10-0-134-125 kubenswrapper[4405]: W1104 02:32:33.618576 4405 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 04 02:32:33.631846 ip-10-0-134-125 kubenswrapper[4405]: W1104 02:32:33.618579 4405 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 04 02:32:33.631846 ip-10-0-134-125 kubenswrapper[4405]: W1104 02:32:33.618581 4405 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 04 02:32:33.631846 ip-10-0-134-125 kubenswrapper[4405]: W1104 02:32:33.618584 4405 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 04 02:32:33.631846 ip-10-0-134-125 kubenswrapper[4405]: W1104 02:32:33.618588 4405 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 04 02:32:33.631846 ip-10-0-134-125 kubenswrapper[4405]: W1104 02:32:33.618591 4405 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 04 02:32:33.631846 ip-10-0-134-125 kubenswrapper[4405]: W1104 02:32:33.618593 4405 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 04 02:32:33.631846 ip-10-0-134-125 kubenswrapper[4405]: W1104 02:32:33.618596 4405 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 04 02:32:33.631846 ip-10-0-134-125 kubenswrapper[4405]: W1104 02:32:33.618599 4405 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 04 02:32:33.631846 ip-10-0-134-125 kubenswrapper[4405]: W1104 02:32:33.618601 4405 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 04 02:32:33.631846 ip-10-0-134-125 kubenswrapper[4405]: W1104 02:32:33.618604 4405 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 04 02:32:33.631846 ip-10-0-134-125 kubenswrapper[4405]: W1104 02:32:33.618607 4405 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 04 02:32:33.632405 ip-10-0-134-125 kubenswrapper[4405]: W1104 02:32:33.618609 4405 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 04 02:32:33.632405 ip-10-0-134-125 kubenswrapper[4405]: W1104 02:32:33.618612 4405 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 04 02:32:33.632405 ip-10-0-134-125 kubenswrapper[4405]: W1104 02:32:33.618614 4405 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 04 02:32:33.632405 ip-10-0-134-125 kubenswrapper[4405]: W1104 02:32:33.618617 4405 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 04 02:32:33.632405 ip-10-0-134-125 kubenswrapper[4405]: W1104 02:32:33.618619 4405 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 04 02:32:33.632405 ip-10-0-134-125 kubenswrapper[4405]: W1104 02:32:33.618622 4405 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 04 02:32:33.632405 ip-10-0-134-125 kubenswrapper[4405]: W1104 02:32:33.618624 4405 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 04 02:32:33.632405 ip-10-0-134-125 kubenswrapper[4405]: W1104 02:32:33.618627 4405 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 04 02:32:33.632405 ip-10-0-134-125 kubenswrapper[4405]: W1104 02:32:33.618630 4405 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 04 02:32:33.632405 ip-10-0-134-125 kubenswrapper[4405]: W1104 02:32:33.618632 4405 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 04 02:32:33.632405 ip-10-0-134-125 kubenswrapper[4405]: W1104 02:32:33.618635 4405 feature_gate.go:386] unrecognized feature gate: Example2 Nov 04 02:32:33.632405 ip-10-0-134-125 kubenswrapper[4405]: W1104 02:32:33.618638 4405 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 04 02:32:33.632405 ip-10-0-134-125 kubenswrapper[4405]: W1104 02:32:33.618640 4405 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 04 02:32:33.632405 ip-10-0-134-125 kubenswrapper[4405]: W1104 02:32:33.618643 4405 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 04 02:32:33.632405 ip-10-0-134-125 kubenswrapper[4405]: W1104 02:32:33.618646 4405 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 04 02:32:33.632405 ip-10-0-134-125 kubenswrapper[4405]: W1104 02:32:33.618649 4405 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 04 02:32:33.632405 ip-10-0-134-125 kubenswrapper[4405]: W1104 02:32:33.618651 4405 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 04 02:32:33.632405 ip-10-0-134-125 kubenswrapper[4405]: W1104 02:32:33.618654 4405 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 04 02:32:33.632405 ip-10-0-134-125 kubenswrapper[4405]: W1104 02:32:33.618657 4405 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 04 02:32:33.633047 ip-10-0-134-125 kubenswrapper[4405]: W1104 02:32:33.618660 4405 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 04 02:32:33.633047 ip-10-0-134-125 kubenswrapper[4405]: W1104 02:32:33.618662 4405 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 04 02:32:33.633047 ip-10-0-134-125 kubenswrapper[4405]: W1104 02:32:33.618678 4405 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 04 02:32:33.633047 ip-10-0-134-125 kubenswrapper[4405]: W1104 02:32:33.618681 4405 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 04 02:32:33.633047 ip-10-0-134-125 kubenswrapper[4405]: W1104 02:32:33.618684 4405 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 04 02:32:33.633047 ip-10-0-134-125 kubenswrapper[4405]: W1104 02:32:33.618688 4405 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 04 02:32:33.633047 ip-10-0-134-125 kubenswrapper[4405]: W1104 02:32:33.618691 4405 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 04 02:32:33.633047 ip-10-0-134-125 kubenswrapper[4405]: W1104 02:32:33.618693 4405 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 04 02:32:33.633047 ip-10-0-134-125 kubenswrapper[4405]: W1104 02:32:33.618696 4405 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 04 02:32:33.633047 ip-10-0-134-125 kubenswrapper[4405]: W1104 02:32:33.618698 4405 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 04 02:32:33.633047 ip-10-0-134-125 kubenswrapper[4405]: W1104 02:32:33.618701 4405 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 04 02:32:33.633047 ip-10-0-134-125 kubenswrapper[4405]: W1104 02:32:33.618704 4405 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 04 02:32:33.633047 ip-10-0-134-125 kubenswrapper[4405]: W1104 02:32:33.618706 4405 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 04 02:32:33.633047 ip-10-0-134-125 kubenswrapper[4405]: W1104 02:32:33.618709 4405 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 04 02:32:33.633047 ip-10-0-134-125 kubenswrapper[4405]: W1104 02:32:33.618711 4405 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 04 02:32:33.633047 ip-10-0-134-125 kubenswrapper[4405]: W1104 02:32:33.618714 4405 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 04 02:32:33.633047 ip-10-0-134-125 kubenswrapper[4405]: W1104 02:32:33.618716 4405 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 04 02:32:33.633047 ip-10-0-134-125 kubenswrapper[4405]: W1104 02:32:33.618719 4405 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 04 02:32:33.633047 ip-10-0-134-125 kubenswrapper[4405]: W1104 02:32:33.618721 4405 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 04 02:32:33.633944 ip-10-0-134-125 kubenswrapper[4405]: W1104 02:32:33.618724 4405 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 04 02:32:33.633944 ip-10-0-134-125 kubenswrapper[4405]: W1104 02:32:33.618726 4405 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 04 02:32:33.633944 ip-10-0-134-125 kubenswrapper[4405]: W1104 02:32:33.618728 4405 feature_gate.go:386] unrecognized feature gate: EVPN Nov 04 02:32:33.633944 ip-10-0-134-125 kubenswrapper[4405]: W1104 02:32:33.618731 4405 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 04 02:32:33.633944 ip-10-0-134-125 kubenswrapper[4405]: W1104 02:32:33.618734 4405 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 04 02:32:33.633944 ip-10-0-134-125 kubenswrapper[4405]: W1104 02:32:33.618736 4405 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 04 02:32:33.633944 ip-10-0-134-125 kubenswrapper[4405]: W1104 02:32:33.618739 4405 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 04 02:32:33.633944 ip-10-0-134-125 kubenswrapper[4405]: W1104 02:32:33.618742 4405 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 04 02:32:33.633944 ip-10-0-134-125 kubenswrapper[4405]: W1104 02:32:33.618744 4405 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 04 02:32:33.633944 ip-10-0-134-125 kubenswrapper[4405]: W1104 02:32:33.618747 4405 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 04 02:32:33.633944 ip-10-0-134-125 kubenswrapper[4405]: W1104 02:32:33.618749 4405 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 04 02:32:33.633944 ip-10-0-134-125 kubenswrapper[4405]: W1104 02:32:33.618752 4405 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 04 02:32:33.633944 ip-10-0-134-125 kubenswrapper[4405]: W1104 02:32:33.618754 4405 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 04 02:32:33.633944 ip-10-0-134-125 kubenswrapper[4405]: W1104 02:32:33.618757 4405 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 04 02:32:33.633944 ip-10-0-134-125 kubenswrapper[4405]: W1104 02:32:33.618759 4405 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 04 02:32:33.633944 ip-10-0-134-125 kubenswrapper[4405]: W1104 02:32:33.618763 4405 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 04 02:32:33.633944 ip-10-0-134-125 kubenswrapper[4405]: W1104 02:32:33.618767 4405 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 04 02:32:33.633944 ip-10-0-134-125 kubenswrapper[4405]: W1104 02:32:33.618770 4405 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 04 02:32:33.633944 ip-10-0-134-125 kubenswrapper[4405]: W1104 02:32:33.618775 4405 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 04 02:32:33.633944 ip-10-0-134-125 kubenswrapper[4405]: W1104 02:32:33.618778 4405 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 04 02:32:33.634876 ip-10-0-134-125 kubenswrapper[4405]: W1104 02:32:33.618780 4405 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 04 02:32:33.634876 ip-10-0-134-125 kubenswrapper[4405]: W1104 02:32:33.618783 4405 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 04 02:32:33.634876 ip-10-0-134-125 kubenswrapper[4405]: W1104 02:32:33.618786 4405 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 04 02:32:33.634876 ip-10-0-134-125 kubenswrapper[4405]: W1104 02:32:33.618789 4405 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 04 02:32:33.634876 ip-10-0-134-125 kubenswrapper[4405]: W1104 02:32:33.618791 4405 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 04 02:32:33.634876 ip-10-0-134-125 kubenswrapper[4405]: W1104 02:32:33.618794 4405 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 04 02:32:33.634876 ip-10-0-134-125 kubenswrapper[4405]: W1104 02:32:33.618796 4405 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 04 02:32:33.634876 ip-10-0-134-125 kubenswrapper[4405]: W1104 02:32:33.618799 4405 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 04 02:32:33.634876 ip-10-0-134-125 kubenswrapper[4405]: W1104 02:32:33.618802 4405 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 04 02:32:33.634876 ip-10-0-134-125 kubenswrapper[4405]: W1104 02:32:33.618804 4405 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 04 02:32:33.634876 ip-10-0-134-125 kubenswrapper[4405]: W1104 02:32:33.618807 4405 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 04 02:32:33.634876 ip-10-0-134-125 kubenswrapper[4405]: W1104 02:32:33.618809 4405 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 04 02:32:33.634876 ip-10-0-134-125 kubenswrapper[4405]: W1104 02:32:33.618812 4405 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 04 02:32:33.634876 ip-10-0-134-125 kubenswrapper[4405]: W1104 02:32:33.618816 4405 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 04 02:32:33.634876 ip-10-0-134-125 kubenswrapper[4405]: W1104 02:32:33.618819 4405 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 04 02:32:33.634876 ip-10-0-134-125 kubenswrapper[4405]: W1104 02:32:33.618822 4405 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 04 02:32:33.634876 ip-10-0-134-125 kubenswrapper[4405]: W1104 02:32:33.618826 4405 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 04 02:32:33.634876 ip-10-0-134-125 kubenswrapper[4405]: W1104 02:32:33.618829 4405 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 04 02:32:33.634876 ip-10-0-134-125 kubenswrapper[4405]: W1104 02:32:33.618832 4405 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 04 02:32:33.635462 ip-10-0-134-125 kubenswrapper[4405]: W1104 02:32:33.618835 4405 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 04 02:32:33.635462 ip-10-0-134-125 kubenswrapper[4405]: W1104 02:32:33.618837 4405 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 04 02:32:33.635462 ip-10-0-134-125 kubenswrapper[4405]: W1104 02:32:33.618840 4405 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 04 02:32:33.635462 ip-10-0-134-125 kubenswrapper[4405]: W1104 02:32:33.618843 4405 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 04 02:32:33.635462 ip-10-0-134-125 kubenswrapper[4405]: W1104 02:32:33.618845 4405 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 04 02:32:33.635462 ip-10-0-134-125 kubenswrapper[4405]: W1104 02:32:33.618848 4405 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 04 02:32:33.635462 ip-10-0-134-125 kubenswrapper[4405]: W1104 02:32:33.618850 4405 feature_gate.go:386] unrecognized feature gate: Example Nov 04 02:32:33.635462 ip-10-0-134-125 kubenswrapper[4405]: W1104 02:32:33.618853 4405 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 04 02:32:33.635462 ip-10-0-134-125 kubenswrapper[4405]: W1104 02:32:33.618855 4405 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 04 02:32:33.635462 ip-10-0-134-125 kubenswrapper[4405]: W1104 02:32:33.618858 4405 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 04 02:32:33.635462 ip-10-0-134-125 kubenswrapper[4405]: W1104 02:32:33.618861 4405 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 04 02:32:33.635462 ip-10-0-134-125 kubenswrapper[4405]: W1104 02:32:33.618865 4405 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 04 02:32:33.635462 ip-10-0-134-125 kubenswrapper[4405]: W1104 02:32:33.618868 4405 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 04 02:32:33.635462 ip-10-0-134-125 kubenswrapper[4405]: W1104 02:32:33.618871 4405 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 04 02:32:33.635462 ip-10-0-134-125 kubenswrapper[4405]: W1104 02:32:33.618873 4405 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 04 02:32:33.635462 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:32:33.618903 4405 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Nov 04 02:32:33.635995 ip-10-0-134-125 kubenswrapper[4405]: W1104 02:32:33.618938 4405 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 04 02:32:33.635995 ip-10-0-134-125 kubenswrapper[4405]: W1104 02:32:33.618943 4405 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 04 02:32:33.635995 ip-10-0-134-125 kubenswrapper[4405]: W1104 02:32:33.618946 4405 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 04 02:32:33.635995 ip-10-0-134-125 kubenswrapper[4405]: W1104 02:32:33.618949 4405 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 04 02:32:33.635995 ip-10-0-134-125 kubenswrapper[4405]: W1104 02:32:33.618952 4405 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 04 02:32:33.635995 ip-10-0-134-125 kubenswrapper[4405]: W1104 02:32:33.618956 4405 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 04 02:32:33.635995 ip-10-0-134-125 kubenswrapper[4405]: W1104 02:32:33.618959 4405 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 04 02:32:33.635995 ip-10-0-134-125 kubenswrapper[4405]: W1104 02:32:33.618962 4405 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 04 02:32:33.635995 ip-10-0-134-125 kubenswrapper[4405]: W1104 02:32:33.618965 4405 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 04 02:32:33.635995 ip-10-0-134-125 kubenswrapper[4405]: W1104 02:32:33.618967 4405 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 04 02:32:33.635995 ip-10-0-134-125 kubenswrapper[4405]: W1104 02:32:33.618970 4405 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 04 02:32:33.635995 ip-10-0-134-125 kubenswrapper[4405]: W1104 02:32:33.618973 4405 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 04 02:32:33.635995 ip-10-0-134-125 kubenswrapper[4405]: W1104 02:32:33.618976 4405 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 04 02:32:33.635995 ip-10-0-134-125 kubenswrapper[4405]: W1104 02:32:33.618978 4405 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 04 02:32:33.635995 ip-10-0-134-125 kubenswrapper[4405]: W1104 02:32:33.618981 4405 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 04 02:32:33.635995 ip-10-0-134-125 kubenswrapper[4405]: W1104 02:32:33.618983 4405 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 04 02:32:33.635995 ip-10-0-134-125 kubenswrapper[4405]: W1104 02:32:33.618986 4405 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 04 02:32:33.635995 ip-10-0-134-125 kubenswrapper[4405]: W1104 02:32:33.618988 4405 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 04 02:32:33.635995 ip-10-0-134-125 kubenswrapper[4405]: W1104 02:32:33.618991 4405 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 04 02:32:33.636650 ip-10-0-134-125 kubenswrapper[4405]: W1104 02:32:33.618994 4405 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 04 02:32:33.636650 ip-10-0-134-125 kubenswrapper[4405]: W1104 02:32:33.618996 4405 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 04 02:32:33.636650 ip-10-0-134-125 kubenswrapper[4405]: W1104 02:32:33.618998 4405 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 04 02:32:33.636650 ip-10-0-134-125 kubenswrapper[4405]: W1104 02:32:33.619002 4405 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 04 02:32:33.636650 ip-10-0-134-125 kubenswrapper[4405]: W1104 02:32:33.619006 4405 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 04 02:32:33.636650 ip-10-0-134-125 kubenswrapper[4405]: W1104 02:32:33.619009 4405 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 04 02:32:33.636650 ip-10-0-134-125 kubenswrapper[4405]: W1104 02:32:33.619012 4405 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 04 02:32:33.636650 ip-10-0-134-125 kubenswrapper[4405]: W1104 02:32:33.619015 4405 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 04 02:32:33.636650 ip-10-0-134-125 kubenswrapper[4405]: W1104 02:32:33.619020 4405 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 04 02:32:33.636650 ip-10-0-134-125 kubenswrapper[4405]: W1104 02:32:33.619023 4405 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 04 02:32:33.636650 ip-10-0-134-125 kubenswrapper[4405]: W1104 02:32:33.619025 4405 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 04 02:32:33.636650 ip-10-0-134-125 kubenswrapper[4405]: W1104 02:32:33.619028 4405 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 04 02:32:33.636650 ip-10-0-134-125 kubenswrapper[4405]: W1104 02:32:33.619030 4405 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 04 02:32:33.636650 ip-10-0-134-125 kubenswrapper[4405]: W1104 02:32:33.619033 4405 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 04 02:32:33.636650 ip-10-0-134-125 kubenswrapper[4405]: W1104 02:32:33.619035 4405 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 04 02:32:33.636650 ip-10-0-134-125 kubenswrapper[4405]: W1104 02:32:33.619038 4405 feature_gate.go:386] unrecognized feature gate: Example Nov 04 02:32:33.636650 ip-10-0-134-125 kubenswrapper[4405]: W1104 02:32:33.619041 4405 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 04 02:32:33.636650 ip-10-0-134-125 kubenswrapper[4405]: W1104 02:32:33.619043 4405 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 04 02:32:33.636650 ip-10-0-134-125 kubenswrapper[4405]: W1104 02:32:33.619047 4405 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 04 02:32:33.636650 ip-10-0-134-125 kubenswrapper[4405]: W1104 02:32:33.619050 4405 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 04 02:32:33.637216 ip-10-0-134-125 kubenswrapper[4405]: W1104 02:32:33.619053 4405 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 04 02:32:33.637216 ip-10-0-134-125 kubenswrapper[4405]: W1104 02:32:33.619056 4405 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 04 02:32:33.637216 ip-10-0-134-125 kubenswrapper[4405]: W1104 02:32:33.619058 4405 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 04 02:32:33.637216 ip-10-0-134-125 kubenswrapper[4405]: W1104 02:32:33.619061 4405 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 04 02:32:33.637216 ip-10-0-134-125 kubenswrapper[4405]: W1104 02:32:33.619064 4405 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 04 02:32:33.637216 ip-10-0-134-125 kubenswrapper[4405]: W1104 02:32:33.619066 4405 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 04 02:32:33.637216 ip-10-0-134-125 kubenswrapper[4405]: W1104 02:32:33.619069 4405 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 04 02:32:33.637216 ip-10-0-134-125 kubenswrapper[4405]: W1104 02:32:33.619072 4405 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 04 02:32:33.637216 ip-10-0-134-125 kubenswrapper[4405]: W1104 02:32:33.619074 4405 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 04 02:32:33.637216 ip-10-0-134-125 kubenswrapper[4405]: W1104 02:32:33.619077 4405 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 04 02:32:33.637216 ip-10-0-134-125 kubenswrapper[4405]: W1104 02:32:33.619079 4405 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 04 02:32:33.637216 ip-10-0-134-125 kubenswrapper[4405]: W1104 02:32:33.619082 4405 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 04 02:32:33.637216 ip-10-0-134-125 kubenswrapper[4405]: W1104 02:32:33.619085 4405 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 04 02:32:33.637216 ip-10-0-134-125 kubenswrapper[4405]: W1104 02:32:33.619087 4405 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 04 02:32:33.637216 ip-10-0-134-125 kubenswrapper[4405]: W1104 02:32:33.619090 4405 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 04 02:32:33.637216 ip-10-0-134-125 kubenswrapper[4405]: W1104 02:32:33.619092 4405 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 04 02:32:33.637216 ip-10-0-134-125 kubenswrapper[4405]: W1104 02:32:33.619095 4405 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 04 02:32:33.637216 ip-10-0-134-125 kubenswrapper[4405]: W1104 02:32:33.619097 4405 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 04 02:32:33.637216 ip-10-0-134-125 kubenswrapper[4405]: W1104 02:32:33.619100 4405 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 04 02:32:33.637828 ip-10-0-134-125 kubenswrapper[4405]: W1104 02:32:33.619102 4405 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 04 02:32:33.637828 ip-10-0-134-125 kubenswrapper[4405]: W1104 02:32:33.619107 4405 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 04 02:32:33.637828 ip-10-0-134-125 kubenswrapper[4405]: W1104 02:32:33.619110 4405 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 04 02:32:33.637828 ip-10-0-134-125 kubenswrapper[4405]: W1104 02:32:33.619112 4405 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 04 02:32:33.637828 ip-10-0-134-125 kubenswrapper[4405]: W1104 02:32:33.619115 4405 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 04 02:32:33.637828 ip-10-0-134-125 kubenswrapper[4405]: W1104 02:32:33.619117 4405 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 04 02:32:33.637828 ip-10-0-134-125 kubenswrapper[4405]: W1104 02:32:33.619120 4405 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 04 02:32:33.637828 ip-10-0-134-125 kubenswrapper[4405]: W1104 02:32:33.619122 4405 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 04 02:32:33.637828 ip-10-0-134-125 kubenswrapper[4405]: W1104 02:32:33.619125 4405 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 04 02:32:33.637828 ip-10-0-134-125 kubenswrapper[4405]: W1104 02:32:33.619128 4405 feature_gate.go:386] unrecognized feature gate: Example2 Nov 04 02:32:33.637828 ip-10-0-134-125 kubenswrapper[4405]: W1104 02:32:33.619130 4405 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 04 02:32:33.637828 ip-10-0-134-125 kubenswrapper[4405]: W1104 02:32:33.619134 4405 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 04 02:32:33.637828 ip-10-0-134-125 kubenswrapper[4405]: W1104 02:32:33.619137 4405 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 04 02:32:33.637828 ip-10-0-134-125 kubenswrapper[4405]: W1104 02:32:33.619140 4405 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 04 02:32:33.637828 ip-10-0-134-125 kubenswrapper[4405]: W1104 02:32:33.619142 4405 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 04 02:32:33.637828 ip-10-0-134-125 kubenswrapper[4405]: W1104 02:32:33.619145 4405 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 04 02:32:33.637828 ip-10-0-134-125 kubenswrapper[4405]: W1104 02:32:33.619148 4405 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 04 02:32:33.637828 ip-10-0-134-125 kubenswrapper[4405]: W1104 02:32:33.619150 4405 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 04 02:32:33.637828 ip-10-0-134-125 kubenswrapper[4405]: W1104 02:32:33.619153 4405 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 04 02:32:33.638331 ip-10-0-134-125 kubenswrapper[4405]: W1104 02:32:33.619155 4405 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 04 02:32:33.638331 ip-10-0-134-125 kubenswrapper[4405]: W1104 02:32:33.619158 4405 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 04 02:32:33.638331 ip-10-0-134-125 kubenswrapper[4405]: W1104 02:32:33.619160 4405 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 04 02:32:33.638331 ip-10-0-134-125 kubenswrapper[4405]: W1104 02:32:33.619163 4405 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 04 02:32:33.638331 ip-10-0-134-125 kubenswrapper[4405]: W1104 02:32:33.619166 4405 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 04 02:32:33.638331 ip-10-0-134-125 kubenswrapper[4405]: W1104 02:32:33.619169 4405 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 04 02:32:33.638331 ip-10-0-134-125 kubenswrapper[4405]: W1104 02:32:33.619171 4405 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 04 02:32:33.638331 ip-10-0-134-125 kubenswrapper[4405]: W1104 02:32:33.619174 4405 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 04 02:32:33.638331 ip-10-0-134-125 kubenswrapper[4405]: W1104 02:32:33.619177 4405 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 04 02:32:33.638331 ip-10-0-134-125 kubenswrapper[4405]: W1104 02:32:33.619179 4405 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 04 02:32:33.638331 ip-10-0-134-125 kubenswrapper[4405]: W1104 02:32:33.619182 4405 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 04 02:32:33.638331 ip-10-0-134-125 kubenswrapper[4405]: W1104 02:32:33.619185 4405 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 04 02:32:33.638331 ip-10-0-134-125 kubenswrapper[4405]: W1104 02:32:33.619187 4405 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 04 02:32:33.638331 ip-10-0-134-125 kubenswrapper[4405]: W1104 02:32:33.619190 4405 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 04 02:32:33.638331 ip-10-0-134-125 kubenswrapper[4405]: W1104 02:32:33.619194 4405 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 04 02:32:33.638331 ip-10-0-134-125 kubenswrapper[4405]: W1104 02:32:33.619196 4405 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 04 02:32:33.638331 ip-10-0-134-125 kubenswrapper[4405]: W1104 02:32:33.619199 4405 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 04 02:32:33.638331 ip-10-0-134-125 kubenswrapper[4405]: W1104 02:32:33.619202 4405 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 04 02:32:33.638331 ip-10-0-134-125 kubenswrapper[4405]: W1104 02:32:33.619204 4405 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 04 02:32:33.638331 ip-10-0-134-125 kubenswrapper[4405]: W1104 02:32:33.619207 4405 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 04 02:32:33.638859 ip-10-0-134-125 kubenswrapper[4405]: W1104 02:32:33.619209 4405 feature_gate.go:386] unrecognized feature gate: EVPN Nov 04 02:32:33.638859 ip-10-0-134-125 kubenswrapper[4405]: W1104 02:32:33.619212 4405 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 04 02:32:33.638859 ip-10-0-134-125 kubenswrapper[4405]: W1104 02:32:33.619215 4405 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 04 02:32:33.638859 ip-10-0-134-125 kubenswrapper[4405]: W1104 02:32:33.619218 4405 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 04 02:32:33.638859 ip-10-0-134-125 kubenswrapper[4405]: W1104 02:32:33.619222 4405 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 04 02:32:33.638859 ip-10-0-134-125 kubenswrapper[4405]: W1104 02:32:33.619225 4405 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 04 02:32:33.638859 ip-10-0-134-125 kubenswrapper[4405]: W1104 02:32:33.619228 4405 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 04 02:32:33.638859 ip-10-0-134-125 kubenswrapper[4405]: W1104 02:32:33.619230 4405 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 04 02:32:33.638859 ip-10-0-134-125 kubenswrapper[4405]: W1104 02:32:33.619233 4405 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 04 02:32:33.638859 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:32:33.624470 4405 server.go:525] "Kubelet version" kubeletVersion="v1.35.9" Nov 04 02:32:33.638859 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:32:33.624490 4405 server.go:527] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Nov 04 02:32:33.638859 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:32:33.624518 4405 watchdog_linux.go:95] "Systemd watchdog is not enabled" Nov 04 02:32:33.638859 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:32:33.624525 4405 watchdog_linux.go:138] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Nov 04 02:32:33.638859 ip-10-0-134-125 kubenswrapper[4405]: W1104 02:32:33.624638 4405 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 04 02:32:33.638859 ip-10-0-134-125 kubenswrapper[4405]: W1104 02:32:33.624646 4405 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 04 02:32:33.638859 ip-10-0-134-125 kubenswrapper[4405]: W1104 02:32:33.624652 4405 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 04 02:32:33.638859 ip-10-0-134-125 kubenswrapper[4405]: W1104 02:32:33.624656 4405 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 04 02:32:33.638859 ip-10-0-134-125 kubenswrapper[4405]: W1104 02:32:33.624661 4405 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 04 02:32:33.638859 ip-10-0-134-125 kubenswrapper[4405]: W1104 02:32:33.624682 4405 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 04 02:32:33.638859 ip-10-0-134-125 kubenswrapper[4405]: W1104 02:32:33.624686 4405 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 04 02:32:33.639389 ip-10-0-134-125 kubenswrapper[4405]: W1104 02:32:33.624690 4405 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 04 02:32:33.639389 ip-10-0-134-125 kubenswrapper[4405]: W1104 02:32:33.624695 4405 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 04 02:32:33.639389 ip-10-0-134-125 kubenswrapper[4405]: W1104 02:32:33.624699 4405 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 04 02:32:33.639389 ip-10-0-134-125 kubenswrapper[4405]: W1104 02:32:33.624704 4405 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 04 02:32:33.639389 ip-10-0-134-125 kubenswrapper[4405]: W1104 02:32:33.624708 4405 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 04 02:32:33.639389 ip-10-0-134-125 kubenswrapper[4405]: W1104 02:32:33.624712 4405 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 04 02:32:33.639389 ip-10-0-134-125 kubenswrapper[4405]: W1104 02:32:33.624717 4405 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 04 02:32:33.639389 ip-10-0-134-125 kubenswrapper[4405]: W1104 02:32:33.624721 4405 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 04 02:32:33.639389 ip-10-0-134-125 kubenswrapper[4405]: W1104 02:32:33.624734 4405 feature_gate.go:386] unrecognized feature gate: EVPN Nov 04 02:32:33.639389 ip-10-0-134-125 kubenswrapper[4405]: W1104 02:32:33.624740 4405 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 04 02:32:33.639389 ip-10-0-134-125 kubenswrapper[4405]: W1104 02:32:33.624744 4405 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 04 02:32:33.639389 ip-10-0-134-125 kubenswrapper[4405]: W1104 02:32:33.624749 4405 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 04 02:32:33.639389 ip-10-0-134-125 kubenswrapper[4405]: W1104 02:32:33.624754 4405 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 04 02:32:33.639389 ip-10-0-134-125 kubenswrapper[4405]: W1104 02:32:33.624758 4405 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 04 02:32:33.639389 ip-10-0-134-125 kubenswrapper[4405]: W1104 02:32:33.624763 4405 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 04 02:32:33.639389 ip-10-0-134-125 kubenswrapper[4405]: W1104 02:32:33.624767 4405 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 04 02:32:33.639389 ip-10-0-134-125 kubenswrapper[4405]: W1104 02:32:33.624771 4405 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 04 02:32:33.639389 ip-10-0-134-125 kubenswrapper[4405]: W1104 02:32:33.624775 4405 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 04 02:32:33.639389 ip-10-0-134-125 kubenswrapper[4405]: W1104 02:32:33.624780 4405 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 04 02:32:33.639903 ip-10-0-134-125 kubenswrapper[4405]: W1104 02:32:33.624784 4405 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 04 02:32:33.639903 ip-10-0-134-125 kubenswrapper[4405]: W1104 02:32:33.624789 4405 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 04 02:32:33.639903 ip-10-0-134-125 kubenswrapper[4405]: W1104 02:32:33.624794 4405 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 04 02:32:33.639903 ip-10-0-134-125 kubenswrapper[4405]: W1104 02:32:33.624799 4405 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 04 02:32:33.639903 ip-10-0-134-125 kubenswrapper[4405]: W1104 02:32:33.624803 4405 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 04 02:32:33.639903 ip-10-0-134-125 kubenswrapper[4405]: W1104 02:32:33.624807 4405 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 04 02:32:33.639903 ip-10-0-134-125 kubenswrapper[4405]: W1104 02:32:33.624812 4405 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 04 02:32:33.639903 ip-10-0-134-125 kubenswrapper[4405]: W1104 02:32:33.624815 4405 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 04 02:32:33.639903 ip-10-0-134-125 kubenswrapper[4405]: W1104 02:32:33.624820 4405 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 04 02:32:33.639903 ip-10-0-134-125 kubenswrapper[4405]: W1104 02:32:33.624823 4405 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 04 02:32:33.639903 ip-10-0-134-125 kubenswrapper[4405]: W1104 02:32:33.624827 4405 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 04 02:32:33.639903 ip-10-0-134-125 kubenswrapper[4405]: W1104 02:32:33.624831 4405 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 04 02:32:33.639903 ip-10-0-134-125 kubenswrapper[4405]: W1104 02:32:33.624835 4405 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 04 02:32:33.639903 ip-10-0-134-125 kubenswrapper[4405]: W1104 02:32:33.624839 4405 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 04 02:32:33.639903 ip-10-0-134-125 kubenswrapper[4405]: W1104 02:32:33.624843 4405 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 04 02:32:33.639903 ip-10-0-134-125 kubenswrapper[4405]: W1104 02:32:33.624847 4405 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 04 02:32:33.639903 ip-10-0-134-125 kubenswrapper[4405]: W1104 02:32:33.624852 4405 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 04 02:32:33.639903 ip-10-0-134-125 kubenswrapper[4405]: W1104 02:32:33.624857 4405 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 04 02:32:33.639903 ip-10-0-134-125 kubenswrapper[4405]: W1104 02:32:33.624861 4405 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 04 02:32:33.639903 ip-10-0-134-125 kubenswrapper[4405]: W1104 02:32:33.624865 4405 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 04 02:32:33.640432 ip-10-0-134-125 kubenswrapper[4405]: W1104 02:32:33.624870 4405 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 04 02:32:33.640432 ip-10-0-134-125 kubenswrapper[4405]: W1104 02:32:33.624877 4405 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 04 02:32:33.640432 ip-10-0-134-125 kubenswrapper[4405]: W1104 02:32:33.624893 4405 feature_gate.go:386] unrecognized feature gate: Example Nov 04 02:32:33.640432 ip-10-0-134-125 kubenswrapper[4405]: W1104 02:32:33.624898 4405 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 04 02:32:33.640432 ip-10-0-134-125 kubenswrapper[4405]: W1104 02:32:33.624903 4405 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 04 02:32:33.640432 ip-10-0-134-125 kubenswrapper[4405]: W1104 02:32:33.624907 4405 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 04 02:32:33.640432 ip-10-0-134-125 kubenswrapper[4405]: W1104 02:32:33.624911 4405 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 04 02:32:33.640432 ip-10-0-134-125 kubenswrapper[4405]: W1104 02:32:33.624916 4405 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 04 02:32:33.640432 ip-10-0-134-125 kubenswrapper[4405]: W1104 02:32:33.624920 4405 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 04 02:32:33.640432 ip-10-0-134-125 kubenswrapper[4405]: W1104 02:32:33.624925 4405 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 04 02:32:33.640432 ip-10-0-134-125 kubenswrapper[4405]: W1104 02:32:33.624929 4405 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 04 02:32:33.640432 ip-10-0-134-125 kubenswrapper[4405]: W1104 02:32:33.624933 4405 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 04 02:32:33.640432 ip-10-0-134-125 kubenswrapper[4405]: W1104 02:32:33.624938 4405 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 04 02:32:33.640432 ip-10-0-134-125 kubenswrapper[4405]: W1104 02:32:33.624942 4405 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 04 02:32:33.640432 ip-10-0-134-125 kubenswrapper[4405]: W1104 02:32:33.624946 4405 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 04 02:32:33.640432 ip-10-0-134-125 kubenswrapper[4405]: W1104 02:32:33.624951 4405 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 04 02:32:33.640432 ip-10-0-134-125 kubenswrapper[4405]: W1104 02:32:33.624955 4405 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 04 02:32:33.640432 ip-10-0-134-125 kubenswrapper[4405]: W1104 02:32:33.624959 4405 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 04 02:32:33.640432 ip-10-0-134-125 kubenswrapper[4405]: W1104 02:32:33.624964 4405 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 04 02:32:33.640955 ip-10-0-134-125 kubenswrapper[4405]: W1104 02:32:33.624968 4405 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 04 02:32:33.640955 ip-10-0-134-125 kubenswrapper[4405]: W1104 02:32:33.624972 4405 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 04 02:32:33.640955 ip-10-0-134-125 kubenswrapper[4405]: W1104 02:32:33.624976 4405 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 04 02:32:33.640955 ip-10-0-134-125 kubenswrapper[4405]: W1104 02:32:33.624981 4405 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 04 02:32:33.640955 ip-10-0-134-125 kubenswrapper[4405]: W1104 02:32:33.624985 4405 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 04 02:32:33.640955 ip-10-0-134-125 kubenswrapper[4405]: W1104 02:32:33.624989 4405 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 04 02:32:33.640955 ip-10-0-134-125 kubenswrapper[4405]: W1104 02:32:33.624993 4405 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 04 02:32:33.640955 ip-10-0-134-125 kubenswrapper[4405]: W1104 02:32:33.625000 4405 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 04 02:32:33.640955 ip-10-0-134-125 kubenswrapper[4405]: W1104 02:32:33.625006 4405 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 04 02:32:33.640955 ip-10-0-134-125 kubenswrapper[4405]: W1104 02:32:33.625011 4405 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 04 02:32:33.640955 ip-10-0-134-125 kubenswrapper[4405]: W1104 02:32:33.625016 4405 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 04 02:32:33.640955 ip-10-0-134-125 kubenswrapper[4405]: W1104 02:32:33.625022 4405 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 04 02:32:33.640955 ip-10-0-134-125 kubenswrapper[4405]: W1104 02:32:33.625026 4405 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 04 02:32:33.640955 ip-10-0-134-125 kubenswrapper[4405]: W1104 02:32:33.625030 4405 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 04 02:32:33.640955 ip-10-0-134-125 kubenswrapper[4405]: W1104 02:32:33.625035 4405 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 04 02:32:33.640955 ip-10-0-134-125 kubenswrapper[4405]: W1104 02:32:33.625039 4405 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 04 02:32:33.640955 ip-10-0-134-125 kubenswrapper[4405]: W1104 02:32:33.625053 4405 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 04 02:32:33.640955 ip-10-0-134-125 kubenswrapper[4405]: W1104 02:32:33.625058 4405 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 04 02:32:33.640955 ip-10-0-134-125 kubenswrapper[4405]: W1104 02:32:33.625062 4405 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 04 02:32:33.640955 ip-10-0-134-125 kubenswrapper[4405]: W1104 02:32:33.625067 4405 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 04 02:32:33.641479 ip-10-0-134-125 kubenswrapper[4405]: W1104 02:32:33.625071 4405 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 04 02:32:33.641479 ip-10-0-134-125 kubenswrapper[4405]: W1104 02:32:33.625075 4405 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 04 02:32:33.641479 ip-10-0-134-125 kubenswrapper[4405]: W1104 02:32:33.625079 4405 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 04 02:32:33.641479 ip-10-0-134-125 kubenswrapper[4405]: W1104 02:32:33.625084 4405 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 04 02:32:33.641479 ip-10-0-134-125 kubenswrapper[4405]: W1104 02:32:33.625088 4405 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 04 02:32:33.641479 ip-10-0-134-125 kubenswrapper[4405]: W1104 02:32:33.625092 4405 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 04 02:32:33.641479 ip-10-0-134-125 kubenswrapper[4405]: W1104 02:32:33.625097 4405 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 04 02:32:33.641479 ip-10-0-134-125 kubenswrapper[4405]: W1104 02:32:33.625101 4405 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 04 02:32:33.641479 ip-10-0-134-125 kubenswrapper[4405]: W1104 02:32:33.625105 4405 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 04 02:32:33.641479 ip-10-0-134-125 kubenswrapper[4405]: W1104 02:32:33.625109 4405 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 04 02:32:33.641479 ip-10-0-134-125 kubenswrapper[4405]: W1104 02:32:33.625114 4405 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 04 02:32:33.641479 ip-10-0-134-125 kubenswrapper[4405]: W1104 02:32:33.625118 4405 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 04 02:32:33.641479 ip-10-0-134-125 kubenswrapper[4405]: W1104 02:32:33.625122 4405 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 04 02:32:33.641479 ip-10-0-134-125 kubenswrapper[4405]: W1104 02:32:33.625126 4405 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 04 02:32:33.641479 ip-10-0-134-125 kubenswrapper[4405]: W1104 02:32:33.625130 4405 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 04 02:32:33.641479 ip-10-0-134-125 kubenswrapper[4405]: W1104 02:32:33.625135 4405 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 04 02:32:33.641479 ip-10-0-134-125 kubenswrapper[4405]: W1104 02:32:33.625139 4405 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 04 02:32:33.641479 ip-10-0-134-125 kubenswrapper[4405]: W1104 02:32:33.625143 4405 feature_gate.go:386] unrecognized feature gate: Example2 Nov 04 02:32:33.641479 ip-10-0-134-125 kubenswrapper[4405]: W1104 02:32:33.625148 4405 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 04 02:32:33.641479 ip-10-0-134-125 kubenswrapper[4405]: W1104 02:32:33.625152 4405 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 04 02:32:33.642090 ip-10-0-134-125 kubenswrapper[4405]: W1104 02:32:33.625156 4405 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 04 02:32:33.642090 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:32:33.625198 4405 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Nov 04 02:32:33.642090 ip-10-0-134-125 kubenswrapper[4405]: W1104 02:32:33.625493 4405 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 04 02:32:33.642090 ip-10-0-134-125 kubenswrapper[4405]: W1104 02:32:33.625502 4405 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 04 02:32:33.642090 ip-10-0-134-125 kubenswrapper[4405]: W1104 02:32:33.625509 4405 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 04 02:32:33.642090 ip-10-0-134-125 kubenswrapper[4405]: W1104 02:32:33.625515 4405 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 04 02:32:33.642090 ip-10-0-134-125 kubenswrapper[4405]: W1104 02:32:33.625519 4405 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 04 02:32:33.642090 ip-10-0-134-125 kubenswrapper[4405]: W1104 02:32:33.625523 4405 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 04 02:32:33.642090 ip-10-0-134-125 kubenswrapper[4405]: W1104 02:32:33.625528 4405 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 04 02:32:33.642090 ip-10-0-134-125 kubenswrapper[4405]: W1104 02:32:33.625540 4405 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 04 02:32:33.642090 ip-10-0-134-125 kubenswrapper[4405]: W1104 02:32:33.625545 4405 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 04 02:32:33.642090 ip-10-0-134-125 kubenswrapper[4405]: W1104 02:32:33.625549 4405 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 04 02:32:33.642090 ip-10-0-134-125 kubenswrapper[4405]: W1104 02:32:33.625554 4405 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 04 02:32:33.642090 ip-10-0-134-125 kubenswrapper[4405]: W1104 02:32:33.625558 4405 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 04 02:32:33.642090 ip-10-0-134-125 kubenswrapper[4405]: W1104 02:32:33.625562 4405 feature_gate.go:386] unrecognized feature gate: Example2 Nov 04 02:32:33.642090 ip-10-0-134-125 kubenswrapper[4405]: W1104 02:32:33.625567 4405 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 04 02:32:33.642518 ip-10-0-134-125 kubenswrapper[4405]: W1104 02:32:33.625571 4405 feature_gate.go:386] unrecognized feature gate: Example Nov 04 02:32:33.642518 ip-10-0-134-125 kubenswrapper[4405]: W1104 02:32:33.625575 4405 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 04 02:32:33.642518 ip-10-0-134-125 kubenswrapper[4405]: W1104 02:32:33.625580 4405 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 04 02:32:33.642518 ip-10-0-134-125 kubenswrapper[4405]: W1104 02:32:33.625584 4405 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 04 02:32:33.642518 ip-10-0-134-125 kubenswrapper[4405]: W1104 02:32:33.625588 4405 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 04 02:32:33.642518 ip-10-0-134-125 kubenswrapper[4405]: W1104 02:32:33.625592 4405 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 04 02:32:33.642518 ip-10-0-134-125 kubenswrapper[4405]: W1104 02:32:33.625596 4405 feature_gate.go:386] unrecognized feature gate: EVPN Nov 04 02:32:33.642518 ip-10-0-134-125 kubenswrapper[4405]: W1104 02:32:33.625601 4405 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 04 02:32:33.642518 ip-10-0-134-125 kubenswrapper[4405]: W1104 02:32:33.625608 4405 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 04 02:32:33.642518 ip-10-0-134-125 kubenswrapper[4405]: W1104 02:32:33.625613 4405 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 04 02:32:33.642518 ip-10-0-134-125 kubenswrapper[4405]: W1104 02:32:33.625617 4405 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 04 02:32:33.642518 ip-10-0-134-125 kubenswrapper[4405]: W1104 02:32:33.625621 4405 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 04 02:32:33.642518 ip-10-0-134-125 kubenswrapper[4405]: W1104 02:32:33.625626 4405 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 04 02:32:33.642518 ip-10-0-134-125 kubenswrapper[4405]: W1104 02:32:33.625630 4405 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 04 02:32:33.642518 ip-10-0-134-125 kubenswrapper[4405]: W1104 02:32:33.625637 4405 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 04 02:32:33.642518 ip-10-0-134-125 kubenswrapper[4405]: W1104 02:32:33.625643 4405 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 04 02:32:33.642518 ip-10-0-134-125 kubenswrapper[4405]: W1104 02:32:33.625648 4405 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 04 02:32:33.642518 ip-10-0-134-125 kubenswrapper[4405]: W1104 02:32:33.625653 4405 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 04 02:32:33.642518 ip-10-0-134-125 kubenswrapper[4405]: W1104 02:32:33.625658 4405 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 04 02:32:33.643036 ip-10-0-134-125 kubenswrapper[4405]: W1104 02:32:33.625662 4405 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 04 02:32:33.643036 ip-10-0-134-125 kubenswrapper[4405]: W1104 02:32:33.625685 4405 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 04 02:32:33.643036 ip-10-0-134-125 kubenswrapper[4405]: W1104 02:32:33.625690 4405 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 04 02:32:33.643036 ip-10-0-134-125 kubenswrapper[4405]: W1104 02:32:33.625694 4405 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 04 02:32:33.643036 ip-10-0-134-125 kubenswrapper[4405]: W1104 02:32:33.625698 4405 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 04 02:32:33.643036 ip-10-0-134-125 kubenswrapper[4405]: W1104 02:32:33.625702 4405 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 04 02:32:33.643036 ip-10-0-134-125 kubenswrapper[4405]: W1104 02:32:33.625706 4405 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 04 02:32:33.643036 ip-10-0-134-125 kubenswrapper[4405]: W1104 02:32:33.625719 4405 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 04 02:32:33.643036 ip-10-0-134-125 kubenswrapper[4405]: W1104 02:32:33.625723 4405 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 04 02:32:33.643036 ip-10-0-134-125 kubenswrapper[4405]: W1104 02:32:33.625728 4405 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 04 02:32:33.643036 ip-10-0-134-125 kubenswrapper[4405]: W1104 02:32:33.625732 4405 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 04 02:32:33.643036 ip-10-0-134-125 kubenswrapper[4405]: W1104 02:32:33.625736 4405 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 04 02:32:33.643036 ip-10-0-134-125 kubenswrapper[4405]: W1104 02:32:33.625740 4405 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 04 02:32:33.643036 ip-10-0-134-125 kubenswrapper[4405]: W1104 02:32:33.625744 4405 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 04 02:32:33.643036 ip-10-0-134-125 kubenswrapper[4405]: W1104 02:32:33.625749 4405 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 04 02:32:33.643036 ip-10-0-134-125 kubenswrapper[4405]: W1104 02:32:33.625753 4405 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 04 02:32:33.643036 ip-10-0-134-125 kubenswrapper[4405]: W1104 02:32:33.625757 4405 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 04 02:32:33.643036 ip-10-0-134-125 kubenswrapper[4405]: W1104 02:32:33.625761 4405 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 04 02:32:33.643036 ip-10-0-134-125 kubenswrapper[4405]: W1104 02:32:33.625766 4405 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 04 02:32:33.643545 ip-10-0-134-125 kubenswrapper[4405]: W1104 02:32:33.625770 4405 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 04 02:32:33.643545 ip-10-0-134-125 kubenswrapper[4405]: W1104 02:32:33.625774 4405 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 04 02:32:33.643545 ip-10-0-134-125 kubenswrapper[4405]: W1104 02:32:33.625781 4405 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 04 02:32:33.643545 ip-10-0-134-125 kubenswrapper[4405]: W1104 02:32:33.625786 4405 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 04 02:32:33.643545 ip-10-0-134-125 kubenswrapper[4405]: W1104 02:32:33.625790 4405 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 04 02:32:33.643545 ip-10-0-134-125 kubenswrapper[4405]: W1104 02:32:33.625794 4405 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 04 02:32:33.643545 ip-10-0-134-125 kubenswrapper[4405]: W1104 02:32:33.625799 4405 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 04 02:32:33.643545 ip-10-0-134-125 kubenswrapper[4405]: W1104 02:32:33.625803 4405 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 04 02:32:33.643545 ip-10-0-134-125 kubenswrapper[4405]: W1104 02:32:33.625807 4405 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 04 02:32:33.643545 ip-10-0-134-125 kubenswrapper[4405]: W1104 02:32:33.625812 4405 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 04 02:32:33.643545 ip-10-0-134-125 kubenswrapper[4405]: W1104 02:32:33.625816 4405 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 04 02:32:33.643545 ip-10-0-134-125 kubenswrapper[4405]: W1104 02:32:33.625821 4405 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 04 02:32:33.643545 ip-10-0-134-125 kubenswrapper[4405]: W1104 02:32:33.625825 4405 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 04 02:32:33.643545 ip-10-0-134-125 kubenswrapper[4405]: W1104 02:32:33.625829 4405 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 04 02:32:33.643545 ip-10-0-134-125 kubenswrapper[4405]: W1104 02:32:33.625833 4405 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 04 02:32:33.643545 ip-10-0-134-125 kubenswrapper[4405]: W1104 02:32:33.625838 4405 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 04 02:32:33.643545 ip-10-0-134-125 kubenswrapper[4405]: W1104 02:32:33.625842 4405 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 04 02:32:33.643545 ip-10-0-134-125 kubenswrapper[4405]: W1104 02:32:33.625846 4405 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 04 02:32:33.643545 ip-10-0-134-125 kubenswrapper[4405]: W1104 02:32:33.625850 4405 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 04 02:32:33.643545 ip-10-0-134-125 kubenswrapper[4405]: W1104 02:32:33.625855 4405 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 04 02:32:33.644091 ip-10-0-134-125 kubenswrapper[4405]: W1104 02:32:33.625859 4405 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 04 02:32:33.644091 ip-10-0-134-125 kubenswrapper[4405]: W1104 02:32:33.625871 4405 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 04 02:32:33.644091 ip-10-0-134-125 kubenswrapper[4405]: W1104 02:32:33.625876 4405 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 04 02:32:33.644091 ip-10-0-134-125 kubenswrapper[4405]: W1104 02:32:33.625880 4405 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 04 02:32:33.644091 ip-10-0-134-125 kubenswrapper[4405]: W1104 02:32:33.625884 4405 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 04 02:32:33.644091 ip-10-0-134-125 kubenswrapper[4405]: W1104 02:32:33.625888 4405 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 04 02:32:33.644091 ip-10-0-134-125 kubenswrapper[4405]: W1104 02:32:33.625893 4405 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 04 02:32:33.644091 ip-10-0-134-125 kubenswrapper[4405]: W1104 02:32:33.625897 4405 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 04 02:32:33.644091 ip-10-0-134-125 kubenswrapper[4405]: W1104 02:32:33.625902 4405 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 04 02:32:33.644091 ip-10-0-134-125 kubenswrapper[4405]: W1104 02:32:33.625907 4405 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 04 02:32:33.644091 ip-10-0-134-125 kubenswrapper[4405]: W1104 02:32:33.625911 4405 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 04 02:32:33.644091 ip-10-0-134-125 kubenswrapper[4405]: W1104 02:32:33.625916 4405 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 04 02:32:33.644091 ip-10-0-134-125 kubenswrapper[4405]: W1104 02:32:33.625920 4405 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 04 02:32:33.644091 ip-10-0-134-125 kubenswrapper[4405]: W1104 02:32:33.625924 4405 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 04 02:32:33.644091 ip-10-0-134-125 kubenswrapper[4405]: W1104 02:32:33.625930 4405 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 04 02:32:33.644091 ip-10-0-134-125 kubenswrapper[4405]: W1104 02:32:33.625935 4405 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 04 02:32:33.644091 ip-10-0-134-125 kubenswrapper[4405]: W1104 02:32:33.625950 4405 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 04 02:32:33.644091 ip-10-0-134-125 kubenswrapper[4405]: W1104 02:32:33.625955 4405 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 04 02:32:33.644091 ip-10-0-134-125 kubenswrapper[4405]: W1104 02:32:33.625959 4405 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 04 02:32:33.644091 ip-10-0-134-125 kubenswrapper[4405]: W1104 02:32:33.625964 4405 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 04 02:32:33.644630 ip-10-0-134-125 kubenswrapper[4405]: W1104 02:32:33.625968 4405 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 04 02:32:33.644630 ip-10-0-134-125 kubenswrapper[4405]: W1104 02:32:33.625972 4405 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 04 02:32:33.644630 ip-10-0-134-125 kubenswrapper[4405]: W1104 02:32:33.625976 4405 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 04 02:32:33.644630 ip-10-0-134-125 kubenswrapper[4405]: W1104 02:32:33.625981 4405 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 04 02:32:33.644630 ip-10-0-134-125 kubenswrapper[4405]: W1104 02:32:33.625986 4405 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 04 02:32:33.644630 ip-10-0-134-125 kubenswrapper[4405]: W1104 02:32:33.625990 4405 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 04 02:32:33.644630 ip-10-0-134-125 kubenswrapper[4405]: W1104 02:32:33.625995 4405 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 04 02:32:33.644630 ip-10-0-134-125 kubenswrapper[4405]: W1104 02:32:33.625999 4405 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 04 02:32:33.644630 ip-10-0-134-125 kubenswrapper[4405]: W1104 02:32:33.626003 4405 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 04 02:32:33.644630 ip-10-0-134-125 kubenswrapper[4405]: W1104 02:32:33.626008 4405 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 04 02:32:33.644630 ip-10-0-134-125 kubenswrapper[4405]: W1104 02:32:33.626012 4405 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 04 02:32:33.644630 ip-10-0-134-125 kubenswrapper[4405]: W1104 02:32:33.626016 4405 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 04 02:32:33.644630 ip-10-0-134-125 kubenswrapper[4405]: W1104 02:32:33.626020 4405 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 04 02:32:33.644630 ip-10-0-134-125 kubenswrapper[4405]: W1104 02:32:33.626028 4405 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 04 02:32:33.644630 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:32:33.626079 4405 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Nov 04 02:32:33.644630 ip-10-0-134-125 kubenswrapper[4405]: W1104 02:32:33.626149 4405 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 04 02:32:33.645124 ip-10-0-134-125 kubenswrapper[4405]: W1104 02:32:33.626156 4405 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 04 02:32:33.645124 ip-10-0-134-125 kubenswrapper[4405]: W1104 02:32:33.626161 4405 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 04 02:32:33.645124 ip-10-0-134-125 kubenswrapper[4405]: W1104 02:32:33.626166 4405 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 04 02:32:33.645124 ip-10-0-134-125 kubenswrapper[4405]: W1104 02:32:33.626170 4405 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 04 02:32:33.645124 ip-10-0-134-125 kubenswrapper[4405]: W1104 02:32:33.626175 4405 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 04 02:32:33.645124 ip-10-0-134-125 kubenswrapper[4405]: W1104 02:32:33.626179 4405 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 04 02:32:33.645124 ip-10-0-134-125 kubenswrapper[4405]: W1104 02:32:33.626184 4405 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 04 02:32:33.645124 ip-10-0-134-125 kubenswrapper[4405]: W1104 02:32:33.626188 4405 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 04 02:32:33.645124 ip-10-0-134-125 kubenswrapper[4405]: W1104 02:32:33.626192 4405 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 04 02:32:33.645124 ip-10-0-134-125 kubenswrapper[4405]: W1104 02:32:33.626197 4405 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 04 02:32:33.645124 ip-10-0-134-125 kubenswrapper[4405]: W1104 02:32:33.626201 4405 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 04 02:32:33.645124 ip-10-0-134-125 kubenswrapper[4405]: W1104 02:32:33.626208 4405 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 04 02:32:33.645124 ip-10-0-134-125 kubenswrapper[4405]: W1104 02:32:33.626213 4405 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 04 02:32:33.645124 ip-10-0-134-125 kubenswrapper[4405]: W1104 02:32:33.626217 4405 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 04 02:32:33.645124 ip-10-0-134-125 kubenswrapper[4405]: W1104 02:32:33.626221 4405 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 04 02:32:33.645124 ip-10-0-134-125 kubenswrapper[4405]: W1104 02:32:33.626226 4405 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 04 02:32:33.645124 ip-10-0-134-125 kubenswrapper[4405]: W1104 02:32:33.626230 4405 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 04 02:32:33.645124 ip-10-0-134-125 kubenswrapper[4405]: W1104 02:32:33.626234 4405 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 04 02:32:33.645124 ip-10-0-134-125 kubenswrapper[4405]: W1104 02:32:33.626241 4405 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 04 02:32:33.645609 ip-10-0-134-125 kubenswrapper[4405]: W1104 02:32:33.626246 4405 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 04 02:32:33.645609 ip-10-0-134-125 kubenswrapper[4405]: W1104 02:32:33.626253 4405 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 04 02:32:33.645609 ip-10-0-134-125 kubenswrapper[4405]: W1104 02:32:33.626258 4405 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 04 02:32:33.645609 ip-10-0-134-125 kubenswrapper[4405]: W1104 02:32:33.626263 4405 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 04 02:32:33.645609 ip-10-0-134-125 kubenswrapper[4405]: W1104 02:32:33.626267 4405 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 04 02:32:33.645609 ip-10-0-134-125 kubenswrapper[4405]: W1104 02:32:33.626272 4405 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 04 02:32:33.645609 ip-10-0-134-125 kubenswrapper[4405]: W1104 02:32:33.626276 4405 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 04 02:32:33.645609 ip-10-0-134-125 kubenswrapper[4405]: W1104 02:32:33.626280 4405 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 04 02:32:33.645609 ip-10-0-134-125 kubenswrapper[4405]: W1104 02:32:33.626284 4405 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 04 02:32:33.645609 ip-10-0-134-125 kubenswrapper[4405]: W1104 02:32:33.626288 4405 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 04 02:32:33.645609 ip-10-0-134-125 kubenswrapper[4405]: W1104 02:32:33.626295 4405 feature_gate.go:386] unrecognized feature gate: Example2 Nov 04 02:32:33.645609 ip-10-0-134-125 kubenswrapper[4405]: W1104 02:32:33.626299 4405 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 04 02:32:33.645609 ip-10-0-134-125 kubenswrapper[4405]: W1104 02:32:33.626312 4405 feature_gate.go:386] unrecognized feature gate: Example Nov 04 02:32:33.645609 ip-10-0-134-125 kubenswrapper[4405]: W1104 02:32:33.626317 4405 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 04 02:32:33.645609 ip-10-0-134-125 kubenswrapper[4405]: W1104 02:32:33.626321 4405 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 04 02:32:33.645609 ip-10-0-134-125 kubenswrapper[4405]: W1104 02:32:33.626325 4405 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 04 02:32:33.645609 ip-10-0-134-125 kubenswrapper[4405]: W1104 02:32:33.626330 4405 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 04 02:32:33.645609 ip-10-0-134-125 kubenswrapper[4405]: W1104 02:32:33.626334 4405 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 04 02:32:33.645609 ip-10-0-134-125 kubenswrapper[4405]: W1104 02:32:33.626338 4405 feature_gate.go:386] unrecognized feature gate: EVPN Nov 04 02:32:33.645609 ip-10-0-134-125 kubenswrapper[4405]: W1104 02:32:33.626342 4405 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 04 02:32:33.646140 ip-10-0-134-125 kubenswrapper[4405]: W1104 02:32:33.626347 4405 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 04 02:32:33.646140 ip-10-0-134-125 kubenswrapper[4405]: W1104 02:32:33.626351 4405 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 04 02:32:33.646140 ip-10-0-134-125 kubenswrapper[4405]: W1104 02:32:33.626355 4405 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 04 02:32:33.646140 ip-10-0-134-125 kubenswrapper[4405]: W1104 02:32:33.626359 4405 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 04 02:32:33.646140 ip-10-0-134-125 kubenswrapper[4405]: W1104 02:32:33.626366 4405 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 04 02:32:33.646140 ip-10-0-134-125 kubenswrapper[4405]: W1104 02:32:33.626370 4405 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 04 02:32:33.646140 ip-10-0-134-125 kubenswrapper[4405]: W1104 02:32:33.626377 4405 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 04 02:32:33.646140 ip-10-0-134-125 kubenswrapper[4405]: W1104 02:32:33.626383 4405 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 04 02:32:33.646140 ip-10-0-134-125 kubenswrapper[4405]: W1104 02:32:33.626390 4405 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 04 02:32:33.646140 ip-10-0-134-125 kubenswrapper[4405]: W1104 02:32:33.626395 4405 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 04 02:32:33.646140 ip-10-0-134-125 kubenswrapper[4405]: W1104 02:32:33.626399 4405 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 04 02:32:33.646140 ip-10-0-134-125 kubenswrapper[4405]: W1104 02:32:33.626404 4405 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 04 02:32:33.646140 ip-10-0-134-125 kubenswrapper[4405]: W1104 02:32:33.626408 4405 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 04 02:32:33.646140 ip-10-0-134-125 kubenswrapper[4405]: W1104 02:32:33.626412 4405 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 04 02:32:33.646140 ip-10-0-134-125 kubenswrapper[4405]: W1104 02:32:33.626417 4405 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 04 02:32:33.646140 ip-10-0-134-125 kubenswrapper[4405]: W1104 02:32:33.626421 4405 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 04 02:32:33.646140 ip-10-0-134-125 kubenswrapper[4405]: W1104 02:32:33.626425 4405 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 04 02:32:33.646140 ip-10-0-134-125 kubenswrapper[4405]: W1104 02:32:33.626430 4405 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 04 02:32:33.646908 ip-10-0-134-125 kubenswrapper[4405]: W1104 02:32:33.626433 4405 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 04 02:32:33.646908 ip-10-0-134-125 kubenswrapper[4405]: W1104 02:32:33.626437 4405 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 04 02:32:33.646908 ip-10-0-134-125 kubenswrapper[4405]: W1104 02:32:33.626441 4405 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 04 02:32:33.646908 ip-10-0-134-125 kubenswrapper[4405]: W1104 02:32:33.626446 4405 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 04 02:32:33.646908 ip-10-0-134-125 kubenswrapper[4405]: W1104 02:32:33.626452 4405 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 04 02:32:33.646908 ip-10-0-134-125 kubenswrapper[4405]: W1104 02:32:33.626457 4405 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 04 02:32:33.646908 ip-10-0-134-125 kubenswrapper[4405]: W1104 02:32:33.626461 4405 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 04 02:32:33.646908 ip-10-0-134-125 kubenswrapper[4405]: W1104 02:32:33.626473 4405 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 04 02:32:33.646908 ip-10-0-134-125 kubenswrapper[4405]: W1104 02:32:33.626478 4405 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 04 02:32:33.646908 ip-10-0-134-125 kubenswrapper[4405]: W1104 02:32:33.626482 4405 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 04 02:32:33.646908 ip-10-0-134-125 kubenswrapper[4405]: W1104 02:32:33.626486 4405 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 04 02:32:33.646908 ip-10-0-134-125 kubenswrapper[4405]: W1104 02:32:33.626490 4405 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 04 02:32:33.646908 ip-10-0-134-125 kubenswrapper[4405]: W1104 02:32:33.626494 4405 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 04 02:32:33.646908 ip-10-0-134-125 kubenswrapper[4405]: W1104 02:32:33.626498 4405 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 04 02:32:33.646908 ip-10-0-134-125 kubenswrapper[4405]: W1104 02:32:33.626503 4405 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 04 02:32:33.646908 ip-10-0-134-125 kubenswrapper[4405]: W1104 02:32:33.626507 4405 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 04 02:32:33.646908 ip-10-0-134-125 kubenswrapper[4405]: W1104 02:32:33.626512 4405 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 04 02:32:33.646908 ip-10-0-134-125 kubenswrapper[4405]: W1104 02:32:33.626516 4405 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 04 02:32:33.646908 ip-10-0-134-125 kubenswrapper[4405]: W1104 02:32:33.626522 4405 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 04 02:32:33.646908 ip-10-0-134-125 kubenswrapper[4405]: W1104 02:32:33.626526 4405 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 04 02:32:33.647873 ip-10-0-134-125 kubenswrapper[4405]: W1104 02:32:33.626531 4405 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 04 02:32:33.647873 ip-10-0-134-125 kubenswrapper[4405]: W1104 02:32:33.626535 4405 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 04 02:32:33.647873 ip-10-0-134-125 kubenswrapper[4405]: W1104 02:32:33.626539 4405 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 04 02:32:33.647873 ip-10-0-134-125 kubenswrapper[4405]: W1104 02:32:33.626543 4405 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 04 02:32:33.647873 ip-10-0-134-125 kubenswrapper[4405]: W1104 02:32:33.626548 4405 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 04 02:32:33.647873 ip-10-0-134-125 kubenswrapper[4405]: W1104 02:32:33.626552 4405 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 04 02:32:33.647873 ip-10-0-134-125 kubenswrapper[4405]: W1104 02:32:33.626556 4405 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 04 02:32:33.647873 ip-10-0-134-125 kubenswrapper[4405]: W1104 02:32:33.626560 4405 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 04 02:32:33.647873 ip-10-0-134-125 kubenswrapper[4405]: W1104 02:32:33.626565 4405 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 04 02:32:33.647873 ip-10-0-134-125 kubenswrapper[4405]: W1104 02:32:33.626569 4405 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 04 02:32:33.647873 ip-10-0-134-125 kubenswrapper[4405]: W1104 02:32:33.626573 4405 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 04 02:32:33.647873 ip-10-0-134-125 kubenswrapper[4405]: W1104 02:32:33.626578 4405 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 04 02:32:33.647873 ip-10-0-134-125 kubenswrapper[4405]: W1104 02:32:33.626582 4405 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 04 02:32:33.647873 ip-10-0-134-125 kubenswrapper[4405]: W1104 02:32:33.626586 4405 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 04 02:32:33.647873 ip-10-0-134-125 kubenswrapper[4405]: W1104 02:32:33.626591 4405 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 04 02:32:33.647873 ip-10-0-134-125 kubenswrapper[4405]: W1104 02:32:33.626595 4405 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 04 02:32:33.647873 ip-10-0-134-125 kubenswrapper[4405]: W1104 02:32:33.626601 4405 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 04 02:32:33.647873 ip-10-0-134-125 kubenswrapper[4405]: W1104 02:32:33.626606 4405 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 04 02:32:33.647873 ip-10-0-134-125 kubenswrapper[4405]: W1104 02:32:33.626610 4405 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 04 02:32:33.647873 ip-10-0-134-125 kubenswrapper[4405]: W1104 02:32:33.626614 4405 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 04 02:32:33.648611 ip-10-0-134-125 kubenswrapper[4405]: W1104 02:32:33.626634 4405 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 04 02:32:33.648611 ip-10-0-134-125 kubenswrapper[4405]: W1104 02:32:33.626639 4405 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 04 02:32:33.648611 ip-10-0-134-125 kubenswrapper[4405]: W1104 02:32:33.626644 4405 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 04 02:32:33.648611 ip-10-0-134-125 kubenswrapper[4405]: W1104 02:32:33.626648 4405 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 04 02:32:33.648611 ip-10-0-134-125 kubenswrapper[4405]: W1104 02:32:33.626653 4405 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 04 02:32:33.648611 ip-10-0-134-125 kubenswrapper[4405]: W1104 02:32:33.626657 4405 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 04 02:32:33.648611 ip-10-0-134-125 kubenswrapper[4405]: W1104 02:32:33.626661 4405 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 04 02:32:33.648611 ip-10-0-134-125 kubenswrapper[4405]: W1104 02:32:33.626684 4405 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 04 02:32:33.648611 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:32:33.626763 4405 server.go:957] "Client rotation is on, will bootstrap in background" Nov 04 02:32:33.648611 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:32:33.628370 4405 bootstrap.go:85] "Current kubeconfig file contents are still valid, no bootstrap necessary" Nov 04 02:32:33.648611 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:32:33.628452 4405 certificate_store.go:147] "Loading cert/key pair from a file" filePath="/var/lib/kubelet/pki/kubelet-client-current.pem" Nov 04 02:32:33.648611 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:32:33.628784 4405 server.go:1014] "Starting client certificate rotation" Nov 04 02:32:33.648611 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:32:33.629317 4405 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Nov 04 02:32:33.648611 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:32:33.629374 4405 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-11-03 02:26:54 +0000 UTC" deadline="2028-06-02 10:02:55.261490243 +0000 UTC" Nov 04 02:32:33.648611 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:32:33.629409 4405 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="13831h30m21.632083231s" Nov 04 02:32:33.648611 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:32:33.629928 4405 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Nov 04 02:32:33.649059 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:32:33.631689 4405 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Nov 04 02:32:33.649059 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:32:33.633128 4405 log.go:25] "Validated CRI v1 runtime API" Nov 04 02:32:33.649059 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:32:33.634945 4405 log.go:25] "Validated CRI v1 image API" Nov 04 02:32:33.649059 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:32:33.635712 4405 server.go:1424] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Nov 04 02:32:33.649059 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:32:33.637153 4405 fs.go:135] Filesystem UUIDs: map[19404a3c-de09-4630-be9f-0cbcbcb7c656:/dev/nvme0n1p4 2a6e893d-37f1-444b-9f75-98f43a4d48bf:/dev/nvme0n1p3 7B77-95E7:/dev/nvme0n1p2] Nov 04 02:32:33.649222 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:32:33.637174 4405 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:24 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:26 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/0a7d3eed97ff0f3d2ca2a3dee83c822cb30b9b1d44f646553cd7922c3a9541b9/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/0a7d3eed97ff0f3d2ca2a3dee83c822cb30b9b1d44f646553cd7922c3a9541b9/userdata/shm major:0 minor:84 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/40c8e62d5261acd2a0d99afde27dd20c1b7bcb27efa2c4e4823b219a202588d6/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/40c8e62d5261acd2a0d99afde27dd20c1b7bcb27efa2c4e4823b219a202588d6/userdata/shm major:0 minor:63 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/469d4eddcf15bea0eb81f4a450042830e9b4e23d956d0f061ab35341b066dcd5/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/469d4eddcf15bea0eb81f4a450042830e9b4e23d956d0f061ab35341b066dcd5/userdata/shm major:0 minor:210 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/47b9f0efad770fcb435f4488c99815037d5feb96967427da3c63d396c4ec20da/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/47b9f0efad770fcb435f4488c99815037d5feb96967427da3c63d396c4ec20da/userdata/shm major:0 minor:64 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/616dfef053572a7bd0df5fb2615000eae7a447ca31cea0519cec9e737621ef3e/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/616dfef053572a7bd0df5fb2615000eae7a447ca31cea0519cec9e737621ef3e/userdata/shm major:0 minor:92 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/687e7943a5e73801885e494b71552b0607afdb68a04c24bb344b9c0621fe2080/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/687e7943a5e73801885e494b71552b0607afdb68a04c24bb344b9c0621fe2080/userdata/shm major:0 minor:62 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/6d8c7e634cc46f6ff9bcbd86093ccd32f6171bc08ffac03df9cf46af38e95fca/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/6d8c7e634cc46f6ff9bcbd86093ccd32f6171bc08ffac03df9cf46af38e95fca/userdata/shm major:0 minor:88 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/7f29a771e6ee4b7e85aa9aa1134769a08873c5be8f3c263500189b2051fd67b8/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/7f29a771e6ee4b7e85aa9aa1134769a08873c5be8f3c263500189b2051fd67b8/userdata/shm major:0 minor:224 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/8c8e7de721ec2b40adbe23f38728b8f083e9ec26f592a50f20039593ff57d23b/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/8c8e7de721ec2b40adbe23f38728b8f083e9ec26f592a50f20039593ff57d23b/userdata/shm major:0 minor:202 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/bb86d16e06bc56cd7e46a4be8f1338e7c72edac6c81b7c132d32bd3d4feb5415/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/bb86d16e06bc56cd7e46a4be8f1338e7c72edac6c81b7c132d32bd3d4feb5415/userdata/shm major:0 minor:98 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/e2d730f5206d875c1b306c4525077cc3151c5f42596a1e751c4a0de261d0d629/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/e2d730f5206d875c1b306c4525077cc3151c5f42596a1e751c4a0de261d0d629/userdata/shm major:0 minor:60 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/e3f6ee0a8c70b4d6cb30ed7c29e4392e9078922367521917c71e40e38a5db2c3/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/e3f6ee0a8c70b4d6cb30ed7c29e4392e9078922367521917c71e40e38a5db2c3/userdata/shm major:0 minor:61 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/eb581f44938bcc48c524533eb00406852e02b243984608e724360926c929b6c6/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/eb581f44938bcc48c524533eb00406852e02b243984608e724360926c929b6c6/userdata/shm major:0 minor:207 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/f8bda2cfe1f2e4c6f7d909b602917ba1869ed772ade09d26192081e4ec879d5c/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/f8bda2cfe1f2e4c6f7d909b602917ba1869ed772ade09d26192081e4ec879d5c/userdata/shm major:0 minor:80 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:32 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/3984ce7b-96c8-464a-aa14-28487cc5b373/volumes/kubernetes.io~projected/kube-api-access-plfvd:{mountpoint:/var/lib/kubelet/pods/3984ce7b-96c8-464a-aa14-28487cc5b373/volumes/kubernetes.io~projected/kube-api-access-plfvd major:0 minor:57 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/3a7ab1db-ad88-40d9-a7e9-b511bbd4584c/volumes/kubernetes.io~secret/original-pull-secret:{mountpoint:/var/lib/kubelet/pods/3a7ab1db-ad88-40d9-a7e9-b511bbd4584c/volumes/kubernetes.io~secret/original-pull-secret major:0 minor:206 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/4acab1fc-c591-475a-a15f-790be8131547/volumes/kubernetes.io~projected/kube-api-access-bkf4l:{mountpoint:/var/lib/kubelet/pods/4acab1fc-c591-475a-a15f-790be8131547/volumes/kubernetes.io~projected/kube-api-access-bkf4l major:0 minor:218 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/4f76b032-61eb-4796-b082-47c23ff73d18/volumes/kubernetes.io~projected/kube-api-access-8lqpv:{mountpoint:/var/lib/kubelet/pods/4f76b032-61eb-4796-b082-47c23ff73d18/volumes/kubernetes.io~projected/kube-api-access-8lqpv major:0 minor:197 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/524e33f8-723b-4868-a4f5-5d7eb92a5fe3/volumes/kubernetes.io~projected/kube-api-access-stzcf:{mountpoint:/var/lib/kubelet/pods/524e33f8-723b-4868-a4f5-5d7eb92a5fe3/volumes/kubernetes.io~projected/kube-api-access-stzcf major:0 minor:58 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/5590efb1-88b8-41ad-b0ee-37b2ff9442de/volumes/kubernetes.io~projected/kube-api-access-4rpxv:{mountpoint:/var/lib/kubelet/pods/5590efb1-88b8-41ad-b0ee-37b2ff9442de/volumes/kubernetes.io~projected/kube-api-access-4rpxv major:0 minor:56 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/5a5c4b25-7a35-4290-ad1e-04b4a3617877/volumes/kubernetes.io~projected/kube-api-access-h288h:{mountpoint:/var/lib/kubelet/pods/5a5c4b25-7a35-4290-ad1e-04b4a3617877/volumes/kubernetes.io~projected/kube-api-access-h288h major:0 minor:223 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/5b77e30b-cb0c-49c1-80b9-330a138dbfb7/volumes/kubernetes.io~secret/agent-certs:{mountpoint:/var/lib/kubelet/pods/5b77e30b-cb0c-49c1-80b9-330a138dbfb7/volumes/kubernetes.io~secret/agent-certs major:0 minor:49 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/6e27a76f-fbdb-4ce6-9df6-659cab9ad06c/volumes/kubernetes.io~projected/kube-api-access-dddmg:{mountpoint:/var/lib/kubelet/pods/6e27a76f-fbdb-4ce6-9df6-659cab9ad06c/volumes/kubernetes.io~projected/kube-api-access-dddmg major:0 minor:208 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/7cb29949-bfe4-47a2-bc17-115a8d4d895d/volumes/kubernetes.io~projected/kube-api-access-946v4:{mountpoint:/var/lib/kubelet/pods/7cb29949-bfe4-47a2-bc17-115a8d4d895d/volumes/kubernetes.io~projected/kube-api-access-946v4 major:0 minor:53 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/8b030f29-dfa6-42b1-81a7-07b539ad9aa9/volumes/kubernetes.io~projected/kube-api-access-8ll77:{mountpoint:/var/lib/kubelet/pods/8b030f29-dfa6-42b1-81a7-07b539ad9aa9/volumes/kubernetes.io~projected/kube-api-access-8ll77 major:0 minor:59 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/a245d359-742e-489d-9c58-eea4e686286c/volumes/kubernetes.io~projected/kube-api-access-2rpzn:{mountpoint:/var/lib/kubelet/pods/a245d359-742e-489d-9c58-eea4e686286c/volumes/kubernetes.io~projected/kube-api-access-2rpzn major:0 minor:201 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/a245d359-742e-489d-9c58-eea4e686286c/volumes/kubernetes.io~secret/cert:{mountpoint:/var/lib/kubelet/pods/a245d359-742e-489d-9c58-eea4e686286c/volumes/kubernetes.io~secret/cert major:0 minor:45 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/b37e5c97-77c6-4bb4-a9e4-e1ae27484457/volumes/kubernetes.io~projected/kube-api-access-zpbsm:{mountpoint:/var/lib/kubelet/pods/b37e5c97-77c6-4bb4-a9e4-e1ae27484457/volumes/kubernetes.io~projected/kube-api-access-zpbsm major:0 minor:52 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/cb1915a0-5505-4f32-8f79-2f638465e404/volumes/kubernetes.io~empty-dir/etc-tuned:{mountpoint:/var/lib/kubelet/pods/cb1915a0-5505-4f32-8f79-2f638465e404/volumes/kubernetes.io~empty-dir/etc-tuned major:0 minor:47 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/cb1915a0-5505-4f32-8f79-2f638465e404/volumes/kubernetes.io~empty-dir/tmp:{mountpoint:/var/lib/kubelet/pods/cb1915a0-5505-4f32-8f79-2f638465e404/volumes/kubernetes.io~empty-dir/tmp major:0 minor:50 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/cb1915a0-5505-4f32-8f79-2f638465e404/volumes/kubernetes.io~projected/kube-api-access-n6mc7:{mountpoint:/var/lib/kubelet/pods/cb1915a0-5505-4f32-8f79-2f638465e404/volumes/kubernetes.io~projected/kube-api-access-n6mc7 major:0 minor:51 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/f58e4ac8-c132-4716-b6c0-8d8ddb9c19e5/volumes/kubernetes.io~projected/kube-api-access-b4snb:{mountpoint:/var/lib/kubelet/pods/f58e4ac8-c132-4716-b6c0-8d8ddb9c19e5/volumes/kubernetes.io~projected/kube-api-access-b4snb major:0 minor:54 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/ffbb2d83-5a93-4a5b-bce3-e40afb8f58ea/volume-subpaths/run-systemd/ovnkube-controller/6:{mountpoint:/var/lib/kubelet/pods/ffbb2d83-5a93-4a5b-bce3-e40afb8f58ea/volume-subpaths/run-systemd/ovnkube-controller/6 major:0 minor:26 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/ffbb2d83-5a93-4a5b-bce3-e40afb8f58ea/volumes/kubernetes.io~projected/kube-api-access-tp4jl:{mountpoint:/var/lib/kubelet/pods/ffbb2d83-5a93-4a5b-bce3-e40afb8f58ea/volumes/kubernetes.io~projected/kube-api-access-tp4jl major:0 minor:55 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/ffbb2d83-5a93-4a5b-bce3-e40afb8f58ea/volumes/kubernetes.io~secret/ovn-node-metrics-cert:{mountpoint:/var/lib/kubelet/pods/ffbb2d83-5a93-4a5b-bce3-e40afb8f58ea/volumes/kubernetes.io~secret/ovn-node-metrics-cert major:0 minor:48 fsType:tmpfs blockSize:0} composefs_0-34:{mountpoint:/ major:0 minor:34 fsType:overlay blockSize:0} overlay_0-100:{mountpoint:/var/lib/containers/storage/overlay/09bfc663ef82a2c5e895486e3c6a5a00939da10b2f325ff96d52ea391ec9b51e/merged major:0 minor:100 fsType:overlay blockSize:0} overlay_0-102:{mountpoint:/var/lib/containers/storage/overlay/5ec2f874cb223689df107db494b5a50db26b5f68b2fa0a790250ab3012516dab/merged major:0 minor:102 fsType:overlay blockSize:0} overlay_0-104:{mountpoint:/var/lib/containers/storage/overlay/ce40f5bba090d16e74343d2c1b549e3d5560d706dd0cceebf66dac2936fed517/merged major:0 minor:104 fsType:overlay blockSize:0} overlay_0-109:{mountpoint:/var/lib/containers/storage/overlay/51cb0dfddfec583eb229dc3bbae3e63bf2b954dfde173dad2f4654a291b84d4b/merged major:0 minor:109 fsType:overlay blockSize:0} overlay_0-114:{mountpoint:/var/lib/containers/storage/overlay/3cdb2a2b7c2a11855a82132dad68165139d9702c5f95e402839fe9eb8f5d2dc0/merged major:0 minor:114 fsType:overlay blockSize:0} overlay_0-119:{mountpoint:/var/lib/containers/storage/overlay/b0f5a0285beca268b30b2e2165143874a3b53a8c4b5dbc2be4d2fc9e01c9ef8d/merged major:0 minor:119 fsType:overlay blockSize:0} overlay_0-124:{mountpoint:/var/lib/containers/storage/overlay/90faa887fa76bb002253b853be4d3fd9c1a91f232e1987cef29c0a1eaaf26c0b/merged major:0 minor:124 fsType:overlay blockSize:0} overlay_0-127:{mountpoint:/var/lib/containers/storage/overlay/03dcd2e184a6361066d84a9460f9da14518ce4ddbd55b8803624f35f3dea345d/merged major:0 minor:127 fsType:overlay blockSize:0} overlay_0-129:{mountpoint:/var/lib/containers/storage/overlay/b79c7cc2260ab3eb89536e90460a317760bb7b74ba0ea19c76d62345332c736d/merged major:0 minor:129 fsType:overlay blockSize:0} overlay_0-134:{mountpoint:/var/lib/containers/storage/overlay/4d2941761387cfb6a448dc40e6d93b4981532f5474f03b3952277c843e1454ba/merged major:0 minor:134 fsType:overlay blockSize:0} overlay_0-140:{mountpoint:/var/lib/containers/storage/overlay/e72074d2f31a50ca98ea4c8e42106ed324b1f7d96643d0de14422b5b8a4cc20e/merged major:0 minor:140 fsType:overlay blockSize:0} overlay_0-144:{mountpoint:/var/lib/containers/storage/overlay/f910d03dab4506c7d014a88ad4445f9c85433792d82932b523969db73ccfd749/merged major:0 minor:144 fsType:overlay blockSize:0} overlay_0-152:{mountpoint:/var/lib/containers/storage/overlay/3454754e274c3819495e35beb469d5965c5ddea2ac94e0e05cd5f66dbf36bb52/merged major:0 minor:152 fsType:overlay blockSize:0} overlay_0-157:{mountpoint:/var/lib/containers/storage/overlay/93267517238f1fb59cd00dd72cca443c666cf47cde1dd1bd7d0454a1d01fb9b5/merged major:0 minor:157 fsType:overlay blockSize:0} overlay_0-162:{mountpoint:/var/lib/containers/storage/overlay/5a6ac0bf91b58beee5d2374015fb67f693c0d997a131fc3cec5db2d68372c598/merged major:0 minor:162 fsType:overlay blockSize:0} overlay_0-167:{mountpoint:/var/lib/containers/storage/overlay/5432d82e1888f124225a053da5c7b95072b1f18963a0948be7081d8de9375d79/merged major:0 minor:167 fsType:overlay blockSize:0} overlay_0-172:{mountpoint:/var/lib/containers/storage/overlay/7b9fb74b938ef1e5669667bd081714002c55bfa667841783b9791505c4644be3/merged major:0 minor:172 fsType:overlay blockSize:0} overlay_0-177:{mountpoint:/var/lib/containers/storage/overlay/9bd2ce0c9f622a793164753e86c1fa815d24d1e0bd1ffa07fc2c8f26e88dfe1b/merged major:0 minor:177 fsType:overlay blockSize:0} overlay_0-182:{mountpoint:/var/lib/containers/storage/overlay/37df1d2d175e8f1739f451e3c3505fca163b070b6bc290e262354b1d8c77f5f7/merged major:0 minor:182 fsType:overlay blockSize:0} overlay_0-187:{mountpoint:/var/lib/containers/storage/overlay/52533ce3c9fb4aa109a8b3397e93443633e0a8cb2d0eead21d3f67eb4c9c3352/merged major:0 minor:187 fsType:overlay blockSize:0} overlay_0-192:{mountpoint:/var/lib/containers/storage/overlay/485cf8a36c08eaaf6e9dde786e94b5dc653b687c7532415b0b13d737f59fbf13/merged major:0 minor:192 fsType:overlay blockSize:0} overlay_0-204:{mountpoint:/var/lib/containers/storage/overlay/1492ac962a8fe63f6dd5ef024ea508a5c7375ddc56f34d61bb8908a5fc1ed28c/merged major:0 minor:204 fsType:overlay blockSize:0} overlay_0-212:{mountpoint:/var/lib/containers/storage/overlay/bbc07259f62bdbee066426cdfd2613411f5e4692381526949ee11368366ed5bb/merged major:0 minor:212 fsType:overlay blockSize:0} overlay_0-214:{mountpoint:/var/lib/containers/storage/overlay/e58805b29201332e1fe5c437c5410ab5ab5df7049225b25839b5ebee9fc114c9/merged major:0 minor:214 fsType:overlay blockSize:0} overlay_0-216:{mountpoint:/var/lib/containers/storage/overlay/261c668ee4cc3e88b9e930873ffdb4eaab6eadbcefefcb56ccbbf198ab3c55db/merged major:0 minor:216 fsType:overlay blockSize:0} overlay_0-226:{mountpoint:/var/lib/containers/storage/overlay/41602d5f6902a224615da3f5b2090b61bba266f6972d293fca3dcaa233e441ce/merged major:0 minor:226 fsType:overlay blockSize:0} overlay_0-228:{mountpoint:/var/lib/containers/storage/overlay/f1ab5ee1d067471c3198e8d0fa79af8bd4fc7256a897eb7aff79e6a7d30b648e/merged major:0 minor:228 fsType:overlay blockSize:0} overlay_0-230:{mountpoint:/var/lib/containers/storage/overlay/47c982eb3964576d09fb79486d48b9a7db795ae2d97672e98ad2eb497876e1e8/merged major:0 minor:230 fsType:overlay blockSize:0} overlay_0-70:{mountpoint:/var/lib/containers/storage/overlay/70941fa8c65204a6c4df3fe028cbfdb798a2e56b7edbf175b2e466cc1036f71a/merged major:0 minor:70 fsType:overlay blockSize:0} overlay_0-72:{mountpoint:/var/lib/containers/storage/overlay/65855d109e8abe78479e9c1aa111dd3c4a76ea896679e048723d5955a3d1c3e8/merged major:0 minor:72 fsType:overlay blockSize:0} overlay_0-74:{mountpoint:/var/lib/containers/storage/overlay/13d4dc2f99802481cc7fdd658edf882da9e245bf655cc818a16c5d9b63201b44/merged major:0 minor:74 fsType:overlay blockSize:0} overlay_0-76:{mountpoint:/var/lib/containers/storage/overlay/b84daf6aa413962dfae4210aec8b7482aaa8c94e18bf1837cad080f0f4cb15e3/merged major:0 minor:76 fsType:overlay blockSize:0} overlay_0-78:{mountpoint:/var/lib/containers/storage/overlay/5ec050cbe2844c75dfaff62092a064ceceb9b4a5a307659a9e30055f8cf5b953/merged major:0 minor:78 fsType:overlay blockSize:0} overlay_0-82:{mountpoint:/var/lib/containers/storage/overlay/4a7f0d6d55a4a8395e91e769f80ad5af7ad467c7332b3fd478c28a0362f64a9c/merged major:0 minor:82 fsType:overlay blockSize:0} overlay_0-86:{mountpoint:/var/lib/containers/storage/overlay/567eb2ae47f7e601798071f88d956c63d21ee04428b9885d54d2b7f2c92125a7/merged major:0 minor:86 fsType:overlay blockSize:0} overlay_0-90:{mountpoint:/var/lib/containers/storage/overlay/b059d4c7c12facc105328a930ad0ba31eb336cd2ecb22bd1e4ec2642273313d6/merged major:0 minor:90 fsType:overlay blockSize:0} overlay_0-94:{mountpoint:/var/lib/containers/storage/overlay/1b9fbdb2e871f5d38b66c2ee1321dcd14f643d6f07d024a34683c43be2b73003/merged major:0 minor:94 fsType:overlay blockSize:0} overlay_0-96:{mountpoint:/var/lib/containers/storage/overlay/d3f01abbf28e81927b05611d63970420b03020847fe798babac62aa69504d302/merged major:0 minor:96 fsType:overlay blockSize:0}] Nov 04 02:32:33.649391 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:32:33.643613 4405 manager.go:217] Machine: {Timestamp:2026-11-04 02:32:33.643129058 +0000 UTC m=+0.073829168 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3099766 MemoryCapacity:32812011520 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2833fcb94c4b503126b825579d6ddb SystemUUID:ec2833fc-b94c-4b50-3126-b825579d6ddb BootID:2cd229f4-4cf0-4425-8796-f243d5685184 Filesystems:[{Device:/var/lib/kubelet/pods/5a5c4b25-7a35-4290-ad1e-04b4a3617877/volumes/kubernetes.io~projected/kube-api-access-h288h DeviceMajor:0 DeviceMinor:223 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-230 DeviceMajor:0 DeviceMinor:230 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:26 Capacity:6562402304 Type:vfs Inodes:819200 HasInodes:true} {Device:/var/lib/kubelet/pods/cb1915a0-5505-4f32-8f79-2f638465e404/volumes/kubernetes.io~projected/kube-api-access-n6mc7 DeviceMajor:0 DeviceMinor:51 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:/var/lib/kubelet/pods/5590efb1-88b8-41ad-b0ee-37b2ff9442de/volumes/kubernetes.io~projected/kube-api-access-4rpxv DeviceMajor:0 DeviceMinor:56 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:/run/containers/storage/overlay-containers/616dfef053572a7bd0df5fb2615000eae7a447ca31cea0519cec9e737621ef3e/userdata/shm DeviceMajor:0 DeviceMinor:92 Capacity:67108864 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-114 DeviceMajor:0 DeviceMinor:114 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-134 DeviceMajor:0 DeviceMinor:134 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/4acab1fc-c591-475a-a15f-790be8131547/volumes/kubernetes.io~projected/kube-api-access-bkf4l DeviceMajor:0 DeviceMinor:218 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:/run/containers/storage/overlay-containers/40c8e62d5261acd2a0d99afde27dd20c1b7bcb27efa2c4e4823b219a202588d6/userdata/shm DeviceMajor:0 DeviceMinor:63 Capacity:67108864 Type:vfs Inodes:4005372 HasInodes:true} {Device:/run/containers/storage/overlay-containers/47b9f0efad770fcb435f4488c99815037d5feb96967427da3c63d396c4ec20da/userdata/shm DeviceMajor:0 DeviceMinor:64 Capacity:67108864 Type:vfs Inodes:4005372 HasInodes:true} {Device:/run/containers/storage/overlay-containers/0a7d3eed97ff0f3d2ca2a3dee83c822cb30b9b1d44f646553cd7922c3a9541b9/userdata/shm DeviceMajor:0 DeviceMinor:84 Capacity:67108864 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-96 DeviceMajor:0 DeviceMinor:96 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-177 DeviceMajor:0 DeviceMinor:177 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-187 DeviceMajor:0 DeviceMinor:187 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/5b77e30b-cb0c-49c1-80b9-330a138dbfb7/volumes/kubernetes.io~secret/agent-certs DeviceMajor:0 DeviceMinor:49 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:/var/lib/kubelet/pods/f58e4ac8-c132-4716-b6c0-8d8ddb9c19e5/volumes/kubernetes.io~projected/kube-api-access-b4snb DeviceMajor:0 DeviceMinor:54 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:/run/containers/storage/overlay-containers/e3f6ee0a8c70b4d6cb30ed7c29e4392e9078922367521917c71e40e38a5db2c3/userdata/shm DeviceMajor:0 DeviceMinor:61 Capacity:67108864 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-70 DeviceMajor:0 DeviceMinor:70 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-100 DeviceMajor:0 DeviceMinor:100 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/a245d359-742e-489d-9c58-eea4e686286c/volumes/kubernetes.io~secret/cert DeviceMajor:0 DeviceMinor:45 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:/run/containers/storage/overlay-containers/e2d730f5206d875c1b306c4525077cc3151c5f42596a1e751c4a0de261d0d629/userdata/shm DeviceMajor:0 DeviceMinor:60 Capacity:67108864 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-74 DeviceMajor:0 DeviceMinor:74 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-129 DeviceMajor:0 DeviceMinor:129 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-157 DeviceMajor:0 DeviceMinor:157 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-86 DeviceMajor:0 DeviceMinor:86 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-140 DeviceMajor:0 DeviceMinor:140 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/cb1915a0-5505-4f32-8f79-2f638465e404/volumes/kubernetes.io~empty-dir/etc-tuned DeviceMajor:0 DeviceMinor:47 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-90 DeviceMajor:0 DeviceMinor:90 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-104 DeviceMajor:0 DeviceMinor:104 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-124 DeviceMajor:0 DeviceMinor:124 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-167 DeviceMajor:0 DeviceMinor:167 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/6d8c7e634cc46f6ff9bcbd86093ccd32f6171bc08ffac03df9cf46af38e95fca/userdata/shm DeviceMajor:0 DeviceMinor:88 Capacity:67108864 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-127 DeviceMajor:0 DeviceMinor:127 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-152 DeviceMajor:0 DeviceMinor:152 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-192 DeviceMajor:0 DeviceMinor:192 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/var/lib/kubelet/pods/3984ce7b-96c8-464a-aa14-28487cc5b373/volumes/kubernetes.io~projected/kube-api-access-plfvd DeviceMajor:0 DeviceMinor:57 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-182 DeviceMajor:0 DeviceMinor:182 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-212 DeviceMajor:0 DeviceMinor:212 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/cb1915a0-5505-4f32-8f79-2f638465e404/volumes/kubernetes.io~empty-dir/tmp DeviceMajor:0 DeviceMinor:50 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:/var/lib/kubelet/pods/524e33f8-723b-4868-a4f5-5d7eb92a5fe3/volumes/kubernetes.io~projected/kube-api-access-stzcf DeviceMajor:0 DeviceMinor:58 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:/run/containers/storage/overlay-containers/687e7943a5e73801885e494b71552b0607afdb68a04c24bb344b9c0621fe2080/userdata/shm DeviceMajor:0 DeviceMinor:62 Capacity:67108864 Type:vfs Inodes:4005372 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:32 Capacity:16406007808 Type:vfs Inodes:1048576 HasInodes:true} {Device:/var/lib/kubelet/pods/ffbb2d83-5a93-4a5b-bce3-e40afb8f58ea/volumes/kubernetes.io~secret/ovn-node-metrics-cert DeviceMajor:0 DeviceMinor:48 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:/var/lib/kubelet/pods/b37e5c97-77c6-4bb4-a9e4-e1ae27484457/volumes/kubernetes.io~projected/kube-api-access-zpbsm DeviceMajor:0 DeviceMinor:52 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-82 DeviceMajor:0 DeviceMinor:82 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-94 DeviceMajor:0 DeviceMinor:94 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-102 DeviceMajor:0 DeviceMinor:102 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-162 DeviceMajor:0 DeviceMinor:162 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/7f29a771e6ee4b7e85aa9aa1134769a08873c5be8f3c263500189b2051fd67b8/userdata/shm DeviceMajor:0 DeviceMinor:224 Capacity:67108864 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-228 DeviceMajor:0 DeviceMinor:228 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-34 DeviceMajor:0 DeviceMinor:34 Capacity:6422528 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/var/lib/kubelet/pods/8b030f29-dfa6-42b1-81a7-07b539ad9aa9/volumes/kubernetes.io~projected/kube-api-access-8ll77 DeviceMajor:0 DeviceMinor:59 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-72 DeviceMajor:0 DeviceMinor:72 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/f8bda2cfe1f2e4c6f7d909b602917ba1869ed772ade09d26192081e4ec879d5c/userdata/shm DeviceMajor:0 DeviceMinor:80 Capacity:67108864 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-144 DeviceMajor:0 DeviceMinor:144 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/ffbb2d83-5a93-4a5b-bce3-e40afb8f58ea/volume-subpaths/run-systemd/ovnkube-controller/6 DeviceMajor:0 DeviceMinor:26 Capacity:6562402304 Type:vfs Inodes:819200 HasInodes:true} {Device:/run/containers/storage/overlay-containers/8c8e7de721ec2b40adbe23f38728b8f083e9ec26f592a50f20039593ff57d23b/userdata/shm DeviceMajor:0 DeviceMinor:202 Capacity:67108864 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-204 DeviceMajor:0 DeviceMinor:204 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:24 Capacity:16406003712 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-76 DeviceMajor:0 DeviceMinor:76 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/bb86d16e06bc56cd7e46a4be8f1338e7c72edac6c81b7c132d32bd3d4feb5415/userdata/shm DeviceMajor:0 DeviceMinor:98 Capacity:67108864 Type:vfs Inodes:4005372 HasInodes:true} {Device:/var/lib/kubelet/pods/6e27a76f-fbdb-4ce6-9df6-659cab9ad06c/volumes/kubernetes.io~projected/kube-api-access-dddmg DeviceMajor:0 DeviceMinor:208 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:/run/containers/storage/overlay-containers/469d4eddcf15bea0eb81f4a450042830e9b4e23d956d0f061ab35341b066dcd5/userdata/shm DeviceMajor:0 DeviceMinor:210 Capacity:67108864 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-214 DeviceMajor:0 DeviceMinor:214 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-216 DeviceMajor:0 DeviceMinor:216 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/ffbb2d83-5a93-4a5b-bce3-e40afb8f58ea/volumes/kubernetes.io~projected/kube-api-access-tp4jl DeviceMajor:0 DeviceMinor:55 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-109 DeviceMajor:0 DeviceMinor:109 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-172 DeviceMajor:0 DeviceMinor:172 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/4f76b032-61eb-4796-b082-47c23ff73d18/volumes/kubernetes.io~projected/kube-api-access-8lqpv DeviceMajor:0 DeviceMinor:197 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:/run/containers/storage/overlay-containers/eb581f44938bcc48c524533eb00406852e02b243984608e724360926c929b6c6/userdata/shm DeviceMajor:0 DeviceMinor:207 Capacity:67108864 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-226 DeviceMajor:0 DeviceMinor:226 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-78 DeviceMajor:0 DeviceMinor:78 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-119 DeviceMajor:0 DeviceMinor:119 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/3a7ab1db-ad88-40d9-a7e9-b511bbd4584c/volumes/kubernetes.io~secret/original-pull-secret DeviceMajor:0 DeviceMinor:206 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:/var/lib/kubelet/pods/7cb29949-bfe4-47a2-bc17-115a8d4d895d/volumes/kubernetes.io~projected/kube-api-access-946v4 DeviceMajor:0 DeviceMinor:53 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:/var/lib/kubelet/pods/a245d359-742e-489d-9c58-eea4e686286c/volumes/kubernetes.io~projected/kube-api-access-2rpzn DeviceMajor:0 DeviceMinor:201 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:469d4eddcf15bea MacAddress:26:1b:b1:8b:52:0f Speed:10000 Mtu:8901} {Name:7f29a771e6ee4b7 MacAddress:7a:df:19:4d:21:d1 Speed:10000 Mtu:8901} {Name:8c8e7de721ec2b4 MacAddress:ea:95:93:26:6a:29 Speed:10000 Mtu:8901} {Name:br-ex MacAddress:02:2c:40:68:17:55 Speed:0 Mtu:9001} {Name:br-int MacAddress:a2:04:12:90:83:f0 Speed:0 Mtu:8901} {Name:eb581f44938bcc4 MacAddress:ca:23:ed:50:66:fc Speed:10000 Mtu:8901} {Name:ens5 MacAddress:02:2c:40:68:17:55 Speed:0 Mtu:9001} {Name:genev_sys_6081 MacAddress:5a:7e:6b:31:90:1b Speed:0 Mtu:65000} {Name:ovn-k8s-mp0 MacAddress:0a:58:0a:86:00:02 Speed:0 Mtu:8901} {Name:ovs-system MacAddress:c2:d9:78:6b:88:6d Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:32812011520 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:34603008 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Nov 04 02:32:33.649391 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:32:33.643878 4405 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Nov 04 02:32:33.649391 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:32:33.643924 4405 manager.go:233] Version: {KernelVersion:5.14.0-687.60.1.el9_8.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.8.20261023-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Nov 04 02:32:33.649391 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:32:33.644224 4405 container_manager_linux.go:275] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Nov 04 02:32:33.649391 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:32:33.644246 4405 container_manager_linux.go:280] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-134-125.us-west-1.compute.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"/system.slice","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{},"system-reserved-compressible":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Nov 04 02:32:33.649391 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:32:33.644389 4405 topology_manager.go:143] "Creating topology manager with none policy" Nov 04 02:32:33.649391 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:32:33.644397 4405 container_manager_linux.go:311] "Creating device plugin manager" Nov 04 02:32:33.649391 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:32:33.644412 4405 manager.go:144] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Nov 04 02:32:33.649391 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:32:33.644424 4405 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Nov 04 02:32:33.649391 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:32:33.644516 4405 container_manager_linux.go:320] "Creating Dynamic Resource Allocation (DRA) manager" Nov 04 02:32:33.649391 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:32:33.644718 4405 state_mem.go:41] "Initialized" logger="CPUManager state memory" Nov 04 02:32:33.649391 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:32:33.644843 4405 server.go:1235] "Using root directory" path="/var/lib/kubelet" Nov 04 02:32:33.649391 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:32:33.644880 4405 kubelet.go:493] "Attempting to sync node with API server" Nov 04 02:32:33.649391 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:32:33.644896 4405 kubelet.go:394] "Adding static pod path" path="/etc/kubernetes/manifests" Nov 04 02:32:33.649391 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:32:33.644907 4405 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Nov 04 02:32:33.649391 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:32:33.644915 4405 kubelet.go:405] "Adding apiserver pod source" Nov 04 02:32:33.649391 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:32:33.644923 4405 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Nov 04 02:32:33.649391 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:32:33.645862 4405 state_checkpoint.go:81] "State checkpoint: restored pod resource state from checkpoint" Nov 04 02:32:33.649391 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:32:33.645878 4405 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Nov 04 02:32:33.649391 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:32:33.646988 4405 kuberuntime_manager.go:304] "Container runtime initialized" containerRuntime="cri-o" version="1.35.10-3.rhaos4.22.git9383502.el9" apiVersion="v1" Nov 04 02:32:33.654650 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:32:33.654634 4405 state_checkpoint.go:81] "State checkpoint: restored pod resource state from checkpoint" Nov 04 02:32:33.654650 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:32:33.654650 4405 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Nov 04 02:32:33.654782 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:32:33.654771 4405 certificate_store.go:147] "Loading cert/key pair from a file" filePath="/var/lib/kubelet/pki/kubelet-server-current.pem" Nov 04 02:32:33.654985 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:32:33.654974 4405 kubelet.go:963] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Nov 04 02:32:33.655025 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:32:33.655010 4405 kubelet.go:990] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Nov 04 02:32:33.655119 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:32:33.655110 4405 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Nov 04 02:32:33.655158 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:32:33.655125 4405 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Nov 04 02:32:33.655158 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:32:33.655131 4405 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Nov 04 02:32:33.655158 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:32:33.655137 4405 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Nov 04 02:32:33.655158 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:32:33.655144 4405 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Nov 04 02:32:33.655158 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:32:33.655153 4405 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Nov 04 02:32:33.655361 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:32:33.655162 4405 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Nov 04 02:32:33.655361 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:32:33.655168 4405 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Nov 04 02:32:33.655361 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:32:33.655176 4405 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Nov 04 02:32:33.655361 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:32:33.655182 4405 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Nov 04 02:32:33.655361 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:32:33.655190 4405 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Nov 04 02:32:33.655361 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:32:33.655199 4405 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Nov 04 02:32:33.655361 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:32:33.655231 4405 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Nov 04 02:32:33.655361 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:32:33.655240 4405 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Nov 04 02:32:33.658792 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:32:33.658778 4405 server.go:1263] "Started kubelet" Nov 04 02:32:33.658919 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:32:33.658882 4405 server.go:182] "Starting to listen" address="0.0.0.0" port=10250 Nov 04 02:32:33.659371 ip-10-0-134-125 systemd[1]: Started Kubernetes Kubelet. Nov 04 02:32:33.660582 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:32:33.660564 4405 server.go:317] "Adding debug handlers to kubelet server" Nov 04 02:32:33.661700 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:32:33.661657 4405 fs_resource_analyzer.go:69] "Starting FS ResourceAnalyzer" Nov 04 02:32:33.661700 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:32:33.661687 4405 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Nov 04 02:32:33.661911 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:32:33.658901 4405 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Nov 04 02:32:33.661911 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:32:33.661756 4405 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-11-03 02:26:55 +0000 UTC" deadline="2028-07-01 07:37:25.586608302 +0000 UTC" Nov 04 02:32:33.661911 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:32:33.661796 4405 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14525h4m51.924815627s" Nov 04 02:32:33.661911 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:32:33.661806 4405 server_v1.go:49] "podresources" method="list" useActivePods=true Nov 04 02:32:33.662685 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:32:33.662105 4405 volume_manager.go:309] "The desired_state_of_world populator starts" Nov 04 02:32:33.662685 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:32:33.662118 4405 volume_manager.go:311] "Starting Kubelet Volume Manager" Nov 04 02:32:33.662685 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:32:33.662158 4405 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Nov 04 02:32:33.662685 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:32:33.662114 4405 server.go:254] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Nov 04 02:32:33.662685 ip-10-0-134-125 kubenswrapper[4405]: E1104 02:32:33.662155 4405 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-134-125.us-west-1.compute.internal\" not found" Nov 04 02:32:33.664199 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:32:33.664163 4405 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5590efb1-88b8-41ad-b0ee-37b2ff9442de" volumeName="kubernetes.io/configmap/5590efb1-88b8-41ad-b0ee-37b2ff9442de-cni-binary-copy" seLinuxMountContext="" Nov 04 02:32:33.664308 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:32:33.664199 4405 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="524e33f8-723b-4868-a4f5-5d7eb92a5fe3" volumeName="kubernetes.io/configmap/524e33f8-723b-4868-a4f5-5d7eb92a5fe3-iptables-alerter-script" seLinuxMountContext="" Nov 04 02:32:33.664308 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:32:33.664215 4405 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5b77e30b-cb0c-49c1-80b9-330a138dbfb7" volumeName="kubernetes.io/secret/5b77e30b-cb0c-49c1-80b9-330a138dbfb7-agent-certs" seLinuxMountContext="" Nov 04 02:32:33.664308 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:32:33.664242 4405 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7cb29949-bfe4-47a2-bc17-115a8d4d895d" volumeName="kubernetes.io/projected/7cb29949-bfe4-47a2-bc17-115a8d4d895d-kube-api-access-946v4" seLinuxMountContext="" Nov 04 02:32:33.664308 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:32:33.664254 4405 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ffbb2d83-5a93-4a5b-bce3-e40afb8f58ea" volumeName="kubernetes.io/projected/ffbb2d83-5a93-4a5b-bce3-e40afb8f58ea-kube-api-access-tp4jl" seLinuxMountContext="" Nov 04 02:32:33.664308 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:32:33.664266 4405 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3984ce7b-96c8-464a-aa14-28487cc5b373" volumeName="kubernetes.io/configmap/3984ce7b-96c8-464a-aa14-28487cc5b373-multus-daemon-config" seLinuxMountContext="" Nov 04 02:32:33.664308 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:32:33.664283 4405 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3a7ab1db-ad88-40d9-a7e9-b511bbd4584c" volumeName="kubernetes.io/secret/3a7ab1db-ad88-40d9-a7e9-b511bbd4584c-original-pull-secret" seLinuxMountContext="" Nov 04 02:32:33.664308 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:32:33.664294 4405 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5a5c4b25-7a35-4290-ad1e-04b4a3617877" volumeName="kubernetes.io/configmap/5a5c4b25-7a35-4290-ad1e-04b4a3617877-trusted-ca-bundle" seLinuxMountContext="" Nov 04 02:32:33.664308 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:32:33.664307 4405 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8b030f29-dfa6-42b1-81a7-07b539ad9aa9" volumeName="kubernetes.io/projected/8b030f29-dfa6-42b1-81a7-07b539ad9aa9-kube-api-access-8ll77" seLinuxMountContext="" Nov 04 02:32:33.664837 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:32:33.664328 4405 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="cb1915a0-5505-4f32-8f79-2f638465e404" volumeName="kubernetes.io/empty-dir/cb1915a0-5505-4f32-8f79-2f638465e404-tmp" seLinuxMountContext="" Nov 04 02:32:33.664837 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:32:33.664341 4405 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ffbb2d83-5a93-4a5b-bce3-e40afb8f58ea" volumeName="kubernetes.io/configmap/ffbb2d83-5a93-4a5b-bce3-e40afb8f58ea-ovnkube-config" seLinuxMountContext="" Nov 04 02:32:33.664837 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:32:33.664365 4405 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5590efb1-88b8-41ad-b0ee-37b2ff9442de" volumeName="kubernetes.io/configmap/5590efb1-88b8-41ad-b0ee-37b2ff9442de-cni-sysctl-allowlist" seLinuxMountContext="" Nov 04 02:32:33.664837 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:32:33.664382 4405 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3984ce7b-96c8-464a-aa14-28487cc5b373" volumeName="kubernetes.io/configmap/3984ce7b-96c8-464a-aa14-28487cc5b373-cni-binary-copy" seLinuxMountContext="" Nov 04 02:32:33.664837 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:32:33.664393 4405 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3984ce7b-96c8-464a-aa14-28487cc5b373" volumeName="kubernetes.io/projected/3984ce7b-96c8-464a-aa14-28487cc5b373-kube-api-access-plfvd" seLinuxMountContext="" Nov 04 02:32:33.664837 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:32:33.664408 4405 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="4f76b032-61eb-4796-b082-47c23ff73d18" volumeName="kubernetes.io/configmap/4f76b032-61eb-4796-b082-47c23ff73d18-config-volume" seLinuxMountContext="" Nov 04 02:32:33.664837 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:32:33.664427 4405 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5590efb1-88b8-41ad-b0ee-37b2ff9442de" volumeName="kubernetes.io/projected/5590efb1-88b8-41ad-b0ee-37b2ff9442de-kube-api-access-4rpxv" seLinuxMountContext="" Nov 04 02:32:33.664837 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:32:33.664439 4405 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5a5c4b25-7a35-4290-ad1e-04b4a3617877" volumeName="kubernetes.io/configmap/5a5c4b25-7a35-4290-ad1e-04b4a3617877-service-ca-bundle" seLinuxMountContext="" Nov 04 02:32:33.664837 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:32:33.664449 4405 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b37e5c97-77c6-4bb4-a9e4-e1ae27484457" volumeName="kubernetes.io/projected/b37e5c97-77c6-4bb4-a9e4-e1ae27484457-kube-api-access-zpbsm" seLinuxMountContext="" Nov 04 02:32:33.664837 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:32:33.664470 4405 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="cb1915a0-5505-4f32-8f79-2f638465e404" volumeName="kubernetes.io/projected/cb1915a0-5505-4f32-8f79-2f638465e404-kube-api-access-n6mc7" seLinuxMountContext="" Nov 04 02:32:33.664837 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:32:33.664494 4405 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="524e33f8-723b-4868-a4f5-5d7eb92a5fe3" volumeName="kubernetes.io/projected/524e33f8-723b-4868-a4f5-5d7eb92a5fe3-kube-api-access-stzcf" seLinuxMountContext="" Nov 04 02:32:33.664837 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:32:33.664534 4405 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5590efb1-88b8-41ad-b0ee-37b2ff9442de" volumeName="kubernetes.io/configmap/5590efb1-88b8-41ad-b0ee-37b2ff9442de-whereabouts-flatfile-configmap" seLinuxMountContext="" Nov 04 02:32:33.664837 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:32:33.664563 4405 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8b030f29-dfa6-42b1-81a7-07b539ad9aa9" volumeName="kubernetes.io/empty-dir/8b030f29-dfa6-42b1-81a7-07b539ad9aa9-tmp-dir" seLinuxMountContext="" Nov 04 02:32:33.664837 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:32:33.664573 4405 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a245d359-742e-489d-9c58-eea4e686286c" volumeName="kubernetes.io/projected/a245d359-742e-489d-9c58-eea4e686286c-kube-api-access-2rpzn" seLinuxMountContext="" Nov 04 02:32:33.664837 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:32:33.664604 4405 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f58e4ac8-c132-4716-b6c0-8d8ddb9c19e5" volumeName="kubernetes.io/projected/f58e4ac8-c132-4716-b6c0-8d8ddb9c19e5-kube-api-access-b4snb" seLinuxMountContext="" Nov 04 02:32:33.664837 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:32:33.664623 4405 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ffbb2d83-5a93-4a5b-bce3-e40afb8f58ea" volumeName="kubernetes.io/configmap/ffbb2d83-5a93-4a5b-bce3-e40afb8f58ea-env-overrides" seLinuxMountContext="" Nov 04 02:32:33.664837 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:32:33.664636 4405 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="4acab1fc-c591-475a-a15f-790be8131547" volumeName="kubernetes.io/projected/4acab1fc-c591-475a-a15f-790be8131547-kube-api-access-bkf4l" seLinuxMountContext="" Nov 04 02:32:33.664837 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:32:33.664646 4405 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="4f76b032-61eb-4796-b082-47c23ff73d18" volumeName="kubernetes.io/projected/4f76b032-61eb-4796-b082-47c23ff73d18-kube-api-access-8lqpv" seLinuxMountContext="" Nov 04 02:32:33.664837 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:32:33.664711 4405 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5a5c4b25-7a35-4290-ad1e-04b4a3617877" volumeName="kubernetes.io/empty-dir/5a5c4b25-7a35-4290-ad1e-04b4a3617877-archives-path" seLinuxMountContext="" Nov 04 02:32:33.664837 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:32:33.664728 4405 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="cb1915a0-5505-4f32-8f79-2f638465e404" volumeName="kubernetes.io/empty-dir/cb1915a0-5505-4f32-8f79-2f638465e404-etc-tuned" seLinuxMountContext="" Nov 04 02:32:33.664837 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:32:33.664739 4405 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="6e27a76f-fbdb-4ce6-9df6-659cab9ad06c" volumeName="kubernetes.io/projected/6e27a76f-fbdb-4ce6-9df6-659cab9ad06c-kube-api-access-dddmg" seLinuxMountContext="" Nov 04 02:32:33.664837 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:32:33.664760 4405 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7cb29949-bfe4-47a2-bc17-115a8d4d895d" volumeName="kubernetes.io/configmap/7cb29949-bfe4-47a2-bc17-115a8d4d895d-serviceca" seLinuxMountContext="" Nov 04 02:32:33.664837 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:32:33.664779 4405 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ffbb2d83-5a93-4a5b-bce3-e40afb8f58ea" volumeName="kubernetes.io/configmap/ffbb2d83-5a93-4a5b-bce3-e40afb8f58ea-ovnkube-script-lib" seLinuxMountContext="" Nov 04 02:32:33.664837 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:32:33.664790 4405 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="4acab1fc-c591-475a-a15f-790be8131547" volumeName="kubernetes.io/configmap/4acab1fc-c591-475a-a15f-790be8131547-kube-rbac-proxy-cm" seLinuxMountContext="" Nov 04 02:32:33.664837 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:32:33.664812 4405 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="4acab1fc-c591-475a-a15f-790be8131547" volumeName="kubernetes.io/empty-dir/4acab1fc-c591-475a-a15f-790be8131547-data-volume" seLinuxMountContext="" Nov 04 02:32:33.666054 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:32:33.664889 4405 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5a5c4b25-7a35-4290-ad1e-04b4a3617877" volumeName="kubernetes.io/projected/5a5c4b25-7a35-4290-ad1e-04b4a3617877-kube-api-access-h288h" seLinuxMountContext="" Nov 04 02:32:33.666054 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:32:33.664919 4405 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5b77e30b-cb0c-49c1-80b9-330a138dbfb7" volumeName="kubernetes.io/configmap/5b77e30b-cb0c-49c1-80b9-330a138dbfb7-konnectivity-ca" seLinuxMountContext="" Nov 04 02:32:33.666054 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:32:33.664943 4405 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ffbb2d83-5a93-4a5b-bce3-e40afb8f58ea" volumeName="kubernetes.io/secret/ffbb2d83-5a93-4a5b-bce3-e40afb8f58ea-ovn-node-metrics-cert" seLinuxMountContext="" Nov 04 02:32:33.666054 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:32:33.664955 4405 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="4f76b032-61eb-4796-b082-47c23ff73d18" volumeName="kubernetes.io/empty-dir/4f76b032-61eb-4796-b082-47c23ff73d18-tmp-dir" seLinuxMountContext="" Nov 04 02:32:33.666054 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:32:33.664970 4405 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a245d359-742e-489d-9c58-eea4e686286c" volumeName="kubernetes.io/secret/a245d359-742e-489d-9c58-eea4e686286c-cert" seLinuxMountContext="" Nov 04 02:32:33.666054 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:32:33.664999 4405 reconstruct.go:97] "Volume reconstruction finished" Nov 04 02:32:33.666054 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:32:33.665006 4405 reconciler.go:29] "Reconciler: start to sync state" Nov 04 02:32:33.666054 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:32:33.665544 4405 factory.go:55] Registering systemd factory Nov 04 02:32:33.666054 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:32:33.665556 4405 factory.go:223] Registration of the systemd container factory successfully Nov 04 02:32:33.666054 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:32:33.665957 4405 factory.go:153] Registering CRI-O factory Nov 04 02:32:33.666054 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:32:33.665980 4405 factory.go:223] Registration of the crio container factory successfully Nov 04 02:32:33.666524 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:32:33.666088 4405 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Nov 04 02:32:33.666524 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:32:33.666112 4405 factory.go:103] Registering Raw factory Nov 04 02:32:33.666524 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:32:33.666129 4405 manager.go:1199] Started watching for new ooms in manager Nov 04 02:32:33.667198 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:32:33.667176 4405 manager.go:319] Starting recovery of all containers Nov 04 02:32:33.678860 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:32:33.678836 4405 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-vwcx8_ffbb2d83-5a93-4a5b-bce3-e40afb8f58ea/ovn-acl-logging/0.log" Nov 04 02:32:33.679418 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:32:33.679384 4405 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-vwcx8_ffbb2d83-5a93-4a5b-bce3-e40afb8f58ea/ovn-acl-logging/0.log" Nov 04 02:32:33.691717 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:32:33.691640 4405 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Nov 04 02:32:33.693165 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:32:33.693137 4405 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Nov 04 02:32:33.693165 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:32:33.693163 4405 status_manager.go:249] "Starting to sync pod status with apiserver" Nov 04 02:32:33.693311 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:32:33.693207 4405 kubelet.go:2521] "Starting kubelet main sync loop" Nov 04 02:32:33.693311 ip-10-0-134-125 kubenswrapper[4405]: E1104 02:32:33.693247 4405 kubelet.go:2545] "Skipping pod synchronization" err="[container runtime status check may not have completed yet, PLEG is not healthy: pleg has yet to be successful]" Nov 04 02:32:33.703396 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:32:33.703367 4405 generic.go:356] "Generic (PLEG): container finished" podID="5590efb1-88b8-41ad-b0ee-37b2ff9442de" containerID="521d111ac8b37148a2e99975bc5c4b1a1b607ddc48fe54f97d58a90d8248e495" exitCode=0 Nov 04 02:32:33.703396 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:32:33.703394 4405 generic.go:356] "Generic (PLEG): container finished" podID="5590efb1-88b8-41ad-b0ee-37b2ff9442de" containerID="5cb7eb23c657a2e50d9f8d7a68926096010528f0b6598f76c429b3485e5eed39" exitCode=0 Nov 04 02:32:33.703541 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:32:33.703403 4405 generic.go:356] "Generic (PLEG): container finished" podID="5590efb1-88b8-41ad-b0ee-37b2ff9442de" containerID="52fa444baef42ca6a81a26cc17ea54dd984a1d9854f060b085428a6d0c43f4ff" exitCode=0 Nov 04 02:32:33.703541 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:32:33.703413 4405 generic.go:356] "Generic (PLEG): container finished" podID="5590efb1-88b8-41ad-b0ee-37b2ff9442de" containerID="47a4ead167d300937b05838143c2ee79d5c09b04650c4f7dfa4325807791e10e" exitCode=0 Nov 04 02:32:33.703541 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:32:33.703418 4405 generic.go:356] "Generic (PLEG): container finished" podID="5590efb1-88b8-41ad-b0ee-37b2ff9442de" containerID="8ac295934f2b3ad233e3ef4ab5980130273940e207feb3bda29db1624fb0d924" exitCode=0 Nov 04 02:32:33.703541 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:32:33.703423 4405 generic.go:356] "Generic (PLEG): container finished" podID="5590efb1-88b8-41ad-b0ee-37b2ff9442de" containerID="21fb821c31c7d33fdcbecb34e143b49e3552980addd21d256bc02be70fc79d40" exitCode=0 Nov 04 02:32:33.705810 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:32:33.705791 4405 generic.go:356] "Generic (PLEG): container finished" podID="c7038986364b9e4776a5785da4dd2f01" containerID="4cf7fe820ecd1cd17891819005268af501eb1d83dc1a8304c2d6bff27e88bc6d" exitCode=0 Nov 04 02:32:33.722319 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:32:33.722290 4405 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-vwcx8_ffbb2d83-5a93-4a5b-bce3-e40afb8f58ea/ovn-acl-logging/0.log" Nov 04 02:32:33.722721 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:32:33.722649 4405 generic.go:356] "Generic (PLEG): container finished" podID="ffbb2d83-5a93-4a5b-bce3-e40afb8f58ea" containerID="27e1d382d0a808b2f653af8eda862b4e951c02cee226bb3ed628e1179030bddb" exitCode=1 Nov 04 02:32:33.734328 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:32:33.734305 4405 manager.go:324] Recovery completed Nov 04 02:32:33.746619 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:32:33.746595 4405 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 04 02:32:33.747827 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:32:33.747808 4405 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-125.us-west-1.compute.internal" event="NodeHasSufficientMemory" Nov 04 02:32:33.747915 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:32:33.747847 4405 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-125.us-west-1.compute.internal" event="NodeHasNoDiskPressure" Nov 04 02:32:33.747915 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:32:33.747862 4405 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-125.us-west-1.compute.internal" event="NodeHasSufficientPID" Nov 04 02:32:33.749321 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:32:33.749304 4405 cpu_manager.go:226] "Starting" policy="none" Nov 04 02:32:33.749321 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:32:33.749315 4405 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 04 02:32:33.749424 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:32:33.749319 4405 cpu_manager.go:227] "Reconciling" reconcilePeriod="10s" Nov 04 02:32:33.749424 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:32:33.749380 4405 state_mem.go:41] "Initialized" logger="CPUManager state checkpoint.CPUManager state memory" Nov 04 02:32:33.749556 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:32:33.749540 4405 state_mem.go:94] "Updated default CPUSet" logger="CPUManager state checkpoint.CPUManager state memory" cpuSet="" Nov 04 02:32:33.749596 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:32:33.749557 4405 state_mem.go:102] "Updated CPUSet assignments" logger="CPUManager state checkpoint.CPUManager state memory" assignments={} Nov 04 02:32:33.749596 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:32:33.749575 4405 state_checkpoint.go:142] "restored state from checkpoint" logger="CPUManager state checkpoint" Nov 04 02:32:33.749596 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:32:33.749581 4405 state_checkpoint.go:143] "defaultCPUSet" logger="CPUManager state checkpoint" defaultCpuSet="" Nov 04 02:32:33.749596 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:32:33.749586 4405 policy_none.go:50] "Start" Nov 04 02:32:33.749596 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:32:33.749598 4405 memory_manager.go:187] "Starting memorymanager" policy="None" Nov 04 02:32:33.749828 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:32:33.749607 4405 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Nov 04 02:32:33.750254 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:32:33.750238 4405 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-125.us-west-1.compute.internal" event="NodeHasSufficientMemory" Nov 04 02:32:33.750320 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:32:33.750265 4405 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-125.us-west-1.compute.internal" event="NodeHasNoDiskPressure" Nov 04 02:32:33.750320 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:32:33.750276 4405 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-125.us-west-1.compute.internal" event="NodeHasSufficientPID" Nov 04 02:32:33.752447 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:32:33.752431 4405 state_mem.go:77] "Updated machine memory state" logger="Memory Manager state checkpoint" Nov 04 02:32:33.752505 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:32:33.752450 4405 state_checkpoint.go:85] "State checkpoint: restored state from checkpoint" logger="Memory Manager state checkpoint" Nov 04 02:32:33.752505 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:32:33.752458 4405 policy_none.go:44] "Start" Nov 04 02:32:33.758059 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:32:33.758030 4405 node_container_manager_linux.go:130] "Enforcing system reserved compressible on cgroup" cgroupName="/system.slice" limits={"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"} Nov 04 02:32:33.761621 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:32:33.761607 4405 manager.go:341] "Starting Device Plugin manager" Nov 04 02:32:33.761715 ip-10-0-134-125 kubenswrapper[4405]: E1104 02:32:33.761637 4405 manager.go:525] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Nov 04 02:32:33.761715 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:32:33.761649 4405 server.go:87] "Starting device plugin registration server" Nov 04 02:32:33.761982 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:32:33.761967 4405 eviction_manager.go:194] "Eviction manager: starting control loop" Nov 04 02:32:33.762043 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:32:33.761983 4405 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Nov 04 02:32:33.762085 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:32:33.762072 4405 plugin_watcher.go:53] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Nov 04 02:32:33.762215 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:32:33.762172 4405 plugin_watcher.go:201] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Nov 04 02:32:33.762304 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:32:33.762229 4405 plugin_manager.go:119] "The desired_state_of_world populator (plugin watcher) starts" Nov 04 02:32:33.762304 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:32:33.762237 4405 plugin_manager.go:121] "Starting Kubelet Plugin Manager" Nov 04 02:32:33.762381 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:32:33.762293 4405 reconciler.go:167] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-11-04T02:32:33.762217957Z","UUID":"e0f8c6a6-7702-4b44-8816-a84fc0435ec8","Handler":null,"Name":"","Endpoint":""} Nov 04 02:32:33.763773 ip-10-0-134-125 kubenswrapper[4405]: E1104 02:32:33.763747 4405 eviction_manager.go:272] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Nov 04 02:32:33.763849 ip-10-0-134-125 kubenswrapper[4405]: E1104 02:32:33.763796 4405 eviction_manager.go:297] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-134-125.us-west-1.compute.internal\" not found" Nov 04 02:32:33.763849 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:32:33.763795 4405 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Nov 04 02:32:33.763849 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:32:33.763831 4405 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Nov 04 02:32:33.793527 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:32:33.793481 4405 kubelet.go:2608] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-125.us-west-1.compute.internal","kube-system/kube-apiserver-proxy-ip-10-0-134-125.us-west-1.compute.internal"] Nov 04 02:32:33.793697 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:32:33.793619 4405 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 04 02:32:33.794621 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:32:33.794599 4405 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-125.us-west-1.compute.internal" event="NodeHasSufficientMemory" Nov 04 02:32:33.794737 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:32:33.794639 4405 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-125.us-west-1.compute.internal" event="NodeHasNoDiskPressure" Nov 04 02:32:33.794737 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:32:33.794691 4405 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-125.us-west-1.compute.internal" event="NodeHasSufficientPID" Nov 04 02:32:33.794925 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:32:33.794915 4405 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 04 02:32:33.795070 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:32:33.795054 4405 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 04 02:32:33.795661 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:32:33.795645 4405 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-125.us-west-1.compute.internal" event="NodeHasSufficientMemory" Nov 04 02:32:33.795661 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:32:33.795684 4405 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-125.us-west-1.compute.internal" event="NodeHasNoDiskPressure" Nov 04 02:32:33.795828 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:32:33.795698 4405 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-125.us-west-1.compute.internal" event="NodeHasSufficientPID" Nov 04 02:32:33.795828 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:32:33.795647 4405 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-125.us-west-1.compute.internal" event="NodeHasSufficientMemory" Nov 04 02:32:33.795828 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:32:33.795781 4405 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-125.us-west-1.compute.internal" event="NodeHasNoDiskPressure" Nov 04 02:32:33.795828 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:32:33.795800 4405 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-125.us-west-1.compute.internal" event="NodeHasSufficientPID" Nov 04 02:32:33.796018 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:32:33.795921 4405 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-125.us-west-1.compute.internal" event={"ID":"c7038986364b9e4776a5785da4dd2f01","Type":"ContainerStarted","Data":"d3f240aff1ddde7aceaab32efe3b3fb9c5ea9ca055c94b8e5e5da75da0ea7d0b"} Nov 04 02:32:33.796018 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:32:33.795956 4405 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-125.us-west-1.compute.internal" event={"ID":"c7038986364b9e4776a5785da4dd2f01","Type":"ContainerDied","Data":"4cf7fe820ecd1cd17891819005268af501eb1d83dc1a8304c2d6bff27e88bc6d"} Nov 04 02:32:33.796018 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:32:33.795970 4405 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-125.us-west-1.compute.internal" event={"ID":"c7038986364b9e4776a5785da4dd2f01","Type":"ContainerStarted","Data":"e2d730f5206d875c1b306c4525077cc3151c5f42596a1e751c4a0de261d0d629"} Nov 04 02:32:33.796018 ip-10-0-134-125 kubenswrapper[4405]: E1104 02:32:33.795985 4405 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-134-125.us-west-1.compute.internal\" not found" node="ip-10-0-134-125.us-west-1.compute.internal" Nov 04 02:32:33.796018 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:32:33.796015 4405 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 04 02:32:33.796183 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:32:33.796023 4405 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-134-125.us-west-1.compute.internal" event={"ID":"ee7b213d76bf8a5295d150c84b46451e","Type":"ContainerStarted","Data":"8013a1cf8342b8cb45922287bcb35dd722adf8508be17006cefe3b209b699b38"} Nov 04 02:32:33.796183 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:32:33.796038 4405 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-134-125.us-west-1.compute.internal" event={"ID":"ee7b213d76bf8a5295d150c84b46451e","Type":"ContainerStarted","Data":"e3f6ee0a8c70b4d6cb30ed7c29e4392e9078922367521917c71e40e38a5db2c3"} Nov 04 02:32:33.796721 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:32:33.796706 4405 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-125.us-west-1.compute.internal" event="NodeHasSufficientMemory" Nov 04 02:32:33.796794 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:32:33.796739 4405 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-125.us-west-1.compute.internal" event="NodeHasNoDiskPressure" Nov 04 02:32:33.796794 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:32:33.796757 4405 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-125.us-west-1.compute.internal" event="NodeHasSufficientPID" Nov 04 02:32:33.796926 ip-10-0-134-125 kubenswrapper[4405]: E1104 02:32:33.796913 4405 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-134-125.us-west-1.compute.internal\" not found" node="ip-10-0-134-125.us-west-1.compute.internal" Nov 04 02:32:33.857130 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:32:33.857092 4405 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161" Nov 04 02:32:33.857130 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:32:33.857101 4405 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161" Nov 04 02:32:33.857354 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:32:33.857206 4405 reconstruct.go:205] "DevicePaths of reconstructed volumes updated" Nov 04 02:32:33.858016 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:32:33.858003 4405 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161" Nov 04 02:32:33.862085 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:32:33.862070 4405 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 04 02:32:33.862910 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:32:33.862877 4405 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-134-125.us-west-1.compute.internal" Nov 04 02:32:33.863062 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:32:33.863045 4405 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-125.us-west-1.compute.internal" event="NodeHasSufficientMemory" Nov 04 02:32:33.863145 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:32:33.863074 4405 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-125.us-west-1.compute.internal" event="NodeHasNoDiskPressure" Nov 04 02:32:33.863145 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:32:33.863084 4405 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-125.us-west-1.compute.internal" event="NodeHasSufficientPID" Nov 04 02:32:33.863145 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:32:33.863128 4405 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-134-125.us-west-1.compute.internal" Nov 04 02:32:33.907484 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:32:33.907460 4405 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161" Nov 04 02:32:33.955328 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:32:33.955260 4405 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Nov 04 02:32:33.955328 ip-10-0-134-125 kubenswrapper[4405]: E1104 02:32:33.955306 4405 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-apiserver-proxy-ip-10-0-134-125.us-west-1.compute.internal\" already exists" pod="kube-system/kube-apiserver-proxy-ip-10-0-134-125.us-west-1.compute.internal" Nov 04 02:32:33.955328 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:32:33.955331 4405 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-125.us-west-1.compute.internal" Nov 04 02:32:33.957431 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:32:33.957410 4405 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/c7038986364b9e4776a5785da4dd2f01-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-134-125.us-west-1.compute.internal\" (UID: \"c7038986364b9e4776a5785da4dd2f01\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-125.us-west-1.compute.internal" Nov 04 02:32:33.957504 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:32:33.957448 4405 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/c7038986364b9e4776a5785da4dd2f01-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-134-125.us-west-1.compute.internal\" (UID: \"c7038986364b9e4776a5785da4dd2f01\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-125.us-west-1.compute.internal" Nov 04 02:32:33.957504 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:32:33.957475 4405 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/ee7b213d76bf8a5295d150c84b46451e-config\") pod \"kube-apiserver-proxy-ip-10-0-134-125.us-west-1.compute.internal\" (UID: \"ee7b213d76bf8a5295d150c84b46451e\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-134-125.us-west-1.compute.internal" Nov 04 02:32:33.974024 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:32:33.974005 4405 kubelet_node_status.go:126] "Node was previously registered" node="ip-10-0-134-125.us-west-1.compute.internal" Nov 04 02:32:33.974087 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:32:33.974065 4405 kubelet_node_status.go:80] "Successfully registered node" node="ip-10-0-134-125.us-west-1.compute.internal" Nov 04 02:32:33.974709 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:32:33.974695 4405 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-125.us-west-1.compute.internal" event="NodeNotReady" Nov 04 02:32:33.974757 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:32:33.974728 4405 setters.go:546] "Node became not ready" node="ip-10-0-134-125.us-west-1.compute.internal" condition={"type":"Ready","status":"False","lastHeartbeatTime":"2026-11-04T02:32:33Z","lastTransitionTime":"2026-11-04T02:32:33Z","reason":"KubeletNotReady","message":"CSINode is not yet initialized"} Nov 04 02:32:34.015929 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:32:34.015906 4405 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Nov 04 02:32:34.015994 ip-10-0-134-125 kubenswrapper[4405]: E1104 02:32:34.015953 4405 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-rbac-proxy-crio-ip-10-0-134-125.us-west-1.compute.internal\" already exists" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-125.us-west-1.compute.internal" Nov 04 02:32:34.020550 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:32:34.020522 4405 nodeinfomanager.go:471] existing CSINode "ip-10-0-134-125.us-west-1.compute.internal" is owned by different node (oldNodeID="a30e3fce-58fc-4baf-b594-3eaceede55b3", newNodeID=""), cleaning up... Nov 04 02:32:34.057962 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:32:34.057931 4405 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/c7038986364b9e4776a5785da4dd2f01-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-134-125.us-west-1.compute.internal\" (UID: \"c7038986364b9e4776a5785da4dd2f01\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-125.us-west-1.compute.internal" Nov 04 02:32:34.058133 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:32:34.057963 4405 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/c7038986364b9e4776a5785da4dd2f01-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-134-125.us-west-1.compute.internal\" (UID: \"c7038986364b9e4776a5785da4dd2f01\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-125.us-west-1.compute.internal" Nov 04 02:32:34.058133 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:32:34.057989 4405 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/ee7b213d76bf8a5295d150c84b46451e-config\") pod \"kube-apiserver-proxy-ip-10-0-134-125.us-west-1.compute.internal\" (UID: \"ee7b213d76bf8a5295d150c84b46451e\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-134-125.us-west-1.compute.internal" Nov 04 02:32:34.058133 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:32:34.058032 4405 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/ee7b213d76bf8a5295d150c84b46451e-config\") pod \"kube-apiserver-proxy-ip-10-0-134-125.us-west-1.compute.internal\" (UID: \"ee7b213d76bf8a5295d150c84b46451e\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-134-125.us-west-1.compute.internal" Nov 04 02:32:34.058133 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:32:34.058048 4405 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/c7038986364b9e4776a5785da4dd2f01-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-134-125.us-west-1.compute.internal\" (UID: \"c7038986364b9e4776a5785da4dd2f01\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-125.us-west-1.compute.internal" Nov 04 02:32:34.058133 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:32:34.058048 4405 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/c7038986364b9e4776a5785da4dd2f01-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-134-125.us-west-1.compute.internal\" (UID: \"c7038986364b9e4776a5785da4dd2f01\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-125.us-west-1.compute.internal" Nov 04 02:32:34.645984 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:32:34.645947 4405 apiserver.go:52] "Watching apiserver" Nov 04 02:32:34.732776 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:32:34.732748 4405 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-134-125.us-west-1.compute.internal" Nov 04 02:32:34.732913 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:32:34.732859 4405 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-125.us-west-1.compute.internal" Nov 04 02:32:34.748928 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:32:34.748907 4405 reflector.go:446] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Nov 04 02:32:34.749311 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:32:34.749293 4405 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6mvxj","openshift-dns/node-resolver-kzzwk","openshift-insights/insights-runtime-extractor-5f2vb","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-125.us-west-1.compute.internal","openshift-network-diagnostics/network-check-target-zd8z9","kube-system/kube-apiserver-proxy-ip-10-0-134-125.us-west-1.compute.internal","openshift-ingress-canary/ingress-canary-mn7dn","openshift-image-registry/node-ca-2h96g","openshift-insights/periodic-gathering-hxwj2-xcshl","openshift-multus/network-metrics-daemon-bdcds","openshift-dns/dns-default-wbzfc","openshift-network-operator/iptables-alerter-sx4w7","kube-system/global-pull-secret-syncer-krvqk","openshift-multus/multus-tglvr","openshift-ovn-kubernetes/ovnkube-node-vwcx8","openshift-multus/multus-additional-cni-plugins-sgx95","kube-system/konnectivity-agent-wcp7z","openshift-cluster-node-tuning-operator/tuned-ncfkk"] Nov 04 02:32:34.751566 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:32:34.751541 4405 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bdcds" Nov 04 02:32:34.753142 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:32:34.753123 4405 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-wbzfc" Nov 04 02:32:34.754320 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:32:34.754303 4405 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-5f2vb" Nov 04 02:32:34.763467 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:32:34.763449 4405 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Nov 04 02:32:34.763602 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:32:34.763588 4405 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/3984ce7b-96c8-464a-aa14-28487cc5b373-multus-socket-dir-parent\") pod \"multus-tglvr\" (UID: \"3984ce7b-96c8-464a-aa14-28487cc5b373\") " pod="openshift-multus/multus-tglvr" Nov 04 02:32:34.763648 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:32:34.763616 4405 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/5590efb1-88b8-41ad-b0ee-37b2ff9442de-cni-binary-copy\") pod \"multus-additional-cni-plugins-sgx95\" (UID: \"5590efb1-88b8-41ad-b0ee-37b2ff9442de\") " pod="openshift-multus/multus-additional-cni-plugins-sgx95" Nov 04 02:32:34.763648 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:32:34.763634 4405 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/f58e4ac8-c132-4716-b6c0-8d8ddb9c19e5-etc-selinux\") pod \"aws-ebs-csi-driver-node-6mvxj\" (UID: \"f58e4ac8-c132-4716-b6c0-8d8ddb9c19e5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6mvxj" Nov 04 02:32:34.763733 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:32:34.763659 4405 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ffbb2d83-5a93-4a5b-bce3-e40afb8f58ea-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-vwcx8\" (UID: \"ffbb2d83-5a93-4a5b-bce3-e40afb8f58ea\") " pod="openshift-ovn-kubernetes/ovnkube-node-vwcx8" Nov 04 02:32:34.763733 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:32:34.763720 4405 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/3984ce7b-96c8-464a-aa14-28487cc5b373-etc-kubernetes\") pod \"multus-tglvr\" (UID: \"3984ce7b-96c8-464a-aa14-28487cc5b373\") " pod="openshift-multus/multus-tglvr" Nov 04 02:32:34.763793 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:32:34.763749 4405 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-946v4\" (UniqueName: \"kubernetes.io/projected/7cb29949-bfe4-47a2-bc17-115a8d4d895d-kube-api-access-946v4\") pod \"node-ca-2h96g\" (UID: \"7cb29949-bfe4-47a2-bc17-115a8d4d895d\") " pod="openshift-image-registry/node-ca-2h96g" Nov 04 02:32:34.763827 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:32:34.763788 4405 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8b030f29-dfa6-42b1-81a7-07b539ad9aa9-tmp-dir\") pod \"node-resolver-kzzwk\" (UID: \"8b030f29-dfa6-42b1-81a7-07b539ad9aa9\") " pod="openshift-dns/node-resolver-kzzwk" Nov 04 02:32:34.763869 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:32:34.763857 4405 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8b030f29-dfa6-42b1-81a7-07b539ad9aa9-tmp-dir\") pod \"node-resolver-kzzwk\" (UID: \"8b030f29-dfa6-42b1-81a7-07b539ad9aa9\") " pod="openshift-dns/node-resolver-kzzwk" Nov 04 02:32:34.763900 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:32:34.763871 4405 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/5590efb1-88b8-41ad-b0ee-37b2ff9442de-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-sgx95\" (UID: \"5590efb1-88b8-41ad-b0ee-37b2ff9442de\") " pod="openshift-multus/multus-additional-cni-plugins-sgx95" Nov 04 02:32:34.763931 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:32:34.763903 4405 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/3a7ab1db-ad88-40d9-a7e9-b511bbd4584c-original-pull-secret\") pod \"global-pull-secret-syncer-krvqk\" (UID: \"3a7ab1db-ad88-40d9-a7e9-b511bbd4584c\") " pod="kube-system/global-pull-secret-syncer-krvqk" Nov 04 02:32:34.763965 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:32:34.763932 4405 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/cb1915a0-5505-4f32-8f79-2f638465e404-var-lib-kubelet\") pod \"tuned-ncfkk\" (UID: \"cb1915a0-5505-4f32-8f79-2f638465e404\") " pod="openshift-cluster-node-tuning-operator/tuned-ncfkk" Nov 04 02:32:34.763965 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:32:34.763955 4405 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/ffbb2d83-5a93-4a5b-bce3-e40afb8f58ea-etc-openvswitch\") pod \"ovnkube-node-vwcx8\" (UID: \"ffbb2d83-5a93-4a5b-bce3-e40afb8f58ea\") " pod="openshift-ovn-kubernetes/ovnkube-node-vwcx8" Nov 04 02:32:34.764039 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:32:34.763983 4405 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/3984ce7b-96c8-464a-aa14-28487cc5b373-system-cni-dir\") pod \"multus-tglvr\" (UID: \"3984ce7b-96c8-464a-aa14-28487cc5b373\") " pod="openshift-multus/multus-tglvr" Nov 04 02:32:34.764039 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:32:34.764006 4405 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/3984ce7b-96c8-464a-aa14-28487cc5b373-hostroot\") pod \"multus-tglvr\" (UID: \"3984ce7b-96c8-464a-aa14-28487cc5b373\") " pod="openshift-multus/multus-tglvr" Nov 04 02:32:34.764039 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:32:34.764024 4405 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/5590efb1-88b8-41ad-b0ee-37b2ff9442de-cnibin\") pod \"multus-additional-cni-plugins-sgx95\" (UID: \"5590efb1-88b8-41ad-b0ee-37b2ff9442de\") " pod="openshift-multus/multus-additional-cni-plugins-sgx95" Nov 04 02:32:34.764151 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:32:34.764044 4405 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zpbsm\" (UniqueName: \"kubernetes.io/projected/b37e5c97-77c6-4bb4-a9e4-e1ae27484457-kube-api-access-zpbsm\") pod \"network-metrics-daemon-bdcds\" (UID: \"b37e5c97-77c6-4bb4-a9e4-e1ae27484457\") " pod="openshift-multus/network-metrics-daemon-bdcds" Nov 04 02:32:34.764151 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:32:34.764072 4405 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/ffbb2d83-5a93-4a5b-bce3-e40afb8f58ea-ovnkube-config\") pod \"ovnkube-node-vwcx8\" (UID: \"ffbb2d83-5a93-4a5b-bce3-e40afb8f58ea\") " pod="openshift-ovn-kubernetes/ovnkube-node-vwcx8" Nov 04 02:32:34.764151 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:32:34.764139 4405 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/ffbb2d83-5a93-4a5b-bce3-e40afb8f58ea-env-overrides\") pod \"ovnkube-node-vwcx8\" (UID: \"ffbb2d83-5a93-4a5b-bce3-e40afb8f58ea\") " pod="openshift-ovn-kubernetes/ovnkube-node-vwcx8" Nov 04 02:32:34.764292 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:32:34.764177 4405 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tp4jl\" (UniqueName: \"kubernetes.io/projected/ffbb2d83-5a93-4a5b-bce3-e40afb8f58ea-kube-api-access-tp4jl\") pod \"ovnkube-node-vwcx8\" (UID: \"ffbb2d83-5a93-4a5b-bce3-e40afb8f58ea\") " pod="openshift-ovn-kubernetes/ovnkube-node-vwcx8" Nov 04 02:32:34.764292 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:32:34.764270 4405 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/cb1915a0-5505-4f32-8f79-2f638465e404-etc-kubernetes\") pod \"tuned-ncfkk\" (UID: \"cb1915a0-5505-4f32-8f79-2f638465e404\") " pod="openshift-cluster-node-tuning-operator/tuned-ncfkk" Nov 04 02:32:34.764385 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:32:34.764293 4405 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/cb1915a0-5505-4f32-8f79-2f638465e404-etc-tuned\") pod \"tuned-ncfkk\" (UID: \"cb1915a0-5505-4f32-8f79-2f638465e404\") " pod="openshift-cluster-node-tuning-operator/tuned-ncfkk" Nov 04 02:32:34.764385 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:32:34.764354 4405 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/cb1915a0-5505-4f32-8f79-2f638465e404-tmp\") pod \"tuned-ncfkk\" (UID: \"cb1915a0-5505-4f32-8f79-2f638465e404\") " pod="openshift-cluster-node-tuning-operator/tuned-ncfkk" Nov 04 02:32:34.764385 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:32:34.764362 4405 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/cb1915a0-5505-4f32-8f79-2f638465e404-etc-tuned\") pod \"tuned-ncfkk\" (UID: \"cb1915a0-5505-4f32-8f79-2f638465e404\") " pod="openshift-cluster-node-tuning-operator/tuned-ncfkk" Nov 04 02:32:34.764524 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:32:34.764393 4405 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/5b77e30b-cb0c-49c1-80b9-330a138dbfb7-agent-certs\") pod \"konnectivity-agent-wcp7z\" (UID: \"5b77e30b-cb0c-49c1-80b9-330a138dbfb7\") " pod="kube-system/konnectivity-agent-wcp7z" Nov 04 02:32:34.764524 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:32:34.764409 4405 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/cb1915a0-5505-4f32-8f79-2f638465e404-tmp\") pod \"tuned-ncfkk\" (UID: \"cb1915a0-5505-4f32-8f79-2f638465e404\") " pod="openshift-cluster-node-tuning-operator/tuned-ncfkk" Nov 04 02:32:34.764524 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:32:34.764428 4405 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/3984ce7b-96c8-464a-aa14-28487cc5b373-cni-binary-copy\") pod \"multus-tglvr\" (UID: \"3984ce7b-96c8-464a-aa14-28487cc5b373\") " pod="openshift-multus/multus-tglvr" Nov 04 02:32:34.764524 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:32:34.764454 4405 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h288h\" (UniqueName: \"kubernetes.io/projected/5a5c4b25-7a35-4290-ad1e-04b4a3617877-kube-api-access-h288h\") pod \"periodic-gathering-hxwj2-xcshl\" (UID: \"5a5c4b25-7a35-4290-ad1e-04b4a3617877\") " pod="openshift-insights/periodic-gathering-hxwj2-xcshl" Nov 04 02:32:34.764524 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:32:34.764474 4405 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/ffbb2d83-5a93-4a5b-bce3-e40afb8f58ea-systemd-units\") pod \"ovnkube-node-vwcx8\" (UID: \"ffbb2d83-5a93-4a5b-bce3-e40afb8f58ea\") " pod="openshift-ovn-kubernetes/ovnkube-node-vwcx8" Nov 04 02:32:34.764524 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:32:34.764492 4405 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dddmg\" (UniqueName: \"kubernetes.io/projected/6e27a76f-fbdb-4ce6-9df6-659cab9ad06c-kube-api-access-dddmg\") pod \"network-check-target-zd8z9\" (UID: \"6e27a76f-fbdb-4ce6-9df6-659cab9ad06c\") " pod="openshift-network-diagnostics/network-check-target-zd8z9" Nov 04 02:32:34.765010 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:32:34.764990 4405 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/f58e4ac8-c132-4716-b6c0-8d8ddb9c19e5-socket-dir\") pod \"aws-ebs-csi-driver-node-6mvxj\" (UID: \"f58e4ac8-c132-4716-b6c0-8d8ddb9c19e5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6mvxj" Nov 04 02:32:34.765069 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:32:34.765040 4405 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/f58e4ac8-c132-4716-b6c0-8d8ddb9c19e5-registration-dir\") pod \"aws-ebs-csi-driver-node-6mvxj\" (UID: \"f58e4ac8-c132-4716-b6c0-8d8ddb9c19e5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6mvxj" Nov 04 02:32:34.765108 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:32:34.765089 4405 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/f58e4ac8-c132-4716-b6c0-8d8ddb9c19e5-sys-fs\") pod \"aws-ebs-csi-driver-node-6mvxj\" (UID: \"f58e4ac8-c132-4716-b6c0-8d8ddb9c19e5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6mvxj" Nov 04 02:32:34.765155 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:32:34.765116 4405 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/cb1915a0-5505-4f32-8f79-2f638465e404-etc-sysctl-d\") pod \"tuned-ncfkk\" (UID: \"cb1915a0-5505-4f32-8f79-2f638465e404\") " pod="openshift-cluster-node-tuning-operator/tuned-ncfkk" Nov 04 02:32:34.765155 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:32:34.765133 4405 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/cb1915a0-5505-4f32-8f79-2f638465e404-sys\") pod \"tuned-ncfkk\" (UID: \"cb1915a0-5505-4f32-8f79-2f638465e404\") " pod="openshift-cluster-node-tuning-operator/tuned-ncfkk" Nov 04 02:32:34.765250 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:32:34.765196 4405 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/3984ce7b-96c8-464a-aa14-28487cc5b373-host-var-lib-kubelet\") pod \"multus-tglvr\" (UID: \"3984ce7b-96c8-464a-aa14-28487cc5b373\") " pod="openshift-multus/multus-tglvr" Nov 04 02:32:34.765250 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:32:34.765224 4405 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/f58e4ac8-c132-4716-b6c0-8d8ddb9c19e5-kubelet-dir\") pod \"aws-ebs-csi-driver-node-6mvxj\" (UID: \"f58e4ac8-c132-4716-b6c0-8d8ddb9c19e5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6mvxj" Nov 04 02:32:34.765351 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:32:34.765249 4405 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/cb1915a0-5505-4f32-8f79-2f638465e404-etc-modprobe-d\") pod \"tuned-ncfkk\" (UID: \"cb1915a0-5505-4f32-8f79-2f638465e404\") " pod="openshift-cluster-node-tuning-operator/tuned-ncfkk" Nov 04 02:32:34.765351 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:32:34.765272 4405 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/cb1915a0-5505-4f32-8f79-2f638465e404-run\") pod \"tuned-ncfkk\" (UID: \"cb1915a0-5505-4f32-8f79-2f638465e404\") " pod="openshift-cluster-node-tuning-operator/tuned-ncfkk" Nov 04 02:32:34.765351 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:32:34.765308 4405 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/ffbb2d83-5a93-4a5b-bce3-e40afb8f58ea-ovnkube-script-lib\") pod \"ovnkube-node-vwcx8\" (UID: \"ffbb2d83-5a93-4a5b-bce3-e40afb8f58ea\") " pod="openshift-ovn-kubernetes/ovnkube-node-vwcx8" Nov 04 02:32:34.765351 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:32:34.765335 4405 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/4acab1fc-c591-475a-a15f-790be8131547-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-5f2vb\" (UID: \"4acab1fc-c591-475a-a15f-790be8131547\") " pod="openshift-insights/insights-runtime-extractor-5f2vb" Nov 04 02:32:34.765515 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:32:34.765361 4405 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bkf4l\" (UniqueName: \"kubernetes.io/projected/4acab1fc-c591-475a-a15f-790be8131547-kube-api-access-bkf4l\") pod \"insights-runtime-extractor-5f2vb\" (UID: \"4acab1fc-c591-475a-a15f-790be8131547\") " pod="openshift-insights/insights-runtime-extractor-5f2vb" Nov 04 02:32:34.765515 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:32:34.765387 4405 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/3984ce7b-96c8-464a-aa14-28487cc5b373-cnibin\") pod \"multus-tglvr\" (UID: \"3984ce7b-96c8-464a-aa14-28487cc5b373\") " pod="openshift-multus/multus-tglvr" Nov 04 02:32:34.765515 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:32:34.765413 4405 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-plfvd\" (UniqueName: \"kubernetes.io/projected/3984ce7b-96c8-464a-aa14-28487cc5b373-kube-api-access-plfvd\") pod \"multus-tglvr\" (UID: \"3984ce7b-96c8-464a-aa14-28487cc5b373\") " pod="openshift-multus/multus-tglvr" Nov 04 02:32:34.765515 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:32:34.765440 4405 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/524e33f8-723b-4868-a4f5-5d7eb92a5fe3-host-slash\") pod \"iptables-alerter-sx4w7\" (UID: \"524e33f8-723b-4868-a4f5-5d7eb92a5fe3\") " pod="openshift-network-operator/iptables-alerter-sx4w7" Nov 04 02:32:34.765515 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:32:34.765477 4405 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8lqpv\" (UniqueName: \"kubernetes.io/projected/4f76b032-61eb-4796-b082-47c23ff73d18-kube-api-access-8lqpv\") pod \"dns-default-wbzfc\" (UID: \"4f76b032-61eb-4796-b082-47c23ff73d18\") " pod="openshift-dns/dns-default-wbzfc" Nov 04 02:32:34.765771 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:32:34.765553 4405 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/ffbb2d83-5a93-4a5b-bce3-e40afb8f58ea-run-systemd\") pod \"ovnkube-node-vwcx8\" (UID: \"ffbb2d83-5a93-4a5b-bce3-e40afb8f58ea\") " pod="openshift-ovn-kubernetes/ovnkube-node-vwcx8" Nov 04 02:32:34.765771 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:32:34.765585 4405 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/ffbb2d83-5a93-4a5b-bce3-e40afb8f58ea-var-lib-openvswitch\") pod \"ovnkube-node-vwcx8\" (UID: \"ffbb2d83-5a93-4a5b-bce3-e40afb8f58ea\") " pod="openshift-ovn-kubernetes/ovnkube-node-vwcx8" Nov 04 02:32:34.765771 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:32:34.765614 4405 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4rpxv\" (UniqueName: \"kubernetes.io/projected/5590efb1-88b8-41ad-b0ee-37b2ff9442de-kube-api-access-4rpxv\") pod \"multus-additional-cni-plugins-sgx95\" (UID: \"5590efb1-88b8-41ad-b0ee-37b2ff9442de\") " pod="openshift-multus/multus-additional-cni-plugins-sgx95" Nov 04 02:32:34.765771 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:32:34.765651 4405 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2rpzn\" (UniqueName: \"kubernetes.io/projected/a245d359-742e-489d-9c58-eea4e686286c-kube-api-access-2rpzn\") pod \"ingress-canary-mn7dn\" (UID: \"a245d359-742e-489d-9c58-eea4e686286c\") " pod="openshift-ingress-canary/ingress-canary-mn7dn" Nov 04 02:32:34.765771 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:32:34.765702 4405 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5a5c4b25-7a35-4290-ad1e-04b4a3617877-trusted-ca-bundle\") pod \"periodic-gathering-hxwj2-xcshl\" (UID: \"5a5c4b25-7a35-4290-ad1e-04b4a3617877\") " pod="openshift-insights/periodic-gathering-hxwj2-xcshl" Nov 04 02:32:34.765771 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:32:34.765728 4405 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/cb1915a0-5505-4f32-8f79-2f638465e404-etc-sysconfig\") pod \"tuned-ncfkk\" (UID: \"cb1915a0-5505-4f32-8f79-2f638465e404\") " pod="openshift-cluster-node-tuning-operator/tuned-ncfkk" Nov 04 02:32:34.765771 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:32:34.765753 4405 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/ffbb2d83-5a93-4a5b-bce3-e40afb8f58ea-run-ovn\") pod \"ovnkube-node-vwcx8\" (UID: \"ffbb2d83-5a93-4a5b-bce3-e40afb8f58ea\") " pod="openshift-ovn-kubernetes/ovnkube-node-vwcx8" Nov 04 02:32:34.766102 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:32:34.765787 4405 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/4acab1fc-c591-475a-a15f-790be8131547-data-volume\") pod \"insights-runtime-extractor-5f2vb\" (UID: \"4acab1fc-c591-475a-a15f-790be8131547\") " pod="openshift-insights/insights-runtime-extractor-5f2vb" Nov 04 02:32:34.766102 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:32:34.765820 4405 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/5590efb1-88b8-41ad-b0ee-37b2ff9442de-tuning-conf-dir\") pod \"multus-additional-cni-plugins-sgx95\" (UID: \"5590efb1-88b8-41ad-b0ee-37b2ff9442de\") " pod="openshift-multus/multus-additional-cni-plugins-sgx95" Nov 04 02:32:34.766102 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:32:34.765846 4405 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/4f76b032-61eb-4796-b082-47c23ff73d18-metrics-tls\") pod \"dns-default-wbzfc\" (UID: \"4f76b032-61eb-4796-b082-47c23ff73d18\") " pod="openshift-dns/dns-default-wbzfc" Nov 04 02:32:34.766102 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:32:34.765871 4405 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/4acab1fc-c591-475a-a15f-790be8131547-crio-socket\") pod \"insights-runtime-extractor-5f2vb\" (UID: \"4acab1fc-c591-475a-a15f-790be8131547\") " pod="openshift-insights/insights-runtime-extractor-5f2vb" Nov 04 02:32:34.766102 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:32:34.765879 4405 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/4acab1fc-c591-475a-a15f-790be8131547-data-volume\") pod \"insights-runtime-extractor-5f2vb\" (UID: \"4acab1fc-c591-475a-a15f-790be8131547\") " pod="openshift-insights/insights-runtime-extractor-5f2vb" Nov 04 02:32:34.766102 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:32:34.765895 4405 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/3984ce7b-96c8-464a-aa14-28487cc5b373-host-var-lib-cni-multus\") pod \"multus-tglvr\" (UID: \"3984ce7b-96c8-464a-aa14-28487cc5b373\") " pod="openshift-multus/multus-tglvr" Nov 04 02:32:34.766102 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:32:34.765933 4405 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/3984ce7b-96c8-464a-aa14-28487cc5b373-multus-daemon-config\") pod \"multus-tglvr\" (UID: \"3984ce7b-96c8-464a-aa14-28487cc5b373\") " pod="openshift-multus/multus-tglvr" Nov 04 02:32:34.766102 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:32:34.765962 4405 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4f76b032-61eb-4796-b082-47c23ff73d18-tmp-dir\") pod \"dns-default-wbzfc\" (UID: \"4f76b032-61eb-4796-b082-47c23ff73d18\") " pod="openshift-dns/dns-default-wbzfc" Nov 04 02:32:34.766102 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:32:34.765987 4405 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/5a5c4b25-7a35-4290-ad1e-04b4a3617877-archives-path\") pod \"periodic-gathering-hxwj2-xcshl\" (UID: \"5a5c4b25-7a35-4290-ad1e-04b4a3617877\") " pod="openshift-insights/periodic-gathering-hxwj2-xcshl" Nov 04 02:32:34.766102 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:32:34.766011 4405 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/cb1915a0-5505-4f32-8f79-2f638465e404-lib-modules\") pod \"tuned-ncfkk\" (UID: \"cb1915a0-5505-4f32-8f79-2f638465e404\") " pod="openshift-cluster-node-tuning-operator/tuned-ncfkk" Nov 04 02:32:34.766102 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:32:34.766018 4405 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4f76b032-61eb-4796-b082-47c23ff73d18-tmp-dir\") pod \"dns-default-wbzfc\" (UID: \"4f76b032-61eb-4796-b082-47c23ff73d18\") " pod="openshift-dns/dns-default-wbzfc" Nov 04 02:32:34.766102 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:32:34.766037 4405 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/ffbb2d83-5a93-4a5b-bce3-e40afb8f58ea-host-kubelet\") pod \"ovnkube-node-vwcx8\" (UID: \"ffbb2d83-5a93-4a5b-bce3-e40afb8f58ea\") " pod="openshift-ovn-kubernetes/ovnkube-node-vwcx8" Nov 04 02:32:34.766102 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:32:34.766062 4405 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/5a5c4b25-7a35-4290-ad1e-04b4a3617877-archives-path\") pod \"periodic-gathering-hxwj2-xcshl\" (UID: \"5a5c4b25-7a35-4290-ad1e-04b4a3617877\") " pod="openshift-insights/periodic-gathering-hxwj2-xcshl" Nov 04 02:32:34.766102 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:32:34.766062 4405 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/8b030f29-dfa6-42b1-81a7-07b539ad9aa9-hosts-file\") pod \"node-resolver-kzzwk\" (UID: \"8b030f29-dfa6-42b1-81a7-07b539ad9aa9\") " pod="openshift-dns/node-resolver-kzzwk" Nov 04 02:32:34.766102 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:32:34.766097 4405 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/5590efb1-88b8-41ad-b0ee-37b2ff9442de-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-sgx95\" (UID: \"5590efb1-88b8-41ad-b0ee-37b2ff9442de\") " pod="openshift-multus/multus-additional-cni-plugins-sgx95" Nov 04 02:32:34.766750 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:32:34.766123 4405 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ffbb2d83-5a93-4a5b-bce3-e40afb8f58ea-host-run-ovn-kubernetes\") pod \"ovnkube-node-vwcx8\" (UID: \"ffbb2d83-5a93-4a5b-bce3-e40afb8f58ea\") " pod="openshift-ovn-kubernetes/ovnkube-node-vwcx8" Nov 04 02:32:34.766750 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:32:34.766152 4405 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/cb1915a0-5505-4f32-8f79-2f638465e404-etc-systemd\") pod \"tuned-ncfkk\" (UID: \"cb1915a0-5505-4f32-8f79-2f638465e404\") " pod="openshift-cluster-node-tuning-operator/tuned-ncfkk" Nov 04 02:32:34.766750 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:32:34.766191 4405 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n6mc7\" (UniqueName: \"kubernetes.io/projected/cb1915a0-5505-4f32-8f79-2f638465e404-kube-api-access-n6mc7\") pod \"tuned-ncfkk\" (UID: \"cb1915a0-5505-4f32-8f79-2f638465e404\") " pod="openshift-cluster-node-tuning-operator/tuned-ncfkk" Nov 04 02:32:34.766750 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:32:34.766217 4405 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/ffbb2d83-5a93-4a5b-bce3-e40afb8f58ea-run-openvswitch\") pod \"ovnkube-node-vwcx8\" (UID: \"ffbb2d83-5a93-4a5b-bce3-e40afb8f58ea\") " pod="openshift-ovn-kubernetes/ovnkube-node-vwcx8" Nov 04 02:32:34.766750 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:32:34.766244 4405 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/ffbb2d83-5a93-4a5b-bce3-e40afb8f58ea-log-socket\") pod \"ovnkube-node-vwcx8\" (UID: \"ffbb2d83-5a93-4a5b-bce3-e40afb8f58ea\") " pod="openshift-ovn-kubernetes/ovnkube-node-vwcx8" Nov 04 02:32:34.766750 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:32:34.766267 4405 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/3984ce7b-96c8-464a-aa14-28487cc5b373-host-run-netns\") pod \"multus-tglvr\" (UID: \"3984ce7b-96c8-464a-aa14-28487cc5b373\") " pod="openshift-multus/multus-tglvr" Nov 04 02:32:34.766750 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:32:34.766311 4405 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b37e5c97-77c6-4bb4-a9e4-e1ae27484457-metrics-certs\") pod \"network-metrics-daemon-bdcds\" (UID: \"b37e5c97-77c6-4bb4-a9e4-e1ae27484457\") " pod="openshift-multus/network-metrics-daemon-bdcds" Nov 04 02:32:34.766750 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:32:34.766344 4405 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/3a7ab1db-ad88-40d9-a7e9-b511bbd4584c-kubelet-config\") pod \"global-pull-secret-syncer-krvqk\" (UID: \"3a7ab1db-ad88-40d9-a7e9-b511bbd4584c\") " pod="kube-system/global-pull-secret-syncer-krvqk" Nov 04 02:32:34.766750 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:32:34.766381 4405 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5a5c4b25-7a35-4290-ad1e-04b4a3617877-service-ca-bundle\") pod \"periodic-gathering-hxwj2-xcshl\" (UID: \"5a5c4b25-7a35-4290-ad1e-04b4a3617877\") " pod="openshift-insights/periodic-gathering-hxwj2-xcshl" Nov 04 02:32:34.766750 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:32:34.766405 4405 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/ffbb2d83-5a93-4a5b-bce3-e40afb8f58ea-host-slash\") pod \"ovnkube-node-vwcx8\" (UID: \"ffbb2d83-5a93-4a5b-bce3-e40afb8f58ea\") " pod="openshift-ovn-kubernetes/ovnkube-node-vwcx8" Nov 04 02:32:34.766750 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:32:34.766453 4405 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/ffbb2d83-5a93-4a5b-bce3-e40afb8f58ea-ovn-node-metrics-cert\") pod \"ovnkube-node-vwcx8\" (UID: \"ffbb2d83-5a93-4a5b-bce3-e40afb8f58ea\") " pod="openshift-ovn-kubernetes/ovnkube-node-vwcx8" Nov 04 02:32:34.766750 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:32:34.766484 4405 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/3984ce7b-96c8-464a-aa14-28487cc5b373-host-var-lib-cni-bin\") pod \"multus-tglvr\" (UID: \"3984ce7b-96c8-464a-aa14-28487cc5b373\") " pod="openshift-multus/multus-tglvr" Nov 04 02:32:34.766750 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:32:34.766522 4405 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/7cb29949-bfe4-47a2-bc17-115a8d4d895d-serviceca\") pod \"node-ca-2h96g\" (UID: \"7cb29949-bfe4-47a2-bc17-115a8d4d895d\") " pod="openshift-image-registry/node-ca-2h96g" Nov 04 02:32:34.766750 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:32:34.766557 4405 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8ll77\" (UniqueName: \"kubernetes.io/projected/8b030f29-dfa6-42b1-81a7-07b539ad9aa9-kube-api-access-8ll77\") pod \"node-resolver-kzzwk\" (UID: \"8b030f29-dfa6-42b1-81a7-07b539ad9aa9\") " pod="openshift-dns/node-resolver-kzzwk" Nov 04 02:32:34.766750 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:32:34.766574 4405 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/5590efb1-88b8-41ad-b0ee-37b2ff9442de-system-cni-dir\") pod \"multus-additional-cni-plugins-sgx95\" (UID: \"5590efb1-88b8-41ad-b0ee-37b2ff9442de\") " pod="openshift-multus/multus-additional-cni-plugins-sgx95" Nov 04 02:32:34.766750 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:32:34.766615 4405 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/a245d359-742e-489d-9c58-eea4e686286c-cert\") pod \"ingress-canary-mn7dn\" (UID: \"a245d359-742e-489d-9c58-eea4e686286c\") " pod="openshift-ingress-canary/ingress-canary-mn7dn" Nov 04 02:32:34.766750 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:32:34.766631 4405 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/3a7ab1db-ad88-40d9-a7e9-b511bbd4584c-dbus\") pod \"global-pull-secret-syncer-krvqk\" (UID: \"3a7ab1db-ad88-40d9-a7e9-b511bbd4584c\") " pod="kube-system/global-pull-secret-syncer-krvqk" Nov 04 02:32:34.766750 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:32:34.766649 4405 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-stzcf\" (UniqueName: \"kubernetes.io/projected/524e33f8-723b-4868-a4f5-5d7eb92a5fe3-kube-api-access-stzcf\") pod \"iptables-alerter-sx4w7\" (UID: \"524e33f8-723b-4868-a4f5-5d7eb92a5fe3\") " pod="openshift-network-operator/iptables-alerter-sx4w7" Nov 04 02:32:34.766750 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:32:34.766732 4405 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/cb1915a0-5505-4f32-8f79-2f638465e404-etc-sysctl-conf\") pod \"tuned-ncfkk\" (UID: \"cb1915a0-5505-4f32-8f79-2f638465e404\") " pod="openshift-cluster-node-tuning-operator/tuned-ncfkk" Nov 04 02:32:34.767314 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:32:34.766786 4405 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/ffbb2d83-5a93-4a5b-bce3-e40afb8f58ea-node-log\") pod \"ovnkube-node-vwcx8\" (UID: \"ffbb2d83-5a93-4a5b-bce3-e40afb8f58ea\") " pod="openshift-ovn-kubernetes/ovnkube-node-vwcx8" Nov 04 02:32:34.767314 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:32:34.766815 4405 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/3984ce7b-96c8-464a-aa14-28487cc5b373-host-run-multus-certs\") pod \"multus-tglvr\" (UID: \"3984ce7b-96c8-464a-aa14-28487cc5b373\") " pod="openshift-multus/multus-tglvr" Nov 04 02:32:34.767314 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:32:34.766832 4405 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/f58e4ac8-c132-4716-b6c0-8d8ddb9c19e5-device-dir\") pod \"aws-ebs-csi-driver-node-6mvxj\" (UID: \"f58e4ac8-c132-4716-b6c0-8d8ddb9c19e5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6mvxj" Nov 04 02:32:34.767314 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:32:34.766858 4405 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b4snb\" (UniqueName: \"kubernetes.io/projected/f58e4ac8-c132-4716-b6c0-8d8ddb9c19e5-kube-api-access-b4snb\") pod \"aws-ebs-csi-driver-node-6mvxj\" (UID: \"f58e4ac8-c132-4716-b6c0-8d8ddb9c19e5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6mvxj" Nov 04 02:32:34.767314 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:32:34.766876 4405 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/cb1915a0-5505-4f32-8f79-2f638465e404-host\") pod \"tuned-ncfkk\" (UID: \"cb1915a0-5505-4f32-8f79-2f638465e404\") " pod="openshift-cluster-node-tuning-operator/tuned-ncfkk" Nov 04 02:32:34.767314 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:32:34.766944 4405 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/ffbb2d83-5a93-4a5b-bce3-e40afb8f58ea-host-cni-netd\") pod \"ovnkube-node-vwcx8\" (UID: \"ffbb2d83-5a93-4a5b-bce3-e40afb8f58ea\") " pod="openshift-ovn-kubernetes/ovnkube-node-vwcx8" Nov 04 02:32:34.767314 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:32:34.766967 4405 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/4acab1fc-c591-475a-a15f-790be8131547-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-5f2vb\" (UID: \"4acab1fc-c591-475a-a15f-790be8131547\") " pod="openshift-insights/insights-runtime-extractor-5f2vb" Nov 04 02:32:34.767314 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:32:34.766989 4405 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/3984ce7b-96c8-464a-aa14-28487cc5b373-host-run-k8s-cni-cncf-io\") pod \"multus-tglvr\" (UID: \"3984ce7b-96c8-464a-aa14-28487cc5b373\") " pod="openshift-multus/multus-tglvr" Nov 04 02:32:34.767314 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:32:34.767019 4405 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/4f76b032-61eb-4796-b082-47c23ff73d18-config-volume\") pod \"dns-default-wbzfc\" (UID: \"4f76b032-61eb-4796-b082-47c23ff73d18\") " pod="openshift-dns/dns-default-wbzfc" Nov 04 02:32:34.767314 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:32:34.767045 4405 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/ffbb2d83-5a93-4a5b-bce3-e40afb8f58ea-host-run-netns\") pod \"ovnkube-node-vwcx8\" (UID: \"ffbb2d83-5a93-4a5b-bce3-e40afb8f58ea\") " pod="openshift-ovn-kubernetes/ovnkube-node-vwcx8" Nov 04 02:32:34.767314 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:32:34.767060 4405 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/ffbb2d83-5a93-4a5b-bce3-e40afb8f58ea-host-cni-bin\") pod \"ovnkube-node-vwcx8\" (UID: \"ffbb2d83-5a93-4a5b-bce3-e40afb8f58ea\") " pod="openshift-ovn-kubernetes/ovnkube-node-vwcx8" Nov 04 02:32:34.767314 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:32:34.767080 4405 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/3984ce7b-96c8-464a-aa14-28487cc5b373-multus-conf-dir\") pod \"multus-tglvr\" (UID: \"3984ce7b-96c8-464a-aa14-28487cc5b373\") " pod="openshift-multus/multus-tglvr" Nov 04 02:32:34.767314 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:32:34.767136 4405 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/7cb29949-bfe4-47a2-bc17-115a8d4d895d-host\") pod \"node-ca-2h96g\" (UID: \"7cb29949-bfe4-47a2-bc17-115a8d4d895d\") " pod="openshift-image-registry/node-ca-2h96g" Nov 04 02:32:34.767314 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:32:34.767168 4405 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/524e33f8-723b-4868-a4f5-5d7eb92a5fe3-iptables-alerter-script\") pod \"iptables-alerter-sx4w7\" (UID: \"524e33f8-723b-4868-a4f5-5d7eb92a5fe3\") " pod="openshift-network-operator/iptables-alerter-sx4w7" Nov 04 02:32:34.767314 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:32:34.767186 4405 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/5b77e30b-cb0c-49c1-80b9-330a138dbfb7-konnectivity-ca\") pod \"konnectivity-agent-wcp7z\" (UID: \"5b77e30b-cb0c-49c1-80b9-330a138dbfb7\") " pod="kube-system/konnectivity-agent-wcp7z" Nov 04 02:32:34.767314 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:32:34.767212 4405 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/3984ce7b-96c8-464a-aa14-28487cc5b373-multus-cni-dir\") pod \"multus-tglvr\" (UID: \"3984ce7b-96c8-464a-aa14-28487cc5b373\") " pod="openshift-multus/multus-tglvr" Nov 04 02:32:34.791524 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:32:34.791495 4405 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Nov 04 02:32:34.791612 ip-10-0-134-125 kubenswrapper[4405]: E1104 02:32:34.791549 4405 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-apiserver-proxy-ip-10-0-134-125.us-west-1.compute.internal\" already exists" pod="kube-system/kube-apiserver-proxy-ip-10-0-134-125.us-west-1.compute.internal" Nov 04 02:32:34.792015 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:32:34.792001 4405 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Nov 04 02:32:34.792052 ip-10-0-134-125 kubenswrapper[4405]: E1104 02:32:34.792037 4405 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-rbac-proxy-crio-ip-10-0-134-125.us-west-1.compute.internal\" already exists" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-125.us-west-1.compute.internal" Nov 04 02:32:34.802585 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:32:34.802555 4405 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-82bnw\"" Nov 04 02:32:34.802802 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:32:34.802783 4405 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Nov 04 02:32:34.802886 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:32:34.802801 4405 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Nov 04 02:32:34.802886 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:32:34.802805 4405 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Nov 04 02:32:34.802886 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:32:34.802806 4405 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Nov 04 02:32:34.802886 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:32:34.802782 4405 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Nov 04 02:32:34.804862 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:32:34.804387 4405 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-gjd4x\"" Nov 04 02:32:34.804862 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:32:34.804505 4405 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/3a7ab1db-ad88-40d9-a7e9-b511bbd4584c-original-pull-secret\") pod \"global-pull-secret-syncer-krvqk\" (UID: \"3a7ab1db-ad88-40d9-a7e9-b511bbd4584c\") " pod="kube-system/global-pull-secret-syncer-krvqk" Nov 04 02:32:34.805097 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:32:34.804956 4405 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-9fmkz\"" Nov 04 02:32:34.807536 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:32:34.807517 4405 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/3984ce7b-96c8-464a-aa14-28487cc5b373-multus-daemon-config\") pod \"multus-tglvr\" (UID: \"3984ce7b-96c8-464a-aa14-28487cc5b373\") " pod="openshift-multus/multus-tglvr" Nov 04 02:32:34.842153 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:32:34.842121 4405 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n6mc7\" (UniqueName: \"kubernetes.io/projected/cb1915a0-5505-4f32-8f79-2f638465e404-kube-api-access-n6mc7\") pod \"tuned-ncfkk\" (UID: \"cb1915a0-5505-4f32-8f79-2f638465e404\") " pod="openshift-cluster-node-tuning-operator/tuned-ncfkk" Nov 04 02:32:34.867970 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:32:34.867940 4405 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/ffbb2d83-5a93-4a5b-bce3-e40afb8f58ea-run-systemd\") pod \"ovnkube-node-vwcx8\" (UID: \"ffbb2d83-5a93-4a5b-bce3-e40afb8f58ea\") " pod="openshift-ovn-kubernetes/ovnkube-node-vwcx8" Nov 04 02:32:34.867970 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:32:34.867973 4405 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/ffbb2d83-5a93-4a5b-bce3-e40afb8f58ea-var-lib-openvswitch\") pod \"ovnkube-node-vwcx8\" (UID: \"ffbb2d83-5a93-4a5b-bce3-e40afb8f58ea\") " pod="openshift-ovn-kubernetes/ovnkube-node-vwcx8" Nov 04 02:32:34.868143 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:32:34.868000 4405 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/cb1915a0-5505-4f32-8f79-2f638465e404-etc-sysconfig\") pod \"tuned-ncfkk\" (UID: \"cb1915a0-5505-4f32-8f79-2f638465e404\") " pod="openshift-cluster-node-tuning-operator/tuned-ncfkk" Nov 04 02:32:34.868143 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:32:34.868027 4405 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/ffbb2d83-5a93-4a5b-bce3-e40afb8f58ea-run-ovn\") pod \"ovnkube-node-vwcx8\" (UID: \"ffbb2d83-5a93-4a5b-bce3-e40afb8f58ea\") " pod="openshift-ovn-kubernetes/ovnkube-node-vwcx8" Nov 04 02:32:34.868143 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:32:34.868043 4405 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/ffbb2d83-5a93-4a5b-bce3-e40afb8f58ea-run-systemd\") pod \"ovnkube-node-vwcx8\" (UID: \"ffbb2d83-5a93-4a5b-bce3-e40afb8f58ea\") " pod="openshift-ovn-kubernetes/ovnkube-node-vwcx8" Nov 04 02:32:34.868143 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:32:34.868056 4405 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/5590efb1-88b8-41ad-b0ee-37b2ff9442de-tuning-conf-dir\") pod \"multus-additional-cni-plugins-sgx95\" (UID: \"5590efb1-88b8-41ad-b0ee-37b2ff9442de\") " pod="openshift-multus/multus-additional-cni-plugins-sgx95" Nov 04 02:32:34.868143 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:32:34.868083 4405 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/4f76b032-61eb-4796-b082-47c23ff73d18-metrics-tls\") pod \"dns-default-wbzfc\" (UID: \"4f76b032-61eb-4796-b082-47c23ff73d18\") " pod="openshift-dns/dns-default-wbzfc" Nov 04 02:32:34.868143 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:32:34.868091 4405 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/cb1915a0-5505-4f32-8f79-2f638465e404-etc-sysconfig\") pod \"tuned-ncfkk\" (UID: \"cb1915a0-5505-4f32-8f79-2f638465e404\") " pod="openshift-cluster-node-tuning-operator/tuned-ncfkk" Nov 04 02:32:34.868143 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:32:34.868091 4405 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/ffbb2d83-5a93-4a5b-bce3-e40afb8f58ea-var-lib-openvswitch\") pod \"ovnkube-node-vwcx8\" (UID: \"ffbb2d83-5a93-4a5b-bce3-e40afb8f58ea\") " pod="openshift-ovn-kubernetes/ovnkube-node-vwcx8" Nov 04 02:32:34.868143 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:32:34.868101 4405 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/ffbb2d83-5a93-4a5b-bce3-e40afb8f58ea-run-ovn\") pod \"ovnkube-node-vwcx8\" (UID: \"ffbb2d83-5a93-4a5b-bce3-e40afb8f58ea\") " pod="openshift-ovn-kubernetes/ovnkube-node-vwcx8" Nov 04 02:32:34.868143 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:32:34.868109 4405 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/4acab1fc-c591-475a-a15f-790be8131547-crio-socket\") pod \"insights-runtime-extractor-5f2vb\" (UID: \"4acab1fc-c591-475a-a15f-790be8131547\") " pod="openshift-insights/insights-runtime-extractor-5f2vb" Nov 04 02:32:34.868143 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:32:34.868145 4405 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/4acab1fc-c591-475a-a15f-790be8131547-crio-socket\") pod \"insights-runtime-extractor-5f2vb\" (UID: \"4acab1fc-c591-475a-a15f-790be8131547\") " pod="openshift-insights/insights-runtime-extractor-5f2vb" Nov 04 02:32:34.868487 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:32:34.868148 4405 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/5590efb1-88b8-41ad-b0ee-37b2ff9442de-tuning-conf-dir\") pod \"multus-additional-cni-plugins-sgx95\" (UID: \"5590efb1-88b8-41ad-b0ee-37b2ff9442de\") " pod="openshift-multus/multus-additional-cni-plugins-sgx95" Nov 04 02:32:34.868487 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:32:34.868174 4405 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/3984ce7b-96c8-464a-aa14-28487cc5b373-host-var-lib-cni-multus\") pod \"multus-tglvr\" (UID: \"3984ce7b-96c8-464a-aa14-28487cc5b373\") " pod="openshift-multus/multus-tglvr" Nov 04 02:32:34.868487 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:32:34.868209 4405 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/3984ce7b-96c8-464a-aa14-28487cc5b373-host-var-lib-cni-multus\") pod \"multus-tglvr\" (UID: \"3984ce7b-96c8-464a-aa14-28487cc5b373\") " pod="openshift-multus/multus-tglvr" Nov 04 02:32:34.868487 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:32:34.868213 4405 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/cb1915a0-5505-4f32-8f79-2f638465e404-lib-modules\") pod \"tuned-ncfkk\" (UID: \"cb1915a0-5505-4f32-8f79-2f638465e404\") " pod="openshift-cluster-node-tuning-operator/tuned-ncfkk" Nov 04 02:32:34.868487 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:32:34.868241 4405 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/ffbb2d83-5a93-4a5b-bce3-e40afb8f58ea-host-kubelet\") pod \"ovnkube-node-vwcx8\" (UID: \"ffbb2d83-5a93-4a5b-bce3-e40afb8f58ea\") " pod="openshift-ovn-kubernetes/ovnkube-node-vwcx8" Nov 04 02:32:34.868487 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:32:34.868257 4405 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/8b030f29-dfa6-42b1-81a7-07b539ad9aa9-hosts-file\") pod \"node-resolver-kzzwk\" (UID: \"8b030f29-dfa6-42b1-81a7-07b539ad9aa9\") " pod="openshift-dns/node-resolver-kzzwk" Nov 04 02:32:34.868487 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:32:34.868273 4405 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ffbb2d83-5a93-4a5b-bce3-e40afb8f58ea-host-run-ovn-kubernetes\") pod \"ovnkube-node-vwcx8\" (UID: \"ffbb2d83-5a93-4a5b-bce3-e40afb8f58ea\") " pod="openshift-ovn-kubernetes/ovnkube-node-vwcx8" Nov 04 02:32:34.868487 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:32:34.868288 4405 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/cb1915a0-5505-4f32-8f79-2f638465e404-etc-systemd\") pod \"tuned-ncfkk\" (UID: \"cb1915a0-5505-4f32-8f79-2f638465e404\") " pod="openshift-cluster-node-tuning-operator/tuned-ncfkk" Nov 04 02:32:34.868487 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:32:34.868306 4405 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/ffbb2d83-5a93-4a5b-bce3-e40afb8f58ea-run-openvswitch\") pod \"ovnkube-node-vwcx8\" (UID: \"ffbb2d83-5a93-4a5b-bce3-e40afb8f58ea\") " pod="openshift-ovn-kubernetes/ovnkube-node-vwcx8" Nov 04 02:32:34.868487 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:32:34.868310 4405 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/cb1915a0-5505-4f32-8f79-2f638465e404-lib-modules\") pod \"tuned-ncfkk\" (UID: \"cb1915a0-5505-4f32-8f79-2f638465e404\") " pod="openshift-cluster-node-tuning-operator/tuned-ncfkk" Nov 04 02:32:34.868487 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:32:34.868340 4405 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/ffbb2d83-5a93-4a5b-bce3-e40afb8f58ea-log-socket\") pod \"ovnkube-node-vwcx8\" (UID: \"ffbb2d83-5a93-4a5b-bce3-e40afb8f58ea\") " pod="openshift-ovn-kubernetes/ovnkube-node-vwcx8" Nov 04 02:32:34.868487 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:32:34.868350 4405 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/ffbb2d83-5a93-4a5b-bce3-e40afb8f58ea-host-kubelet\") pod \"ovnkube-node-vwcx8\" (UID: \"ffbb2d83-5a93-4a5b-bce3-e40afb8f58ea\") " pod="openshift-ovn-kubernetes/ovnkube-node-vwcx8" Nov 04 02:32:34.868487 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:32:34.868363 4405 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/ffbb2d83-5a93-4a5b-bce3-e40afb8f58ea-run-openvswitch\") pod \"ovnkube-node-vwcx8\" (UID: \"ffbb2d83-5a93-4a5b-bce3-e40afb8f58ea\") " pod="openshift-ovn-kubernetes/ovnkube-node-vwcx8" Nov 04 02:32:34.868487 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:32:34.868367 4405 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/3984ce7b-96c8-464a-aa14-28487cc5b373-host-run-netns\") pod \"multus-tglvr\" (UID: \"3984ce7b-96c8-464a-aa14-28487cc5b373\") " pod="openshift-multus/multus-tglvr" Nov 04 02:32:34.868487 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:32:34.868372 4405 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/ffbb2d83-5a93-4a5b-bce3-e40afb8f58ea-log-socket\") pod \"ovnkube-node-vwcx8\" (UID: \"ffbb2d83-5a93-4a5b-bce3-e40afb8f58ea\") " pod="openshift-ovn-kubernetes/ovnkube-node-vwcx8" Nov 04 02:32:34.868487 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:32:34.868342 4405 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/8b030f29-dfa6-42b1-81a7-07b539ad9aa9-hosts-file\") pod \"node-resolver-kzzwk\" (UID: \"8b030f29-dfa6-42b1-81a7-07b539ad9aa9\") " pod="openshift-dns/node-resolver-kzzwk" Nov 04 02:32:34.868487 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:32:34.868388 4405 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ffbb2d83-5a93-4a5b-bce3-e40afb8f58ea-host-run-ovn-kubernetes\") pod \"ovnkube-node-vwcx8\" (UID: \"ffbb2d83-5a93-4a5b-bce3-e40afb8f58ea\") " pod="openshift-ovn-kubernetes/ovnkube-node-vwcx8" Nov 04 02:32:34.868487 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:32:34.868391 4405 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/cb1915a0-5505-4f32-8f79-2f638465e404-etc-systemd\") pod \"tuned-ncfkk\" (UID: \"cb1915a0-5505-4f32-8f79-2f638465e404\") " pod="openshift-cluster-node-tuning-operator/tuned-ncfkk" Nov 04 02:32:34.868487 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:32:34.868408 4405 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/3984ce7b-96c8-464a-aa14-28487cc5b373-host-run-netns\") pod \"multus-tglvr\" (UID: \"3984ce7b-96c8-464a-aa14-28487cc5b373\") " pod="openshift-multus/multus-tglvr" Nov 04 02:32:34.868487 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:32:34.868398 4405 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b37e5c97-77c6-4bb4-a9e4-e1ae27484457-metrics-certs\") pod \"network-metrics-daemon-bdcds\" (UID: \"b37e5c97-77c6-4bb4-a9e4-e1ae27484457\") " pod="openshift-multus/network-metrics-daemon-bdcds" Nov 04 02:32:34.868487 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:32:34.868449 4405 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/3a7ab1db-ad88-40d9-a7e9-b511bbd4584c-kubelet-config\") pod \"global-pull-secret-syncer-krvqk\" (UID: \"3a7ab1db-ad88-40d9-a7e9-b511bbd4584c\") " pod="kube-system/global-pull-secret-syncer-krvqk" Nov 04 02:32:34.868487 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:32:34.868478 4405 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/ffbb2d83-5a93-4a5b-bce3-e40afb8f58ea-host-slash\") pod \"ovnkube-node-vwcx8\" (UID: \"ffbb2d83-5a93-4a5b-bce3-e40afb8f58ea\") " pod="openshift-ovn-kubernetes/ovnkube-node-vwcx8" Nov 04 02:32:34.869399 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:32:34.868503 4405 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/3984ce7b-96c8-464a-aa14-28487cc5b373-host-var-lib-cni-bin\") pod \"multus-tglvr\" (UID: \"3984ce7b-96c8-464a-aa14-28487cc5b373\") " pod="openshift-multus/multus-tglvr" Nov 04 02:32:34.869399 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:32:34.868534 4405 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/5590efb1-88b8-41ad-b0ee-37b2ff9442de-system-cni-dir\") pod \"multus-additional-cni-plugins-sgx95\" (UID: \"5590efb1-88b8-41ad-b0ee-37b2ff9442de\") " pod="openshift-multus/multus-additional-cni-plugins-sgx95" Nov 04 02:32:34.869399 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:32:34.868558 4405 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/3a7ab1db-ad88-40d9-a7e9-b511bbd4584c-kubelet-config\") pod \"global-pull-secret-syncer-krvqk\" (UID: \"3a7ab1db-ad88-40d9-a7e9-b511bbd4584c\") " pod="kube-system/global-pull-secret-syncer-krvqk" Nov 04 02:32:34.869399 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:32:34.868564 4405 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/3984ce7b-96c8-464a-aa14-28487cc5b373-host-var-lib-cni-bin\") pod \"multus-tglvr\" (UID: \"3984ce7b-96c8-464a-aa14-28487cc5b373\") " pod="openshift-multus/multus-tglvr" Nov 04 02:32:34.869399 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:32:34.868566 4405 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/3a7ab1db-ad88-40d9-a7e9-b511bbd4584c-dbus\") pod \"global-pull-secret-syncer-krvqk\" (UID: \"3a7ab1db-ad88-40d9-a7e9-b511bbd4584c\") " pod="kube-system/global-pull-secret-syncer-krvqk" Nov 04 02:32:34.869399 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:32:34.868587 4405 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/5590efb1-88b8-41ad-b0ee-37b2ff9442de-system-cni-dir\") pod \"multus-additional-cni-plugins-sgx95\" (UID: \"5590efb1-88b8-41ad-b0ee-37b2ff9442de\") " pod="openshift-multus/multus-additional-cni-plugins-sgx95" Nov 04 02:32:34.869399 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:32:34.868591 4405 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/cb1915a0-5505-4f32-8f79-2f638465e404-etc-sysctl-conf\") pod \"tuned-ncfkk\" (UID: \"cb1915a0-5505-4f32-8f79-2f638465e404\") " pod="openshift-cluster-node-tuning-operator/tuned-ncfkk" Nov 04 02:32:34.869399 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:32:34.868559 4405 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/ffbb2d83-5a93-4a5b-bce3-e40afb8f58ea-host-slash\") pod \"ovnkube-node-vwcx8\" (UID: \"ffbb2d83-5a93-4a5b-bce3-e40afb8f58ea\") " pod="openshift-ovn-kubernetes/ovnkube-node-vwcx8" Nov 04 02:32:34.869399 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:32:34.868606 4405 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/ffbb2d83-5a93-4a5b-bce3-e40afb8f58ea-node-log\") pod \"ovnkube-node-vwcx8\" (UID: \"ffbb2d83-5a93-4a5b-bce3-e40afb8f58ea\") " pod="openshift-ovn-kubernetes/ovnkube-node-vwcx8" Nov 04 02:32:34.869399 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:32:34.868623 4405 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/3984ce7b-96c8-464a-aa14-28487cc5b373-host-run-multus-certs\") pod \"multus-tglvr\" (UID: \"3984ce7b-96c8-464a-aa14-28487cc5b373\") " pod="openshift-multus/multus-tglvr" Nov 04 02:32:34.869399 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:32:34.868624 4405 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/3a7ab1db-ad88-40d9-a7e9-b511bbd4584c-dbus\") pod \"global-pull-secret-syncer-krvqk\" (UID: \"3a7ab1db-ad88-40d9-a7e9-b511bbd4584c\") " pod="kube-system/global-pull-secret-syncer-krvqk" Nov 04 02:32:34.869399 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:32:34.868639 4405 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/f58e4ac8-c132-4716-b6c0-8d8ddb9c19e5-device-dir\") pod \"aws-ebs-csi-driver-node-6mvxj\" (UID: \"f58e4ac8-c132-4716-b6c0-8d8ddb9c19e5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6mvxj" Nov 04 02:32:34.869399 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:32:34.868646 4405 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/ffbb2d83-5a93-4a5b-bce3-e40afb8f58ea-node-log\") pod \"ovnkube-node-vwcx8\" (UID: \"ffbb2d83-5a93-4a5b-bce3-e40afb8f58ea\") " pod="openshift-ovn-kubernetes/ovnkube-node-vwcx8" Nov 04 02:32:34.869399 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:32:34.868691 4405 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/cb1915a0-5505-4f32-8f79-2f638465e404-host\") pod \"tuned-ncfkk\" (UID: \"cb1915a0-5505-4f32-8f79-2f638465e404\") " pod="openshift-cluster-node-tuning-operator/tuned-ncfkk" Nov 04 02:32:34.869399 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:32:34.868695 4405 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/3984ce7b-96c8-464a-aa14-28487cc5b373-host-run-multus-certs\") pod \"multus-tglvr\" (UID: \"3984ce7b-96c8-464a-aa14-28487cc5b373\") " pod="openshift-multus/multus-tglvr" Nov 04 02:32:34.869399 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:32:34.868696 4405 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/f58e4ac8-c132-4716-b6c0-8d8ddb9c19e5-device-dir\") pod \"aws-ebs-csi-driver-node-6mvxj\" (UID: \"f58e4ac8-c132-4716-b6c0-8d8ddb9c19e5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6mvxj" Nov 04 02:32:34.869399 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:32:34.868715 4405 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/ffbb2d83-5a93-4a5b-bce3-e40afb8f58ea-host-cni-netd\") pod \"ovnkube-node-vwcx8\" (UID: \"ffbb2d83-5a93-4a5b-bce3-e40afb8f58ea\") " pod="openshift-ovn-kubernetes/ovnkube-node-vwcx8" Nov 04 02:32:34.869399 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:32:34.868732 4405 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/cb1915a0-5505-4f32-8f79-2f638465e404-etc-sysctl-conf\") pod \"tuned-ncfkk\" (UID: \"cb1915a0-5505-4f32-8f79-2f638465e404\") " pod="openshift-cluster-node-tuning-operator/tuned-ncfkk" Nov 04 02:32:34.869399 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:32:34.868744 4405 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/4acab1fc-c591-475a-a15f-790be8131547-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-5f2vb\" (UID: \"4acab1fc-c591-475a-a15f-790be8131547\") " pod="openshift-insights/insights-runtime-extractor-5f2vb" Nov 04 02:32:34.869399 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:32:34.868752 4405 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/cb1915a0-5505-4f32-8f79-2f638465e404-host\") pod \"tuned-ncfkk\" (UID: \"cb1915a0-5505-4f32-8f79-2f638465e404\") " pod="openshift-cluster-node-tuning-operator/tuned-ncfkk" Nov 04 02:32:34.869399 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:32:34.868771 4405 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/ffbb2d83-5a93-4a5b-bce3-e40afb8f58ea-host-cni-netd\") pod \"ovnkube-node-vwcx8\" (UID: \"ffbb2d83-5a93-4a5b-bce3-e40afb8f58ea\") " pod="openshift-ovn-kubernetes/ovnkube-node-vwcx8" Nov 04 02:32:34.869399 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:32:34.868772 4405 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/3984ce7b-96c8-464a-aa14-28487cc5b373-host-run-k8s-cni-cncf-io\") pod \"multus-tglvr\" (UID: \"3984ce7b-96c8-464a-aa14-28487cc5b373\") " pod="openshift-multus/multus-tglvr" Nov 04 02:32:34.869399 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:32:34.868803 4405 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/3984ce7b-96c8-464a-aa14-28487cc5b373-host-run-k8s-cni-cncf-io\") pod \"multus-tglvr\" (UID: \"3984ce7b-96c8-464a-aa14-28487cc5b373\") " pod="openshift-multus/multus-tglvr" Nov 04 02:32:34.869399 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:32:34.868804 4405 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/ffbb2d83-5a93-4a5b-bce3-e40afb8f58ea-host-run-netns\") pod \"ovnkube-node-vwcx8\" (UID: \"ffbb2d83-5a93-4a5b-bce3-e40afb8f58ea\") " pod="openshift-ovn-kubernetes/ovnkube-node-vwcx8" Nov 04 02:32:34.869399 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:32:34.868833 4405 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/ffbb2d83-5a93-4a5b-bce3-e40afb8f58ea-host-run-netns\") pod \"ovnkube-node-vwcx8\" (UID: \"ffbb2d83-5a93-4a5b-bce3-e40afb8f58ea\") " pod="openshift-ovn-kubernetes/ovnkube-node-vwcx8" Nov 04 02:32:34.869399 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:32:34.868835 4405 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/ffbb2d83-5a93-4a5b-bce3-e40afb8f58ea-host-cni-bin\") pod \"ovnkube-node-vwcx8\" (UID: \"ffbb2d83-5a93-4a5b-bce3-e40afb8f58ea\") " pod="openshift-ovn-kubernetes/ovnkube-node-vwcx8" Nov 04 02:32:34.869399 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:32:34.868863 4405 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/3984ce7b-96c8-464a-aa14-28487cc5b373-multus-conf-dir\") pod \"multus-tglvr\" (UID: \"3984ce7b-96c8-464a-aa14-28487cc5b373\") " pod="openshift-multus/multus-tglvr" Nov 04 02:32:34.869399 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:32:34.868863 4405 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/ffbb2d83-5a93-4a5b-bce3-e40afb8f58ea-host-cni-bin\") pod \"ovnkube-node-vwcx8\" (UID: \"ffbb2d83-5a93-4a5b-bce3-e40afb8f58ea\") " pod="openshift-ovn-kubernetes/ovnkube-node-vwcx8" Nov 04 02:32:34.869399 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:32:34.868895 4405 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/7cb29949-bfe4-47a2-bc17-115a8d4d895d-host\") pod \"node-ca-2h96g\" (UID: \"7cb29949-bfe4-47a2-bc17-115a8d4d895d\") " pod="openshift-image-registry/node-ca-2h96g" Nov 04 02:32:34.869399 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:32:34.868906 4405 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/3984ce7b-96c8-464a-aa14-28487cc5b373-multus-conf-dir\") pod \"multus-tglvr\" (UID: \"3984ce7b-96c8-464a-aa14-28487cc5b373\") " pod="openshift-multus/multus-tglvr" Nov 04 02:32:34.869399 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:32:34.868928 4405 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/3984ce7b-96c8-464a-aa14-28487cc5b373-multus-cni-dir\") pod \"multus-tglvr\" (UID: \"3984ce7b-96c8-464a-aa14-28487cc5b373\") " pod="openshift-multus/multus-tglvr" Nov 04 02:32:34.869399 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:32:34.868947 4405 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/3984ce7b-96c8-464a-aa14-28487cc5b373-multus-socket-dir-parent\") pod \"multus-tglvr\" (UID: \"3984ce7b-96c8-464a-aa14-28487cc5b373\") " pod="openshift-multus/multus-tglvr" Nov 04 02:32:34.869399 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:32:34.868963 4405 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/f58e4ac8-c132-4716-b6c0-8d8ddb9c19e5-etc-selinux\") pod \"aws-ebs-csi-driver-node-6mvxj\" (UID: \"f58e4ac8-c132-4716-b6c0-8d8ddb9c19e5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6mvxj" Nov 04 02:32:34.869399 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:32:34.868930 4405 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/7cb29949-bfe4-47a2-bc17-115a8d4d895d-host\") pod \"node-ca-2h96g\" (UID: \"7cb29949-bfe4-47a2-bc17-115a8d4d895d\") " pod="openshift-image-registry/node-ca-2h96g" Nov 04 02:32:34.869399 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:32:34.868982 4405 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ffbb2d83-5a93-4a5b-bce3-e40afb8f58ea-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-vwcx8\" (UID: \"ffbb2d83-5a93-4a5b-bce3-e40afb8f58ea\") " pod="openshift-ovn-kubernetes/ovnkube-node-vwcx8" Nov 04 02:32:34.869399 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:32:34.869017 4405 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/3984ce7b-96c8-464a-aa14-28487cc5b373-etc-kubernetes\") pod \"multus-tglvr\" (UID: \"3984ce7b-96c8-464a-aa14-28487cc5b373\") " pod="openshift-multus/multus-tglvr" Nov 04 02:32:34.869399 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:32:34.869020 4405 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ffbb2d83-5a93-4a5b-bce3-e40afb8f58ea-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-vwcx8\" (UID: \"ffbb2d83-5a93-4a5b-bce3-e40afb8f58ea\") " pod="openshift-ovn-kubernetes/ovnkube-node-vwcx8" Nov 04 02:32:34.869399 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:32:34.869034 4405 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/3984ce7b-96c8-464a-aa14-28487cc5b373-multus-socket-dir-parent\") pod \"multus-tglvr\" (UID: \"3984ce7b-96c8-464a-aa14-28487cc5b373\") " pod="openshift-multus/multus-tglvr" Nov 04 02:32:34.869399 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:32:34.869047 4405 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/cb1915a0-5505-4f32-8f79-2f638465e404-var-lib-kubelet\") pod \"tuned-ncfkk\" (UID: \"cb1915a0-5505-4f32-8f79-2f638465e404\") " pod="openshift-cluster-node-tuning-operator/tuned-ncfkk" Nov 04 02:32:34.869399 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:32:34.869039 4405 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/3984ce7b-96c8-464a-aa14-28487cc5b373-multus-cni-dir\") pod \"multus-tglvr\" (UID: \"3984ce7b-96c8-464a-aa14-28487cc5b373\") " pod="openshift-multus/multus-tglvr" Nov 04 02:32:34.869399 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:32:34.869061 4405 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/f58e4ac8-c132-4716-b6c0-8d8ddb9c19e5-etc-selinux\") pod \"aws-ebs-csi-driver-node-6mvxj\" (UID: \"f58e4ac8-c132-4716-b6c0-8d8ddb9c19e5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6mvxj" Nov 04 02:32:34.869399 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:32:34.869056 4405 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/3984ce7b-96c8-464a-aa14-28487cc5b373-etc-kubernetes\") pod \"multus-tglvr\" (UID: \"3984ce7b-96c8-464a-aa14-28487cc5b373\") " pod="openshift-multus/multus-tglvr" Nov 04 02:32:34.869399 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:32:34.869066 4405 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/ffbb2d83-5a93-4a5b-bce3-e40afb8f58ea-etc-openvswitch\") pod \"ovnkube-node-vwcx8\" (UID: \"ffbb2d83-5a93-4a5b-bce3-e40afb8f58ea\") " pod="openshift-ovn-kubernetes/ovnkube-node-vwcx8" Nov 04 02:32:34.869399 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:32:34.869085 4405 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/ffbb2d83-5a93-4a5b-bce3-e40afb8f58ea-etc-openvswitch\") pod \"ovnkube-node-vwcx8\" (UID: \"ffbb2d83-5a93-4a5b-bce3-e40afb8f58ea\") " pod="openshift-ovn-kubernetes/ovnkube-node-vwcx8" Nov 04 02:32:34.869399 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:32:34.869108 4405 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/3984ce7b-96c8-464a-aa14-28487cc5b373-system-cni-dir\") pod \"multus-tglvr\" (UID: \"3984ce7b-96c8-464a-aa14-28487cc5b373\") " pod="openshift-multus/multus-tglvr" Nov 04 02:32:34.869399 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:32:34.869121 4405 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/cb1915a0-5505-4f32-8f79-2f638465e404-var-lib-kubelet\") pod \"tuned-ncfkk\" (UID: \"cb1915a0-5505-4f32-8f79-2f638465e404\") " pod="openshift-cluster-node-tuning-operator/tuned-ncfkk" Nov 04 02:32:34.869399 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:32:34.869133 4405 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/3984ce7b-96c8-464a-aa14-28487cc5b373-hostroot\") pod \"multus-tglvr\" (UID: \"3984ce7b-96c8-464a-aa14-28487cc5b373\") " pod="openshift-multus/multus-tglvr" Nov 04 02:32:34.869399 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:32:34.869159 4405 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/5590efb1-88b8-41ad-b0ee-37b2ff9442de-cnibin\") pod \"multus-additional-cni-plugins-sgx95\" (UID: \"5590efb1-88b8-41ad-b0ee-37b2ff9442de\") " pod="openshift-multus/multus-additional-cni-plugins-sgx95" Nov 04 02:32:34.869399 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:32:34.869137 4405 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/3984ce7b-96c8-464a-aa14-28487cc5b373-system-cni-dir\") pod \"multus-tglvr\" (UID: \"3984ce7b-96c8-464a-aa14-28487cc5b373\") " pod="openshift-multus/multus-tglvr" Nov 04 02:32:34.869399 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:32:34.869188 4405 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/3984ce7b-96c8-464a-aa14-28487cc5b373-hostroot\") pod \"multus-tglvr\" (UID: \"3984ce7b-96c8-464a-aa14-28487cc5b373\") " pod="openshift-multus/multus-tglvr" Nov 04 02:32:34.869399 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:32:34.869185 4405 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/cb1915a0-5505-4f32-8f79-2f638465e404-etc-kubernetes\") pod \"tuned-ncfkk\" (UID: \"cb1915a0-5505-4f32-8f79-2f638465e404\") " pod="openshift-cluster-node-tuning-operator/tuned-ncfkk" Nov 04 02:32:34.869399 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:32:34.869207 4405 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/5590efb1-88b8-41ad-b0ee-37b2ff9442de-cnibin\") pod \"multus-additional-cni-plugins-sgx95\" (UID: \"5590efb1-88b8-41ad-b0ee-37b2ff9442de\") " pod="openshift-multus/multus-additional-cni-plugins-sgx95" Nov 04 02:32:34.869399 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:32:34.869214 4405 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/ffbb2d83-5a93-4a5b-bce3-e40afb8f58ea-systemd-units\") pod \"ovnkube-node-vwcx8\" (UID: \"ffbb2d83-5a93-4a5b-bce3-e40afb8f58ea\") " pod="openshift-ovn-kubernetes/ovnkube-node-vwcx8" Nov 04 02:32:34.869399 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:32:34.869224 4405 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/cb1915a0-5505-4f32-8f79-2f638465e404-etc-kubernetes\") pod \"tuned-ncfkk\" (UID: \"cb1915a0-5505-4f32-8f79-2f638465e404\") " pod="openshift-cluster-node-tuning-operator/tuned-ncfkk" Nov 04 02:32:34.869399 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:32:34.869230 4405 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/f58e4ac8-c132-4716-b6c0-8d8ddb9c19e5-socket-dir\") pod \"aws-ebs-csi-driver-node-6mvxj\" (UID: \"f58e4ac8-c132-4716-b6c0-8d8ddb9c19e5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6mvxj" Nov 04 02:32:34.869399 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:32:34.869246 4405 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/f58e4ac8-c132-4716-b6c0-8d8ddb9c19e5-registration-dir\") pod \"aws-ebs-csi-driver-node-6mvxj\" (UID: \"f58e4ac8-c132-4716-b6c0-8d8ddb9c19e5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6mvxj" Nov 04 02:32:34.869399 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:32:34.869263 4405 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/f58e4ac8-c132-4716-b6c0-8d8ddb9c19e5-sys-fs\") pod \"aws-ebs-csi-driver-node-6mvxj\" (UID: \"f58e4ac8-c132-4716-b6c0-8d8ddb9c19e5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6mvxj" Nov 04 02:32:34.869399 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:32:34.869278 4405 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/cb1915a0-5505-4f32-8f79-2f638465e404-etc-sysctl-d\") pod \"tuned-ncfkk\" (UID: \"cb1915a0-5505-4f32-8f79-2f638465e404\") " pod="openshift-cluster-node-tuning-operator/tuned-ncfkk" Nov 04 02:32:34.869399 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:32:34.869277 4405 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/ffbb2d83-5a93-4a5b-bce3-e40afb8f58ea-systemd-units\") pod \"ovnkube-node-vwcx8\" (UID: \"ffbb2d83-5a93-4a5b-bce3-e40afb8f58ea\") " pod="openshift-ovn-kubernetes/ovnkube-node-vwcx8" Nov 04 02:32:34.869399 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:32:34.869291 4405 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/cb1915a0-5505-4f32-8f79-2f638465e404-sys\") pod \"tuned-ncfkk\" (UID: \"cb1915a0-5505-4f32-8f79-2f638465e404\") " pod="openshift-cluster-node-tuning-operator/tuned-ncfkk" Nov 04 02:32:34.869399 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:32:34.869307 4405 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/3984ce7b-96c8-464a-aa14-28487cc5b373-host-var-lib-kubelet\") pod \"multus-tglvr\" (UID: \"3984ce7b-96c8-464a-aa14-28487cc5b373\") " pod="openshift-multus/multus-tglvr" Nov 04 02:32:34.869399 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:32:34.869316 4405 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/f58e4ac8-c132-4716-b6c0-8d8ddb9c19e5-socket-dir\") pod \"aws-ebs-csi-driver-node-6mvxj\" (UID: \"f58e4ac8-c132-4716-b6c0-8d8ddb9c19e5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6mvxj" Nov 04 02:32:34.869399 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:32:34.869324 4405 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/f58e4ac8-c132-4716-b6c0-8d8ddb9c19e5-registration-dir\") pod \"aws-ebs-csi-driver-node-6mvxj\" (UID: \"f58e4ac8-c132-4716-b6c0-8d8ddb9c19e5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6mvxj" Nov 04 02:32:34.869399 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:32:34.869331 4405 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/f58e4ac8-c132-4716-b6c0-8d8ddb9c19e5-kubelet-dir\") pod \"aws-ebs-csi-driver-node-6mvxj\" (UID: \"f58e4ac8-c132-4716-b6c0-8d8ddb9c19e5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6mvxj" Nov 04 02:32:34.869399 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:32:34.869340 4405 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/f58e4ac8-c132-4716-b6c0-8d8ddb9c19e5-sys-fs\") pod \"aws-ebs-csi-driver-node-6mvxj\" (UID: \"f58e4ac8-c132-4716-b6c0-8d8ddb9c19e5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6mvxj" Nov 04 02:32:34.869399 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:32:34.869362 4405 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/cb1915a0-5505-4f32-8f79-2f638465e404-etc-modprobe-d\") pod \"tuned-ncfkk\" (UID: \"cb1915a0-5505-4f32-8f79-2f638465e404\") " pod="openshift-cluster-node-tuning-operator/tuned-ncfkk" Nov 04 02:32:34.869399 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:32:34.869369 4405 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/3984ce7b-96c8-464a-aa14-28487cc5b373-host-var-lib-kubelet\") pod \"multus-tglvr\" (UID: \"3984ce7b-96c8-464a-aa14-28487cc5b373\") " pod="openshift-multus/multus-tglvr" Nov 04 02:32:34.869399 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:32:34.869374 4405 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/f58e4ac8-c132-4716-b6c0-8d8ddb9c19e5-kubelet-dir\") pod \"aws-ebs-csi-driver-node-6mvxj\" (UID: \"f58e4ac8-c132-4716-b6c0-8d8ddb9c19e5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6mvxj" Nov 04 02:32:34.869399 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:32:34.869362 4405 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/cb1915a0-5505-4f32-8f79-2f638465e404-sys\") pod \"tuned-ncfkk\" (UID: \"cb1915a0-5505-4f32-8f79-2f638465e404\") " pod="openshift-cluster-node-tuning-operator/tuned-ncfkk" Nov 04 02:32:34.869399 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:32:34.869386 4405 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/cb1915a0-5505-4f32-8f79-2f638465e404-run\") pod \"tuned-ncfkk\" (UID: \"cb1915a0-5505-4f32-8f79-2f638465e404\") " pod="openshift-cluster-node-tuning-operator/tuned-ncfkk" Nov 04 02:32:34.869399 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:32:34.869408 4405 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/3984ce7b-96c8-464a-aa14-28487cc5b373-cnibin\") pod \"multus-tglvr\" (UID: \"3984ce7b-96c8-464a-aa14-28487cc5b373\") " pod="openshift-multus/multus-tglvr" Nov 04 02:32:34.869399 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:32:34.869421 4405 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/cb1915a0-5505-4f32-8f79-2f638465e404-run\") pod \"tuned-ncfkk\" (UID: \"cb1915a0-5505-4f32-8f79-2f638465e404\") " pod="openshift-cluster-node-tuning-operator/tuned-ncfkk" Nov 04 02:32:34.869399 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:32:34.869424 4405 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/cb1915a0-5505-4f32-8f79-2f638465e404-etc-sysctl-d\") pod \"tuned-ncfkk\" (UID: \"cb1915a0-5505-4f32-8f79-2f638465e404\") " pod="openshift-cluster-node-tuning-operator/tuned-ncfkk" Nov 04 02:32:34.869399 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:32:34.869431 4405 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/524e33f8-723b-4868-a4f5-5d7eb92a5fe3-host-slash\") pod \"iptables-alerter-sx4w7\" (UID: \"524e33f8-723b-4868-a4f5-5d7eb92a5fe3\") " pod="openshift-network-operator/iptables-alerter-sx4w7" Nov 04 02:32:34.871382 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:32:34.869451 4405 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/cb1915a0-5505-4f32-8f79-2f638465e404-etc-modprobe-d\") pod \"tuned-ncfkk\" (UID: \"cb1915a0-5505-4f32-8f79-2f638465e404\") " pod="openshift-cluster-node-tuning-operator/tuned-ncfkk" Nov 04 02:32:34.871382 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:32:34.869468 4405 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/3984ce7b-96c8-464a-aa14-28487cc5b373-cnibin\") pod \"multus-tglvr\" (UID: \"3984ce7b-96c8-464a-aa14-28487cc5b373\") " pod="openshift-multus/multus-tglvr" Nov 04 02:32:34.871382 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:32:34.869490 4405 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/524e33f8-723b-4868-a4f5-5d7eb92a5fe3-host-slash\") pod \"iptables-alerter-sx4w7\" (UID: \"524e33f8-723b-4868-a4f5-5d7eb92a5fe3\") " pod="openshift-network-operator/iptables-alerter-sx4w7" Nov 04 02:32:34.902596 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:32:34.902517 4405 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Nov 04 02:32:34.902756 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:32:34.902601 4405 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Nov 04 02:32:34.902756 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:32:34.902624 4405 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-fm8qv\"" Nov 04 02:32:34.902854 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:32:34.902844 4405 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-6tfxk\"" Nov 04 02:32:34.902906 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:32:34.902846 4405 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Nov 04 02:32:34.902906 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:32:34.902872 4405 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-6l7l7\"" Nov 04 02:32:34.902906 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:32:34.902889 4405 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Nov 04 02:32:34.903049 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:32:34.902993 4405 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Nov 04 02:32:34.903049 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:32:34.903018 4405 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-node-identity-dockercfg-ffw8m\"" Nov 04 02:32:34.903119 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:32:34.903053 4405 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-lg9c8\"" Nov 04 02:32:34.903405 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:32:34.903330 4405 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-vd7gf\"" Nov 04 02:32:34.903405 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:32:34.903346 4405 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"ingress-canary-dockercfg-l8p2p\"" Nov 04 02:32:34.903405 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:32:34.903390 4405 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Nov 04 02:32:34.903405 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:32:34.903390 4405 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-wt2vt\"" Nov 04 02:32:34.903601 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:32:34.903404 4405 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Nov 04 02:32:34.903601 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:32:34.903459 4405 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"operator-dockercfg-h7xc9\"" Nov 04 02:32:34.903601 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:32:34.903506 4405 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-4v2qx\"" Nov 04 02:32:34.903601 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:32:34.903521 4405 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-8zwfm\"" Nov 04 02:32:34.903601 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:32:34.903573 4405 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent-dockercfg-jmwv6\"" Nov 04 02:32:34.903601 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:32:34.903601 4405 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Nov 04 02:32:34.903887 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:32:34.903626 4405 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Nov 04 02:32:34.903887 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:32:34.903687 4405 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Nov 04 02:32:34.903887 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:32:34.903715 4405 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Nov 04 02:32:34.903887 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:32:34.903762 4405 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Nov 04 02:32:34.903887 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:32:34.903785 4405 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Nov 04 02:32:34.903887 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:32:34.903797 4405 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Nov 04 02:32:34.903887 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:32:34.903798 4405 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Nov 04 02:32:34.903887 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:32:34.903797 4405 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Nov 04 02:32:34.903887 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:32:34.903823 4405 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Nov 04 02:32:34.903887 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:32:34.903844 4405 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-rgzgm\"" Nov 04 02:32:34.903887 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:32:34.903846 4405 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Nov 04 02:32:34.903887 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:32:34.903846 4405 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"service-ca-bundle\"" Nov 04 02:32:34.903887 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:32:34.903868 4405 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Nov 04 02:32:34.903887 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:32:34.903846 4405 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Nov 04 02:32:34.904491 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:32:34.904029 4405 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Nov 04 02:32:34.904491 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:32:34.904075 4405 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Nov 04 02:32:34.904491 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:32:34.904084 4405 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Nov 04 02:32:34.904491 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:32:34.904090 4405 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Nov 04 02:32:34.904491 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:32:34.904112 4405 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Nov 04 02:32:34.904491 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:32:34.904129 4405 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Nov 04 02:32:34.904491 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:32:34.904129 4405 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Nov 04 02:32:34.904491 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:32:34.904148 4405 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Nov 04 02:32:34.904491 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:32:34.904152 4405 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Nov 04 02:32:34.904491 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:32:34.904159 4405 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Nov 04 02:32:34.904491 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:32:34.904150 4405 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Nov 04 02:32:34.904491 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:32:34.904130 4405 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Nov 04 02:32:34.904491 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:32:34.904184 4405 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Nov 04 02:32:34.904491 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:32:34.904279 4405 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/5590efb1-88b8-41ad-b0ee-37b2ff9442de-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-sgx95\" (UID: \"5590efb1-88b8-41ad-b0ee-37b2ff9442de\") " pod="openshift-multus/multus-additional-cni-plugins-sgx95" Nov 04 02:32:34.904491 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:32:34.904315 4405 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/ffbb2d83-5a93-4a5b-bce3-e40afb8f58ea-ovnkube-config\") pod \"ovnkube-node-vwcx8\" (UID: \"ffbb2d83-5a93-4a5b-bce3-e40afb8f58ea\") " pod="openshift-ovn-kubernetes/ovnkube-node-vwcx8" Nov 04 02:32:34.904491 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:32:34.904355 4405 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Nov 04 02:32:34.904491 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:32:34.904358 4405 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Nov 04 02:32:34.904491 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:32:34.904459 4405 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/ffbb2d83-5a93-4a5b-bce3-e40afb8f58ea-env-overrides\") pod \"ovnkube-node-vwcx8\" (UID: \"ffbb2d83-5a93-4a5b-bce3-e40afb8f58ea\") " pod="openshift-ovn-kubernetes/ovnkube-node-vwcx8" Nov 04 02:32:34.905024 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:32:34.904627 4405 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/5b77e30b-cb0c-49c1-80b9-330a138dbfb7-agent-certs\") pod \"konnectivity-agent-wcp7z\" (UID: \"5b77e30b-cb0c-49c1-80b9-330a138dbfb7\") " pod="kube-system/konnectivity-agent-wcp7z" Nov 04 02:32:34.905024 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:32:34.904634 4405 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/3984ce7b-96c8-464a-aa14-28487cc5b373-cni-binary-copy\") pod \"multus-tglvr\" (UID: \"3984ce7b-96c8-464a-aa14-28487cc5b373\") " pod="openshift-multus/multus-tglvr" Nov 04 02:32:34.905658 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:32:34.905640 4405 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/ffbb2d83-5a93-4a5b-bce3-e40afb8f58ea-ovnkube-script-lib\") pod \"ovnkube-node-vwcx8\" (UID: \"ffbb2d83-5a93-4a5b-bce3-e40afb8f58ea\") " pod="openshift-ovn-kubernetes/ovnkube-node-vwcx8" Nov 04 02:32:34.905782 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:32:34.905763 4405 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/4acab1fc-c591-475a-a15f-790be8131547-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-5f2vb\" (UID: \"4acab1fc-c591-475a-a15f-790be8131547\") " pod="openshift-insights/insights-runtime-extractor-5f2vb" Nov 04 02:32:34.906004 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:32:34.905987 4405 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-946v4\" (UniqueName: \"kubernetes.io/projected/7cb29949-bfe4-47a2-bc17-115a8d4d895d-kube-api-access-946v4\") pod \"node-ca-2h96g\" (UID: \"7cb29949-bfe4-47a2-bc17-115a8d4d895d\") " pod="openshift-image-registry/node-ca-2h96g" Nov 04 02:32:34.906066 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:32:34.905994 4405 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zpbsm\" (UniqueName: \"kubernetes.io/projected/b37e5c97-77c6-4bb4-a9e4-e1ae27484457-kube-api-access-zpbsm\") pod \"network-metrics-daemon-bdcds\" (UID: \"b37e5c97-77c6-4bb4-a9e4-e1ae27484457\") " pod="openshift-multus/network-metrics-daemon-bdcds" Nov 04 02:32:34.906243 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:32:34.906225 4405 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h288h\" (UniqueName: \"kubernetes.io/projected/5a5c4b25-7a35-4290-ad1e-04b4a3617877-kube-api-access-h288h\") pod \"periodic-gathering-hxwj2-xcshl\" (UID: \"5a5c4b25-7a35-4290-ad1e-04b4a3617877\") " pod="openshift-insights/periodic-gathering-hxwj2-xcshl" Nov 04 02:32:34.906320 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:32:34.906267 4405 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/5590efb1-88b8-41ad-b0ee-37b2ff9442de-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-sgx95\" (UID: \"5590efb1-88b8-41ad-b0ee-37b2ff9442de\") " pod="openshift-multus/multus-additional-cni-plugins-sgx95" Nov 04 02:32:34.906557 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:32:34.906537 4405 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5a5c4b25-7a35-4290-ad1e-04b4a3617877-service-ca-bundle\") pod \"periodic-gathering-hxwj2-xcshl\" (UID: \"5a5c4b25-7a35-4290-ad1e-04b4a3617877\") " pod="openshift-insights/periodic-gathering-hxwj2-xcshl" Nov 04 02:32:34.906639 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:32:34.906626 4405 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/ffbb2d83-5a93-4a5b-bce3-e40afb8f58ea-ovn-node-metrics-cert\") pod \"ovnkube-node-vwcx8\" (UID: \"ffbb2d83-5a93-4a5b-bce3-e40afb8f58ea\") " pod="openshift-ovn-kubernetes/ovnkube-node-vwcx8" Nov 04 02:32:34.906729 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:32:34.906708 4405 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tp4jl\" (UniqueName: \"kubernetes.io/projected/ffbb2d83-5a93-4a5b-bce3-e40afb8f58ea-kube-api-access-tp4jl\") pod \"ovnkube-node-vwcx8\" (UID: \"ffbb2d83-5a93-4a5b-bce3-e40afb8f58ea\") " pod="openshift-ovn-kubernetes/ovnkube-node-vwcx8" Nov 04 02:32:34.906782 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:32:34.906737 4405 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/7cb29949-bfe4-47a2-bc17-115a8d4d895d-serviceca\") pod \"node-ca-2h96g\" (UID: \"7cb29949-bfe4-47a2-bc17-115a8d4d895d\") " pod="openshift-image-registry/node-ca-2h96g" Nov 04 02:32:34.906902 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:32:34.906888 4405 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/a245d359-742e-489d-9c58-eea4e686286c-cert\") pod \"ingress-canary-mn7dn\" (UID: \"a245d359-742e-489d-9c58-eea4e686286c\") " pod="openshift-ingress-canary/ingress-canary-mn7dn" Nov 04 02:32:34.907656 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:32:34.907640 4405 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/5b77e30b-cb0c-49c1-80b9-330a138dbfb7-konnectivity-ca\") pod \"konnectivity-agent-wcp7z\" (UID: \"5b77e30b-cb0c-49c1-80b9-330a138dbfb7\") " pod="kube-system/konnectivity-agent-wcp7z" Nov 04 02:32:34.907741 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:32:34.907653 4405 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/4f76b032-61eb-4796-b082-47c23ff73d18-config-volume\") pod \"dns-default-wbzfc\" (UID: \"4f76b032-61eb-4796-b082-47c23ff73d18\") " pod="openshift-dns/dns-default-wbzfc" Nov 04 02:32:34.907786 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:32:34.907741 4405 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/524e33f8-723b-4868-a4f5-5d7eb92a5fe3-iptables-alerter-script\") pod \"iptables-alerter-sx4w7\" (UID: \"524e33f8-723b-4868-a4f5-5d7eb92a5fe3\") " pod="openshift-network-operator/iptables-alerter-sx4w7" Nov 04 02:32:34.908590 ip-10-0-134-125 kubenswrapper[4405]: E1104 02:32:34.908575 4405 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found Nov 04 02:32:34.908590 ip-10-0-134-125 kubenswrapper[4405]: E1104 02:32:34.908584 4405 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Nov 04 02:32:34.908713 ip-10-0-134-125 kubenswrapper[4405]: E1104 02:32:34.908694 4405 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4f76b032-61eb-4796-b082-47c23ff73d18-metrics-tls podName:4f76b032-61eb-4796-b082-47c23ff73d18 nodeName:}" failed. No retries permitted until 2026-11-04 02:32:35.408655641 +0000 UTC m=+1.839355767 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/4f76b032-61eb-4796-b082-47c23ff73d18-metrics-tls") pod "dns-default-wbzfc" (UID: "4f76b032-61eb-4796-b082-47c23ff73d18") : secret "dns-default-metrics-tls" not found Nov 04 02:32:34.908713 ip-10-0-134-125 kubenswrapper[4405]: E1104 02:32:34.908710 4405 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b37e5c97-77c6-4bb4-a9e4-e1ae27484457-metrics-certs podName:b37e5c97-77c6-4bb4-a9e4-e1ae27484457 nodeName:}" failed. No retries permitted until 2026-11-04 02:32:35.408703645 +0000 UTC m=+1.839403743 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/b37e5c97-77c6-4bb4-a9e4-e1ae27484457-metrics-certs") pod "network-metrics-daemon-bdcds" (UID: "b37e5c97-77c6-4bb4-a9e4-e1ae27484457") : secret "metrics-daemon-secret" not found Nov 04 02:32:34.909059 ip-10-0-134-125 kubenswrapper[4405]: E1104 02:32:34.909045 4405 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Nov 04 02:32:34.909131 ip-10-0-134-125 kubenswrapper[4405]: E1104 02:32:34.909107 4405 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4acab1fc-c591-475a-a15f-790be8131547-insights-runtime-extractor-tls podName:4acab1fc-c591-475a-a15f-790be8131547 nodeName:}" failed. No retries permitted until 2026-11-04 02:32:35.40909044 +0000 UTC m=+1.839790538 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/4acab1fc-c591-475a-a15f-790be8131547-insights-runtime-extractor-tls") pod "insights-runtime-extractor-5f2vb" (UID: "4acab1fc-c591-475a-a15f-790be8131547") : secret "insights-runtime-extractor-tls" not found Nov 04 02:32:34.912868 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:32:34.912735 4405 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8lqpv\" (UniqueName: \"kubernetes.io/projected/4f76b032-61eb-4796-b082-47c23ff73d18-kube-api-access-8lqpv\") pod \"dns-default-wbzfc\" (UID: \"4f76b032-61eb-4796-b082-47c23ff73d18\") " pod="openshift-dns/dns-default-wbzfc" Nov 04 02:32:34.912868 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:32:34.912799 4405 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bkf4l\" (UniqueName: \"kubernetes.io/projected/4acab1fc-c591-475a-a15f-790be8131547-kube-api-access-bkf4l\") pod \"insights-runtime-extractor-5f2vb\" (UID: \"4acab1fc-c591-475a-a15f-790be8131547\") " pod="openshift-insights/insights-runtime-extractor-5f2vb" Nov 04 02:32:34.912868 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:32:34.912855 4405 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-plfvd\" (UniqueName: \"kubernetes.io/projected/3984ce7b-96c8-464a-aa14-28487cc5b373-kube-api-access-plfvd\") pod \"multus-tglvr\" (UID: \"3984ce7b-96c8-464a-aa14-28487cc5b373\") " pod="openshift-multus/multus-tglvr" Nov 04 02:32:34.913033 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:32:34.912868 4405 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8ll77\" (UniqueName: \"kubernetes.io/projected/8b030f29-dfa6-42b1-81a7-07b539ad9aa9-kube-api-access-8ll77\") pod \"node-resolver-kzzwk\" (UID: \"8b030f29-dfa6-42b1-81a7-07b539ad9aa9\") " pod="openshift-dns/node-resolver-kzzwk" Nov 04 02:32:34.913033 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:32:34.912923 4405 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b4snb\" (UniqueName: \"kubernetes.io/projected/f58e4ac8-c132-4716-b6c0-8d8ddb9c19e5-kube-api-access-b4snb\") pod \"aws-ebs-csi-driver-node-6mvxj\" (UID: \"f58e4ac8-c132-4716-b6c0-8d8ddb9c19e5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6mvxj" Nov 04 02:32:34.913033 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:32:34.912948 4405 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-stzcf\" (UniqueName: \"kubernetes.io/projected/524e33f8-723b-4868-a4f5-5d7eb92a5fe3-kube-api-access-stzcf\") pod \"iptables-alerter-sx4w7\" (UID: \"524e33f8-723b-4868-a4f5-5d7eb92a5fe3\") " pod="openshift-network-operator/iptables-alerter-sx4w7" Nov 04 02:32:34.913033 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:32:34.913002 4405 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2rpzn\" (UniqueName: \"kubernetes.io/projected/a245d359-742e-489d-9c58-eea4e686286c-kube-api-access-2rpzn\") pod \"ingress-canary-mn7dn\" (UID: \"a245d359-742e-489d-9c58-eea4e686286c\") " pod="openshift-ingress-canary/ingress-canary-mn7dn" Nov 04 02:32:34.913143 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:32:34.913076 4405 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dddmg\" (UniqueName: \"kubernetes.io/projected/6e27a76f-fbdb-4ce6-9df6-659cab9ad06c-kube-api-access-dddmg\") pod \"network-check-target-zd8z9\" (UID: \"6e27a76f-fbdb-4ce6-9df6-659cab9ad06c\") " pod="openshift-network-diagnostics/network-check-target-zd8z9" Nov 04 02:32:34.913172 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:32:34.913163 4405 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4rpxv\" (UniqueName: \"kubernetes.io/projected/5590efb1-88b8-41ad-b0ee-37b2ff9442de-kube-api-access-4rpxv\") pod \"multus-additional-cni-plugins-sgx95\" (UID: \"5590efb1-88b8-41ad-b0ee-37b2ff9442de\") " pod="openshift-multus/multus-additional-cni-plugins-sgx95" Nov 04 02:32:34.914167 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:32:34.914150 4405 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/5590efb1-88b8-41ad-b0ee-37b2ff9442de-cni-binary-copy\") pod \"multus-additional-cni-plugins-sgx95\" (UID: \"5590efb1-88b8-41ad-b0ee-37b2ff9442de\") " pod="openshift-multus/multus-additional-cni-plugins-sgx95" Nov 04 02:32:34.952609 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:32:34.952583 4405 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"trusted-ca-bundle\"" Nov 04 02:32:34.956908 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:32:34.956881 4405 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5a5c4b25-7a35-4290-ad1e-04b4a3617877-trusted-ca-bundle\") pod \"periodic-gathering-hxwj2-xcshl\" (UID: \"5a5c4b25-7a35-4290-ad1e-04b4a3617877\") " pod="openshift-insights/periodic-gathering-hxwj2-xcshl" Nov 04 02:32:35.055459 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:32:35.055439 4405 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 04 02:32:35.475076 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:32:35.475038 4405 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b37e5c97-77c6-4bb4-a9e4-e1ae27484457-metrics-certs\") pod \"network-metrics-daemon-bdcds\" (UID: \"b37e5c97-77c6-4bb4-a9e4-e1ae27484457\") " pod="openshift-multus/network-metrics-daemon-bdcds" Nov 04 02:32:35.475222 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:32:35.475086 4405 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/4acab1fc-c591-475a-a15f-790be8131547-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-5f2vb\" (UID: \"4acab1fc-c591-475a-a15f-790be8131547\") " pod="openshift-insights/insights-runtime-extractor-5f2vb" Nov 04 02:32:35.475222 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:32:35.475131 4405 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/4f76b032-61eb-4796-b082-47c23ff73d18-metrics-tls\") pod \"dns-default-wbzfc\" (UID: \"4f76b032-61eb-4796-b082-47c23ff73d18\") " pod="openshift-dns/dns-default-wbzfc" Nov 04 02:32:35.475222 ip-10-0-134-125 kubenswrapper[4405]: E1104 02:32:35.475191 4405 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found Nov 04 02:32:35.475384 ip-10-0-134-125 kubenswrapper[4405]: E1104 02:32:35.475229 4405 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Nov 04 02:32:35.475384 ip-10-0-134-125 kubenswrapper[4405]: E1104 02:32:35.475229 4405 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Nov 04 02:32:35.475384 ip-10-0-134-125 kubenswrapper[4405]: E1104 02:32:35.475265 4405 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b37e5c97-77c6-4bb4-a9e4-e1ae27484457-metrics-certs podName:b37e5c97-77c6-4bb4-a9e4-e1ae27484457 nodeName:}" failed. No retries permitted until 2026-11-04 02:32:36.475249307 +0000 UTC m=+2.905949405 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/b37e5c97-77c6-4bb4-a9e4-e1ae27484457-metrics-certs") pod "network-metrics-daemon-bdcds" (UID: "b37e5c97-77c6-4bb4-a9e4-e1ae27484457") : secret "metrics-daemon-secret" not found Nov 04 02:32:35.475384 ip-10-0-134-125 kubenswrapper[4405]: E1104 02:32:35.475280 4405 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4f76b032-61eb-4796-b082-47c23ff73d18-metrics-tls podName:4f76b032-61eb-4796-b082-47c23ff73d18 nodeName:}" failed. No retries permitted until 2026-11-04 02:32:36.475273781 +0000 UTC m=+2.905973880 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/4f76b032-61eb-4796-b082-47c23ff73d18-metrics-tls") pod "dns-default-wbzfc" (UID: "4f76b032-61eb-4796-b082-47c23ff73d18") : secret "dns-default-metrics-tls" not found Nov 04 02:32:35.475384 ip-10-0-134-125 kubenswrapper[4405]: E1104 02:32:35.475292 4405 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4acab1fc-c591-475a-a15f-790be8131547-insights-runtime-extractor-tls podName:4acab1fc-c591-475a-a15f-790be8131547 nodeName:}" failed. No retries permitted until 2026-11-04 02:32:36.475284976 +0000 UTC m=+2.905985074 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/4acab1fc-c591-475a-a15f-790be8131547-insights-runtime-extractor-tls") pod "insights-runtime-extractor-5f2vb" (UID: "4acab1fc-c591-475a-a15f-790be8131547") : secret "insights-runtime-extractor-tls" not found Nov 04 02:32:36.020685 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:32:36.020645 4405 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-zd8z9" Nov 04 02:32:36.480957 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:32:36.480918 4405 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b37e5c97-77c6-4bb4-a9e4-e1ae27484457-metrics-certs\") pod \"network-metrics-daemon-bdcds\" (UID: \"b37e5c97-77c6-4bb4-a9e4-e1ae27484457\") " pod="openshift-multus/network-metrics-daemon-bdcds" Nov 04 02:32:36.481145 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:32:36.480984 4405 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/4acab1fc-c591-475a-a15f-790be8131547-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-5f2vb\" (UID: \"4acab1fc-c591-475a-a15f-790be8131547\") " pod="openshift-insights/insights-runtime-extractor-5f2vb" Nov 04 02:32:36.481145 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:32:36.481051 4405 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/4f76b032-61eb-4796-b082-47c23ff73d18-metrics-tls\") pod \"dns-default-wbzfc\" (UID: \"4f76b032-61eb-4796-b082-47c23ff73d18\") " pod="openshift-dns/dns-default-wbzfc" Nov 04 02:32:36.481145 ip-10-0-134-125 kubenswrapper[4405]: E1104 02:32:36.481121 4405 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found Nov 04 02:32:36.481266 ip-10-0-134-125 kubenswrapper[4405]: E1104 02:32:36.481149 4405 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Nov 04 02:32:36.481266 ip-10-0-134-125 kubenswrapper[4405]: E1104 02:32:36.481151 4405 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Nov 04 02:32:36.481266 ip-10-0-134-125 kubenswrapper[4405]: E1104 02:32:36.481207 4405 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b37e5c97-77c6-4bb4-a9e4-e1ae27484457-metrics-certs podName:b37e5c97-77c6-4bb4-a9e4-e1ae27484457 nodeName:}" failed. No retries permitted until 2026-11-04 02:32:38.481185256 +0000 UTC m=+4.911885366 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/b37e5c97-77c6-4bb4-a9e4-e1ae27484457-metrics-certs") pod "network-metrics-daemon-bdcds" (UID: "b37e5c97-77c6-4bb4-a9e4-e1ae27484457") : secret "metrics-daemon-secret" not found Nov 04 02:32:36.481266 ip-10-0-134-125 kubenswrapper[4405]: E1104 02:32:36.481228 4405 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4f76b032-61eb-4796-b082-47c23ff73d18-metrics-tls podName:4f76b032-61eb-4796-b082-47c23ff73d18 nodeName:}" failed. No retries permitted until 2026-11-04 02:32:38.481218265 +0000 UTC m=+4.911918392 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/4f76b032-61eb-4796-b082-47c23ff73d18-metrics-tls") pod "dns-default-wbzfc" (UID: "4f76b032-61eb-4796-b082-47c23ff73d18") : secret "dns-default-metrics-tls" not found Nov 04 02:32:36.481266 ip-10-0-134-125 kubenswrapper[4405]: E1104 02:32:36.481244 4405 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4acab1fc-c591-475a-a15f-790be8131547-insights-runtime-extractor-tls podName:4acab1fc-c591-475a-a15f-790be8131547 nodeName:}" failed. No retries permitted until 2026-11-04 02:32:38.481234501 +0000 UTC m=+4.911934614 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/4acab1fc-c591-475a-a15f-790be8131547-insights-runtime-extractor-tls") pod "insights-runtime-extractor-5f2vb" (UID: "4acab1fc-c591-475a-a15f-790be8131547") : secret "insights-runtime-extractor-tls" not found Nov 04 02:32:38.080583 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:32:38.080554 4405 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-vwcx8" Nov 04 02:32:38.495330 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:32:38.495294 4405 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/4f76b032-61eb-4796-b082-47c23ff73d18-metrics-tls\") pod \"dns-default-wbzfc\" (UID: \"4f76b032-61eb-4796-b082-47c23ff73d18\") " pod="openshift-dns/dns-default-wbzfc" Nov 04 02:32:38.495495 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:32:38.495338 4405 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b37e5c97-77c6-4bb4-a9e4-e1ae27484457-metrics-certs\") pod \"network-metrics-daemon-bdcds\" (UID: \"b37e5c97-77c6-4bb4-a9e4-e1ae27484457\") " pod="openshift-multus/network-metrics-daemon-bdcds" Nov 04 02:32:38.495495 ip-10-0-134-125 kubenswrapper[4405]: E1104 02:32:38.495441 4405 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Nov 04 02:32:38.495565 ip-10-0-134-125 kubenswrapper[4405]: E1104 02:32:38.495500 4405 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found Nov 04 02:32:38.495565 ip-10-0-134-125 kubenswrapper[4405]: E1104 02:32:38.495508 4405 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4f76b032-61eb-4796-b082-47c23ff73d18-metrics-tls podName:4f76b032-61eb-4796-b082-47c23ff73d18 nodeName:}" failed. No retries permitted until 2026-11-04 02:32:42.495492808 +0000 UTC m=+8.926192908 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/4f76b032-61eb-4796-b082-47c23ff73d18-metrics-tls") pod "dns-default-wbzfc" (UID: "4f76b032-61eb-4796-b082-47c23ff73d18") : secret "dns-default-metrics-tls" not found Nov 04 02:32:38.495565 ip-10-0-134-125 kubenswrapper[4405]: E1104 02:32:38.495537 4405 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b37e5c97-77c6-4bb4-a9e4-e1ae27484457-metrics-certs podName:b37e5c97-77c6-4bb4-a9e4-e1ae27484457 nodeName:}" failed. No retries permitted until 2026-11-04 02:32:42.495526842 +0000 UTC m=+8.926226944 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/b37e5c97-77c6-4bb4-a9e4-e1ae27484457-metrics-certs") pod "network-metrics-daemon-bdcds" (UID: "b37e5c97-77c6-4bb4-a9e4-e1ae27484457") : secret "metrics-daemon-secret" not found Nov 04 02:32:38.495565 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:32:38.495555 4405 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/4acab1fc-c591-475a-a15f-790be8131547-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-5f2vb\" (UID: \"4acab1fc-c591-475a-a15f-790be8131547\") " pod="openshift-insights/insights-runtime-extractor-5f2vb" Nov 04 02:32:38.495701 ip-10-0-134-125 kubenswrapper[4405]: E1104 02:32:38.495621 4405 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Nov 04 02:32:38.495737 ip-10-0-134-125 kubenswrapper[4405]: E1104 02:32:38.495699 4405 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4acab1fc-c591-475a-a15f-790be8131547-insights-runtime-extractor-tls podName:4acab1fc-c591-475a-a15f-790be8131547 nodeName:}" failed. No retries permitted until 2026-11-04 02:32:42.495658766 +0000 UTC m=+8.926358868 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/4acab1fc-c591-475a-a15f-790be8131547-insights-runtime-extractor-tls") pod "insights-runtime-extractor-5f2vb" (UID: "4acab1fc-c591-475a-a15f-790be8131547") : secret "insights-runtime-extractor-tls" not found Nov 04 02:32:39.607970 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:32:39.607942 4405 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-vwcx8" Nov 04 02:32:39.608407 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:32:39.608139 4405 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Nov 04 02:32:42.522602 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:32:42.522560 4405 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/4f76b032-61eb-4796-b082-47c23ff73d18-metrics-tls\") pod \"dns-default-wbzfc\" (UID: \"4f76b032-61eb-4796-b082-47c23ff73d18\") " pod="openshift-dns/dns-default-wbzfc" Nov 04 02:32:42.522602 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:32:42.522604 4405 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b37e5c97-77c6-4bb4-a9e4-e1ae27484457-metrics-certs\") pod \"network-metrics-daemon-bdcds\" (UID: \"b37e5c97-77c6-4bb4-a9e4-e1ae27484457\") " pod="openshift-multus/network-metrics-daemon-bdcds" Nov 04 02:32:42.523067 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:32:42.522629 4405 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/4acab1fc-c591-475a-a15f-790be8131547-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-5f2vb\" (UID: \"4acab1fc-c591-475a-a15f-790be8131547\") " pod="openshift-insights/insights-runtime-extractor-5f2vb" Nov 04 02:32:42.523067 ip-10-0-134-125 kubenswrapper[4405]: E1104 02:32:42.522729 4405 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Nov 04 02:32:42.523067 ip-10-0-134-125 kubenswrapper[4405]: E1104 02:32:42.522736 4405 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found Nov 04 02:32:42.523067 ip-10-0-134-125 kubenswrapper[4405]: E1104 02:32:42.522788 4405 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4f76b032-61eb-4796-b082-47c23ff73d18-metrics-tls podName:4f76b032-61eb-4796-b082-47c23ff73d18 nodeName:}" failed. No retries permitted until 2026-11-04 02:32:50.522773973 +0000 UTC m=+16.953474071 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/4f76b032-61eb-4796-b082-47c23ff73d18-metrics-tls") pod "dns-default-wbzfc" (UID: "4f76b032-61eb-4796-b082-47c23ff73d18") : secret "dns-default-metrics-tls" not found Nov 04 02:32:42.523067 ip-10-0-134-125 kubenswrapper[4405]: E1104 02:32:42.522812 4405 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b37e5c97-77c6-4bb4-a9e4-e1ae27484457-metrics-certs podName:b37e5c97-77c6-4bb4-a9e4-e1ae27484457 nodeName:}" failed. No retries permitted until 2026-11-04 02:32:50.522794303 +0000 UTC m=+16.953494404 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/b37e5c97-77c6-4bb4-a9e4-e1ae27484457-metrics-certs") pod "network-metrics-daemon-bdcds" (UID: "b37e5c97-77c6-4bb4-a9e4-e1ae27484457") : secret "metrics-daemon-secret" not found Nov 04 02:32:42.523067 ip-10-0-134-125 kubenswrapper[4405]: E1104 02:32:42.522736 4405 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Nov 04 02:32:42.523067 ip-10-0-134-125 kubenswrapper[4405]: E1104 02:32:42.522842 4405 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4acab1fc-c591-475a-a15f-790be8131547-insights-runtime-extractor-tls podName:4acab1fc-c591-475a-a15f-790be8131547 nodeName:}" failed. No retries permitted until 2026-11-04 02:32:50.522836314 +0000 UTC m=+16.953536412 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/4acab1fc-c591-475a-a15f-790be8131547-insights-runtime-extractor-tls") pod "insights-runtime-extractor-5f2vb" (UID: "4acab1fc-c591-475a-a15f-790be8131547") : secret "insights-runtime-extractor-tls" not found Nov 04 02:32:42.752141 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:32:42.752100 4405 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-hxwj2-xcshl" event={"ID":"5a5c4b25-7a35-4290-ad1e-04b4a3617877","Type":"ContainerStarted","Data":"4aac51ec380c8a9549949608c65315e45b58f4f5f73f99f05954b2c8b6f625ad"} Nov 04 02:32:44.241926 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:32:44.241897 4405 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-125.us-west-1.compute.internal" event="NodeReady" Nov 04 02:32:46.059814 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:32:46.059786 4405 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator/migrator-66df5d496b-fv5zv"] Nov 04 02:32:46.061467 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:32:46.061447 4405 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-fv5zv" Nov 04 02:32:46.114798 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:32:46.114762 4405 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-root-ca.crt\"" Nov 04 02:32:46.114798 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:32:46.114768 4405 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"openshift-service-ca.crt\"" Nov 04 02:32:46.149586 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:32:46.149551 4405 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hvx9k\" (UniqueName: \"kubernetes.io/projected/877afef0-87db-4cb4-bcfc-9bc33c32a047-kube-api-access-hvx9k\") pod \"migrator-66df5d496b-fv5zv\" (UID: \"877afef0-87db-4cb4-bcfc-9bc33c32a047\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-fv5zv" Nov 04 02:32:46.164605 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:32:46.164574 4405 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-storage-version-migrator-sa-dockercfg-4p5kw\"" Nov 04 02:32:46.165000 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:32:46.164979 4405 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-66df5d496b-fv5zv"] Nov 04 02:32:46.250434 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:32:46.250402 4405 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hvx9k\" (UniqueName: \"kubernetes.io/projected/877afef0-87db-4cb4-bcfc-9bc33c32a047-kube-api-access-hvx9k\") pod \"migrator-66df5d496b-fv5zv\" (UID: \"877afef0-87db-4cb4-bcfc-9bc33c32a047\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-fv5zv" Nov 04 02:32:46.309491 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:32:46.309454 4405 swap_util.go:78] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Nov 04 02:32:46.311860 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:32:46.311795 4405 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hvx9k\" (UniqueName: \"kubernetes.io/projected/877afef0-87db-4cb4-bcfc-9bc33c32a047-kube-api-access-hvx9k\") pod \"migrator-66df5d496b-fv5zv\" (UID: \"877afef0-87db-4cb4-bcfc-9bc33c32a047\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-fv5zv" Nov 04 02:32:46.370956 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:32:46.370919 4405 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-fv5zv" Nov 04 02:32:46.734253 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:32:46.734215 4405 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-66df5d496b-fv5zv"] Nov 04 02:32:46.737732 ip-10-0-134-125 kubenswrapper[4405]: W1104 02:32:46.737702 4405 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod877afef0_87db_4cb4_bcfc_9bc33c32a047.slice/crio-034359fb42b6c4c06e5acf2e6a77fafc31e02058a2cebba69e6c98a7744ef6c1 WatchSource:0}: Error finding container 034359fb42b6c4c06e5acf2e6a77fafc31e02058a2cebba69e6c98a7744ef6c1: Status 404 returned error can't find the container with id 034359fb42b6c4c06e5acf2e6a77fafc31e02058a2cebba69e6c98a7744ef6c1 Nov 04 02:32:46.762728 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:32:46.762699 4405 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-fv5zv" event={"ID":"877afef0-87db-4cb4-bcfc-9bc33c32a047","Type":"ContainerStarted","Data":"034359fb42b6c4c06e5acf2e6a77fafc31e02058a2cebba69e6c98a7744ef6c1"} Nov 04 02:32:47.466017 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:32:47.465983 4405 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-wcp7z" Nov 04 02:32:47.678097 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:32:47.678068 4405 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-kzzwk_8b030f29-dfa6-42b1-81a7-07b539ad9aa9/dns-node-resolver/0.log" Nov 04 02:32:48.477067 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:32:48.477042 4405 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-2h96g_7cb29949-bfe4-47a2-bc17-115a8d4d895d/node-ca/0.log" Nov 04 02:32:48.537862 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:32:48.537830 4405 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-service-ca/service-ca-5fffc97bdf-h65fn"] Nov 04 02:32:48.539586 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:32:48.539570 4405 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-5fffc97bdf-h65fn" Nov 04 02:32:48.567998 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:32:48.567962 4405 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7jmtl\" (UniqueName: \"kubernetes.io/projected/f54a58d1-e549-40e0-85dc-b90ebc008f46-kube-api-access-7jmtl\") pod \"service-ca-5fffc97bdf-h65fn\" (UID: \"f54a58d1-e549-40e0-85dc-b90ebc008f46\") " pod="openshift-service-ca/service-ca-5fffc97bdf-h65fn" Nov 04 02:32:48.568152 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:32:48.568071 4405 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/f54a58d1-e549-40e0-85dc-b90ebc008f46-signing-cabundle\") pod \"service-ca-5fffc97bdf-h65fn\" (UID: \"f54a58d1-e549-40e0-85dc-b90ebc008f46\") " pod="openshift-service-ca/service-ca-5fffc97bdf-h65fn" Nov 04 02:32:48.568152 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:32:48.568096 4405 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/f54a58d1-e549-40e0-85dc-b90ebc008f46-signing-key\") pod \"service-ca-5fffc97bdf-h65fn\" (UID: \"f54a58d1-e549-40e0-85dc-b90ebc008f46\") " pod="openshift-service-ca/service-ca-5fffc97bdf-h65fn" Nov 04 02:32:48.594219 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:32:48.594187 4405 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"signing-cabundle\"" Nov 04 02:32:48.622478 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:32:48.622449 4405 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"signing-key\"" Nov 04 02:32:48.644168 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:32:48.644137 4405 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"service-ca-dockercfg-rspxf\"" Nov 04 02:32:48.644619 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:32:48.644601 4405 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-5fffc97bdf-h65fn"] Nov 04 02:32:48.669003 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:32:48.668920 4405 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7jmtl\" (UniqueName: \"kubernetes.io/projected/f54a58d1-e549-40e0-85dc-b90ebc008f46-kube-api-access-7jmtl\") pod \"service-ca-5fffc97bdf-h65fn\" (UID: \"f54a58d1-e549-40e0-85dc-b90ebc008f46\") " pod="openshift-service-ca/service-ca-5fffc97bdf-h65fn" Nov 04 02:32:48.669003 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:32:48.668980 4405 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/f54a58d1-e549-40e0-85dc-b90ebc008f46-signing-cabundle\") pod \"service-ca-5fffc97bdf-h65fn\" (UID: \"f54a58d1-e549-40e0-85dc-b90ebc008f46\") " pod="openshift-service-ca/service-ca-5fffc97bdf-h65fn" Nov 04 02:32:48.669177 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:32:48.669146 4405 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/f54a58d1-e549-40e0-85dc-b90ebc008f46-signing-key\") pod \"service-ca-5fffc97bdf-h65fn\" (UID: \"f54a58d1-e549-40e0-85dc-b90ebc008f46\") " pod="openshift-service-ca/service-ca-5fffc97bdf-h65fn" Nov 04 02:32:48.669576 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:32:48.669555 4405 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/f54a58d1-e549-40e0-85dc-b90ebc008f46-signing-cabundle\") pod \"service-ca-5fffc97bdf-h65fn\" (UID: \"f54a58d1-e549-40e0-85dc-b90ebc008f46\") " pod="openshift-service-ca/service-ca-5fffc97bdf-h65fn" Nov 04 02:32:48.671635 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:32:48.671611 4405 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/f54a58d1-e549-40e0-85dc-b90ebc008f46-signing-key\") pod \"service-ca-5fffc97bdf-h65fn\" (UID: \"f54a58d1-e549-40e0-85dc-b90ebc008f46\") " pod="openshift-service-ca/service-ca-5fffc97bdf-h65fn" Nov 04 02:32:48.694387 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:32:48.694366 4405 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"openshift-service-ca.crt\"" Nov 04 02:32:48.694520 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:32:48.694504 4405 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"kube-root-ca.crt\"" Nov 04 02:32:48.733750 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:32:48.733718 4405 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7jmtl\" (UniqueName: \"kubernetes.io/projected/f54a58d1-e549-40e0-85dc-b90ebc008f46-kube-api-access-7jmtl\") pod \"service-ca-5fffc97bdf-h65fn\" (UID: \"f54a58d1-e549-40e0-85dc-b90ebc008f46\") " pod="openshift-service-ca/service-ca-5fffc97bdf-h65fn" Nov 04 02:32:48.773620 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:32:48.773577 4405 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-fv5zv" event={"ID":"877afef0-87db-4cb4-bcfc-9bc33c32a047","Type":"ContainerStarted","Data":"7631c8687b7959fffd80c9427066b0a62d8f6812dd7a02ecf59148aa71298332"} Nov 04 02:32:48.848985 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:32:48.848949 4405 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-5fffc97bdf-h65fn" Nov 04 02:32:48.891935 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:32:48.891881 4405 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-fv5zv" podStartSLOduration=1.223700513 podStartE2EDuration="2.891866584s" totalImagesPullingTime="1.668166071s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 02:32:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 02:32:48.891257942 +0000 UTC m=+15.321958064" watchObservedRunningTime="2026-11-04 02:32:48.891866584 +0000 UTC m=+15.322566700" Nov 04 02:32:49.215285 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:32:49.215255 4405 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-5fffc97bdf-h65fn"] Nov 04 02:32:49.218452 ip-10-0-134-125 kubenswrapper[4405]: W1104 02:32:49.218426 4405 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf54a58d1_e549_40e0_85dc_b90ebc008f46.slice/crio-ea61d0fc55ace75a76d092f2126d05d31033ac22d1937687b41b4ebb8fe69c7c WatchSource:0}: Error finding container ea61d0fc55ace75a76d092f2126d05d31033ac22d1937687b41b4ebb8fe69c7c: Status 404 returned error can't find the container with id ea61d0fc55ace75a76d092f2126d05d31033ac22d1937687b41b4ebb8fe69c7c Nov 04 02:32:49.477895 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:32:49.477815 4405 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-mn7dn_a245d359-742e-489d-9c58-eea4e686286c/serve-healthcheck-canary/0.log" Nov 04 02:32:49.762930 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:32:49.762854 4405 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-vwcx8" Nov 04 02:32:49.776282 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:32:49.776254 4405 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-5fffc97bdf-h65fn" event={"ID":"f54a58d1-e549-40e0-85dc-b90ebc008f46","Type":"ContainerStarted","Data":"ea61d0fc55ace75a76d092f2126d05d31033ac22d1937687b41b4ebb8fe69c7c"} Nov 04 02:32:50.587265 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:32:50.587220 4405 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/4f76b032-61eb-4796-b082-47c23ff73d18-metrics-tls\") pod \"dns-default-wbzfc\" (UID: \"4f76b032-61eb-4796-b082-47c23ff73d18\") " pod="openshift-dns/dns-default-wbzfc" Nov 04 02:32:50.587939 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:32:50.587287 4405 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b37e5c97-77c6-4bb4-a9e4-e1ae27484457-metrics-certs\") pod \"network-metrics-daemon-bdcds\" (UID: \"b37e5c97-77c6-4bb4-a9e4-e1ae27484457\") " pod="openshift-multus/network-metrics-daemon-bdcds" Nov 04 02:32:50.587939 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:32:50.587329 4405 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/4acab1fc-c591-475a-a15f-790be8131547-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-5f2vb\" (UID: \"4acab1fc-c591-475a-a15f-790be8131547\") " pod="openshift-insights/insights-runtime-extractor-5f2vb" Nov 04 02:32:50.587939 ip-10-0-134-125 kubenswrapper[4405]: E1104 02:32:50.587409 4405 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Nov 04 02:32:50.587939 ip-10-0-134-125 kubenswrapper[4405]: E1104 02:32:50.587463 4405 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Nov 04 02:32:50.587939 ip-10-0-134-125 kubenswrapper[4405]: E1104 02:32:50.587467 4405 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found Nov 04 02:32:50.587939 ip-10-0-134-125 kubenswrapper[4405]: E1104 02:32:50.587497 4405 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4f76b032-61eb-4796-b082-47c23ff73d18-metrics-tls podName:4f76b032-61eb-4796-b082-47c23ff73d18 nodeName:}" failed. No retries permitted until 2026-11-04 02:33:06.5874761 +0000 UTC m=+33.018176221 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/4f76b032-61eb-4796-b082-47c23ff73d18-metrics-tls") pod "dns-default-wbzfc" (UID: "4f76b032-61eb-4796-b082-47c23ff73d18") : secret "dns-default-metrics-tls" not found Nov 04 02:32:50.587939 ip-10-0-134-125 kubenswrapper[4405]: E1104 02:32:50.587523 4405 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4acab1fc-c591-475a-a15f-790be8131547-insights-runtime-extractor-tls podName:4acab1fc-c591-475a-a15f-790be8131547 nodeName:}" failed. No retries permitted until 2026-11-04 02:33:06.58750724 +0000 UTC m=+33.018207359 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/4acab1fc-c591-475a-a15f-790be8131547-insights-runtime-extractor-tls") pod "insights-runtime-extractor-5f2vb" (UID: "4acab1fc-c591-475a-a15f-790be8131547") : secret "insights-runtime-extractor-tls" not found Nov 04 02:32:50.587939 ip-10-0-134-125 kubenswrapper[4405]: E1104 02:32:50.587543 4405 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b37e5c97-77c6-4bb4-a9e4-e1ae27484457-metrics-certs podName:b37e5c97-77c6-4bb4-a9e4-e1ae27484457 nodeName:}" failed. No retries permitted until 2026-11-04 02:33:06.587533045 +0000 UTC m=+33.018233178 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/b37e5c97-77c6-4bb4-a9e4-e1ae27484457-metrics-certs") pod "network-metrics-daemon-bdcds" (UID: "b37e5c97-77c6-4bb4-a9e4-e1ae27484457") : secret "metrics-daemon-secret" not found Nov 04 02:32:52.784777 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:32:52.784735 4405 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-5fffc97bdf-h65fn" event={"ID":"f54a58d1-e549-40e0-85dc-b90ebc008f46","Type":"ContainerStarted","Data":"75e591b7d495b9d18a090e866b553becf9a4411fbf10ca4d756c81a59f2f60b8"} Nov 04 02:32:52.899800 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:32:52.899746 4405 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-service-ca/service-ca-5fffc97bdf-h65fn" podStartSLOduration=1.561045702 podStartE2EDuration="4.899733214s" totalImagesPullingTime="3.338687512s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 02:32:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 02:32:52.899246951 +0000 UTC m=+19.329947071" watchObservedRunningTime="2026-11-04 02:32:52.899733214 +0000 UTC m=+19.330433333" Nov 04 02:33:06.022985 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:33:06.022956 4405 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-zd8z9" Nov 04 02:33:06.619434 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:33:06.619371 4405 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b37e5c97-77c6-4bb4-a9e4-e1ae27484457-metrics-certs\") pod \"network-metrics-daemon-bdcds\" (UID: \"b37e5c97-77c6-4bb4-a9e4-e1ae27484457\") " pod="openshift-multus/network-metrics-daemon-bdcds" Nov 04 02:33:06.619637 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:33:06.619448 4405 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/4acab1fc-c591-475a-a15f-790be8131547-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-5f2vb\" (UID: \"4acab1fc-c591-475a-a15f-790be8131547\") " pod="openshift-insights/insights-runtime-extractor-5f2vb" Nov 04 02:33:06.619637 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:33:06.619511 4405 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/4f76b032-61eb-4796-b082-47c23ff73d18-metrics-tls\") pod \"dns-default-wbzfc\" (UID: \"4f76b032-61eb-4796-b082-47c23ff73d18\") " pod="openshift-dns/dns-default-wbzfc" Nov 04 02:33:06.621952 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:33:06.621922 4405 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/4acab1fc-c591-475a-a15f-790be8131547-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-5f2vb\" (UID: \"4acab1fc-c591-475a-a15f-790be8131547\") " pod="openshift-insights/insights-runtime-extractor-5f2vb" Nov 04 02:33:06.622074 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:33:06.622015 4405 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b37e5c97-77c6-4bb4-a9e4-e1ae27484457-metrics-certs\") pod \"network-metrics-daemon-bdcds\" (UID: \"b37e5c97-77c6-4bb4-a9e4-e1ae27484457\") " pod="openshift-multus/network-metrics-daemon-bdcds" Nov 04 02:33:06.622074 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:33:06.622030 4405 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/4f76b032-61eb-4796-b082-47c23ff73d18-metrics-tls\") pod \"dns-default-wbzfc\" (UID: \"4f76b032-61eb-4796-b082-47c23ff73d18\") " pod="openshift-dns/dns-default-wbzfc" Nov 04 02:33:06.851897 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:33:06.851863 4405 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bdcds" Nov 04 02:33:06.854055 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:33:06.854022 4405 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-wbzfc" Nov 04 02:33:06.855096 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:33:06.855080 4405 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-5f2vb" Nov 04 02:33:07.249496 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:33:07.249412 4405 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-bdcds"] Nov 04 02:33:07.290696 ip-10-0-134-125 kubenswrapper[4405]: W1104 02:33:07.290642 4405 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb37e5c97_77c6_4bb4_a9e4_e1ae27484457.slice/crio-993f0e3923e835f554917c9bdc780cd62b0658aaf77c78a191d29d4fef3fb751 WatchSource:0}: Error finding container 993f0e3923e835f554917c9bdc780cd62b0658aaf77c78a191d29d4fef3fb751: Status 404 returned error can't find the container with id 993f0e3923e835f554917c9bdc780cd62b0658aaf77c78a191d29d4fef3fb751 Nov 04 02:33:07.300728 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:33:07.300706 4405 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-wbzfc"] Nov 04 02:33:07.303408 ip-10-0-134-125 kubenswrapper[4405]: W1104 02:33:07.303389 4405 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4f76b032_61eb_4796_b082_47c23ff73d18.slice/crio-c5661871e94487044494fb83b03ca63d39c6836371b87616794ec1600531bbb8 WatchSource:0}: Error finding container c5661871e94487044494fb83b03ca63d39c6836371b87616794ec1600531bbb8: Status 404 returned error can't find the container with id c5661871e94487044494fb83b03ca63d39c6836371b87616794ec1600531bbb8 Nov 04 02:33:07.340144 ip-10-0-134-125 kubenswrapper[4405]: W1104 02:33:07.340113 4405 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4acab1fc_c591_475a_a15f_790be8131547.slice/crio-1dc5a9bbbc276dde1b0a83e6cd5159dc104725246d292dc1f521bbd537adcbd0 WatchSource:0}: Error finding container 1dc5a9bbbc276dde1b0a83e6cd5159dc104725246d292dc1f521bbd537adcbd0: Status 404 returned error can't find the container with id 1dc5a9bbbc276dde1b0a83e6cd5159dc104725246d292dc1f521bbd537adcbd0 Nov 04 02:33:07.350284 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:33:07.350265 4405 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-5f2vb"] Nov 04 02:33:07.824551 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:33:07.824514 4405 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-5f2vb" event={"ID":"4acab1fc-c591-475a-a15f-790be8131547","Type":"ContainerStarted","Data":"b9f256ae0b6805662551fc48bbf8562ad86f917cbdf71285b03fb5858b6bdfd6"} Nov 04 02:33:07.824770 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:33:07.824593 4405 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-5f2vb" event={"ID":"4acab1fc-c591-475a-a15f-790be8131547","Type":"ContainerStarted","Data":"1dc5a9bbbc276dde1b0a83e6cd5159dc104725246d292dc1f521bbd537adcbd0"} Nov 04 02:33:07.825642 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:33:07.825612 4405 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-wbzfc" event={"ID":"4f76b032-61eb-4796-b082-47c23ff73d18","Type":"ContainerStarted","Data":"c5661871e94487044494fb83b03ca63d39c6836371b87616794ec1600531bbb8"} Nov 04 02:33:07.826652 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:33:07.826631 4405 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-bdcds" event={"ID":"b37e5c97-77c6-4bb4-a9e4-e1ae27484457","Type":"ContainerStarted","Data":"993f0e3923e835f554917c9bdc780cd62b0658aaf77c78a191d29d4fef3fb751"} Nov 04 02:33:09.192314 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:33:09.192287 4405 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-6dcc9f78d9-9t79d"] Nov 04 02:33:09.194206 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:33:09.194185 4405 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-6dcc9f78d9-9t79d" Nov 04 02:33:09.236635 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:33:09.236599 4405 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-6dcc9f78d9-9t79d"] Nov 04 02:33:09.266848 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:33:09.266817 4405 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\"" Nov 04 02:33:09.266848 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:33:09.266837 4405 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\"" Nov 04 02:33:09.339039 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:33:09.339005 4405 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/1722b2b5-068d-4327-9eeb-ae14421dab0b-registry-tls\") pod \"image-registry-6dcc9f78d9-9t79d\" (UID: \"1722b2b5-068d-4327-9eeb-ae14421dab0b\") " pod="openshift-image-registry/image-registry-6dcc9f78d9-9t79d" Nov 04 02:33:09.339222 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:33:09.339047 4405 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/1722b2b5-068d-4327-9eeb-ae14421dab0b-registry-certificates\") pod \"image-registry-6dcc9f78d9-9t79d\" (UID: \"1722b2b5-068d-4327-9eeb-ae14421dab0b\") " pod="openshift-image-registry/image-registry-6dcc9f78d9-9t79d" Nov 04 02:33:09.339222 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:33:09.339081 4405 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/1722b2b5-068d-4327-9eeb-ae14421dab0b-installation-pull-secrets\") pod \"image-registry-6dcc9f78d9-9t79d\" (UID: \"1722b2b5-068d-4327-9eeb-ae14421dab0b\") " pod="openshift-image-registry/image-registry-6dcc9f78d9-9t79d" Nov 04 02:33:09.339222 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:33:09.339109 4405 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/1722b2b5-068d-4327-9eeb-ae14421dab0b-bound-sa-token\") pod \"image-registry-6dcc9f78d9-9t79d\" (UID: \"1722b2b5-068d-4327-9eeb-ae14421dab0b\") " pod="openshift-image-registry/image-registry-6dcc9f78d9-9t79d" Nov 04 02:33:09.339222 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:33:09.339147 4405 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/1722b2b5-068d-4327-9eeb-ae14421dab0b-trusted-ca\") pod \"image-registry-6dcc9f78d9-9t79d\" (UID: \"1722b2b5-068d-4327-9eeb-ae14421dab0b\") " pod="openshift-image-registry/image-registry-6dcc9f78d9-9t79d" Nov 04 02:33:09.339222 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:33:09.339178 4405 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/1722b2b5-068d-4327-9eeb-ae14421dab0b-ca-trust-extracted\") pod \"image-registry-6dcc9f78d9-9t79d\" (UID: \"1722b2b5-068d-4327-9eeb-ae14421dab0b\") " pod="openshift-image-registry/image-registry-6dcc9f78d9-9t79d" Nov 04 02:33:09.339222 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:33:09.339215 4405 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/1722b2b5-068d-4327-9eeb-ae14421dab0b-image-registry-private-configuration\") pod \"image-registry-6dcc9f78d9-9t79d\" (UID: \"1722b2b5-068d-4327-9eeb-ae14421dab0b\") " pod="openshift-image-registry/image-registry-6dcc9f78d9-9t79d" Nov 04 02:33:09.339523 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:33:09.339240 4405 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8kqkp\" (UniqueName: \"kubernetes.io/projected/1722b2b5-068d-4327-9eeb-ae14421dab0b-kube-api-access-8kqkp\") pod \"image-registry-6dcc9f78d9-9t79d\" (UID: \"1722b2b5-068d-4327-9eeb-ae14421dab0b\") " pod="openshift-image-registry/image-registry-6dcc9f78d9-9t79d" Nov 04 02:33:09.348711 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:33:09.348625 4405 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\"" Nov 04 02:33:09.348711 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:33:09.348700 4405 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\"" Nov 04 02:33:09.350008 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:33:09.349984 4405 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-drzkl\"" Nov 04 02:33:09.440295 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:33:09.440261 4405 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/1722b2b5-068d-4327-9eeb-ae14421dab0b-ca-trust-extracted\") pod \"image-registry-6dcc9f78d9-9t79d\" (UID: \"1722b2b5-068d-4327-9eeb-ae14421dab0b\") " pod="openshift-image-registry/image-registry-6dcc9f78d9-9t79d" Nov 04 02:33:09.440426 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:33:09.440326 4405 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/1722b2b5-068d-4327-9eeb-ae14421dab0b-image-registry-private-configuration\") pod \"image-registry-6dcc9f78d9-9t79d\" (UID: \"1722b2b5-068d-4327-9eeb-ae14421dab0b\") " pod="openshift-image-registry/image-registry-6dcc9f78d9-9t79d" Nov 04 02:33:09.440426 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:33:09.440351 4405 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8kqkp\" (UniqueName: \"kubernetes.io/projected/1722b2b5-068d-4327-9eeb-ae14421dab0b-kube-api-access-8kqkp\") pod \"image-registry-6dcc9f78d9-9t79d\" (UID: \"1722b2b5-068d-4327-9eeb-ae14421dab0b\") " pod="openshift-image-registry/image-registry-6dcc9f78d9-9t79d" Nov 04 02:33:09.440426 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:33:09.440389 4405 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/1722b2b5-068d-4327-9eeb-ae14421dab0b-registry-tls\") pod \"image-registry-6dcc9f78d9-9t79d\" (UID: \"1722b2b5-068d-4327-9eeb-ae14421dab0b\") " pod="openshift-image-registry/image-registry-6dcc9f78d9-9t79d" Nov 04 02:33:09.440426 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:33:09.440405 4405 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/1722b2b5-068d-4327-9eeb-ae14421dab0b-registry-certificates\") pod \"image-registry-6dcc9f78d9-9t79d\" (UID: \"1722b2b5-068d-4327-9eeb-ae14421dab0b\") " pod="openshift-image-registry/image-registry-6dcc9f78d9-9t79d" Nov 04 02:33:09.440426 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:33:09.440421 4405 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/1722b2b5-068d-4327-9eeb-ae14421dab0b-installation-pull-secrets\") pod \"image-registry-6dcc9f78d9-9t79d\" (UID: \"1722b2b5-068d-4327-9eeb-ae14421dab0b\") " pod="openshift-image-registry/image-registry-6dcc9f78d9-9t79d" Nov 04 02:33:09.440729 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:33:09.440442 4405 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/1722b2b5-068d-4327-9eeb-ae14421dab0b-bound-sa-token\") pod \"image-registry-6dcc9f78d9-9t79d\" (UID: \"1722b2b5-068d-4327-9eeb-ae14421dab0b\") " pod="openshift-image-registry/image-registry-6dcc9f78d9-9t79d" Nov 04 02:33:09.440729 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:33:09.440473 4405 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/1722b2b5-068d-4327-9eeb-ae14421dab0b-trusted-ca\") pod \"image-registry-6dcc9f78d9-9t79d\" (UID: \"1722b2b5-068d-4327-9eeb-ae14421dab0b\") " pod="openshift-image-registry/image-registry-6dcc9f78d9-9t79d" Nov 04 02:33:09.442274 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:33:09.442239 4405 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/1722b2b5-068d-4327-9eeb-ae14421dab0b-ca-trust-extracted\") pod \"image-registry-6dcc9f78d9-9t79d\" (UID: \"1722b2b5-068d-4327-9eeb-ae14421dab0b\") " pod="openshift-image-registry/image-registry-6dcc9f78d9-9t79d" Nov 04 02:33:09.442856 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:33:09.442830 4405 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/1722b2b5-068d-4327-9eeb-ae14421dab0b-registry-certificates\") pod \"image-registry-6dcc9f78d9-9t79d\" (UID: \"1722b2b5-068d-4327-9eeb-ae14421dab0b\") " pod="openshift-image-registry/image-registry-6dcc9f78d9-9t79d" Nov 04 02:33:09.443263 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:33:09.443235 4405 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/1722b2b5-068d-4327-9eeb-ae14421dab0b-trusted-ca\") pod \"image-registry-6dcc9f78d9-9t79d\" (UID: \"1722b2b5-068d-4327-9eeb-ae14421dab0b\") " pod="openshift-image-registry/image-registry-6dcc9f78d9-9t79d" Nov 04 02:33:09.444154 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:33:09.444134 4405 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/1722b2b5-068d-4327-9eeb-ae14421dab0b-installation-pull-secrets\") pod \"image-registry-6dcc9f78d9-9t79d\" (UID: \"1722b2b5-068d-4327-9eeb-ae14421dab0b\") " pod="openshift-image-registry/image-registry-6dcc9f78d9-9t79d" Nov 04 02:33:09.444453 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:33:09.444423 4405 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/1722b2b5-068d-4327-9eeb-ae14421dab0b-image-registry-private-configuration\") pod \"image-registry-6dcc9f78d9-9t79d\" (UID: \"1722b2b5-068d-4327-9eeb-ae14421dab0b\") " pod="openshift-image-registry/image-registry-6dcc9f78d9-9t79d" Nov 04 02:33:09.444564 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:33:09.444541 4405 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/1722b2b5-068d-4327-9eeb-ae14421dab0b-registry-tls\") pod \"image-registry-6dcc9f78d9-9t79d\" (UID: \"1722b2b5-068d-4327-9eeb-ae14421dab0b\") " pod="openshift-image-registry/image-registry-6dcc9f78d9-9t79d" Nov 04 02:33:09.502138 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:33:09.502110 4405 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8kqkp\" (UniqueName: \"kubernetes.io/projected/1722b2b5-068d-4327-9eeb-ae14421dab0b-kube-api-access-8kqkp\") pod \"image-registry-6dcc9f78d9-9t79d\" (UID: \"1722b2b5-068d-4327-9eeb-ae14421dab0b\") " pod="openshift-image-registry/image-registry-6dcc9f78d9-9t79d" Nov 04 02:33:09.502365 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:33:09.502343 4405 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/1722b2b5-068d-4327-9eeb-ae14421dab0b-bound-sa-token\") pod \"image-registry-6dcc9f78d9-9t79d\" (UID: \"1722b2b5-068d-4327-9eeb-ae14421dab0b\") " pod="openshift-image-registry/image-registry-6dcc9f78d9-9t79d" Nov 04 02:33:09.507388 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:33:09.507366 4405 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-6dcc9f78d9-9t79d" Nov 04 02:33:09.836069 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:33:09.836033 4405 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-5f2vb" event={"ID":"4acab1fc-c591-475a-a15f-790be8131547","Type":"ContainerStarted","Data":"f167790e52f5a430af422a73250ecd18769e5ba1e95b1f9d7f7d43c2ed4e19e9"} Nov 04 02:33:09.837923 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:33:09.837890 4405 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-bdcds" event={"ID":"b37e5c97-77c6-4bb4-a9e4-e1ae27484457","Type":"ContainerStarted","Data":"e1cc448e298af2fbe23a8433ec86b9b97ca72ea01d954d98355ec0a29ee9b6d6"} Nov 04 02:33:09.838048 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:33:09.837946 4405 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-bdcds" event={"ID":"b37e5c97-77c6-4bb4-a9e4-e1ae27484457","Type":"ContainerStarted","Data":"141759bdbde5641c9a83161cfb2cb4c5c70bcf0766f3d8c9ddc0b33841e3b650"} Nov 04 02:33:09.993726 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:33:09.993622 4405 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-6dcc9f78d9-9t79d"] Nov 04 02:33:09.998158 ip-10-0-134-125 kubenswrapper[4405]: W1104 02:33:09.998130 4405 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1722b2b5_068d_4327_9eeb_ae14421dab0b.slice/crio-dff58bfcd3fd927db4d683b4cc362544b372295f8729bf07779c93cce2f85229 WatchSource:0}: Error finding container dff58bfcd3fd927db4d683b4cc362544b372295f8729bf07779c93cce2f85229: Status 404 returned error can't find the container with id dff58bfcd3fd927db4d683b4cc362544b372295f8729bf07779c93cce2f85229 Nov 04 02:33:10.586354 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:33:10.586320 4405 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console/downloads-6dd789c8b7-sxjs8"] Nov 04 02:33:10.588237 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:33:10.588217 4405 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-6dd789c8b7-sxjs8" Nov 04 02:33:10.644373 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:33:10.644337 4405 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\"" Nov 04 02:33:10.644562 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:33:10.644401 4405 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\"" Nov 04 02:33:10.644562 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:33:10.644476 4405 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"downloads-dockercfg-gq5j9\"" Nov 04 02:33:10.651443 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:33:10.651420 4405 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-6dd789c8b7-sxjs8"] Nov 04 02:33:10.750243 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:33:10.750199 4405 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-47rh6\" (UniqueName: \"kubernetes.io/projected/31f4e8d6-73e9-4c32-aa32-68995576b7c7-kube-api-access-47rh6\") pod \"downloads-6dd789c8b7-sxjs8\" (UID: \"31f4e8d6-73e9-4c32-aa32-68995576b7c7\") " pod="openshift-console/downloads-6dd789c8b7-sxjs8" Nov 04 02:33:10.750243 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:33:10.750251 4405 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/31f4e8d6-73e9-4c32-aa32-68995576b7c7-tmp\") pod \"downloads-6dd789c8b7-sxjs8\" (UID: \"31f4e8d6-73e9-4c32-aa32-68995576b7c7\") " pod="openshift-console/downloads-6dd789c8b7-sxjs8" Nov 04 02:33:10.843642 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:33:10.843552 4405 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-wbzfc" event={"ID":"4f76b032-61eb-4796-b082-47c23ff73d18","Type":"ContainerStarted","Data":"474ef9ee0579dbcf6d229552572173bdc98546d307c716bb322f26a2cabcd7a8"} Nov 04 02:33:10.843642 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:33:10.843619 4405 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-wbzfc" event={"ID":"4f76b032-61eb-4796-b082-47c23ff73d18","Type":"ContainerStarted","Data":"2719426310af07136ebfa5e501329a1b7bf5c3693276625c958889db4034d9c3"} Nov 04 02:33:10.843884 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:33:10.843653 4405 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-wbzfc" Nov 04 02:33:10.845951 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:33:10.845913 4405 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-6dcc9f78d9-9t79d" event={"ID":"1722b2b5-068d-4327-9eeb-ae14421dab0b","Type":"ContainerStarted","Data":"e81623eb384296114748f9f359e4f037ddcb2b7461aa86f6da5e39013cfb8c88"} Nov 04 02:33:10.846093 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:33:10.845956 4405 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-6dcc9f78d9-9t79d" event={"ID":"1722b2b5-068d-4327-9eeb-ae14421dab0b","Type":"ContainerStarted","Data":"dff58bfcd3fd927db4d683b4cc362544b372295f8729bf07779c93cce2f85229"} Nov 04 02:33:10.846093 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:33:10.846049 4405 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-6dcc9f78d9-9t79d" Nov 04 02:33:10.851387 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:33:10.851363 4405 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-47rh6\" (UniqueName: \"kubernetes.io/projected/31f4e8d6-73e9-4c32-aa32-68995576b7c7-kube-api-access-47rh6\") pod \"downloads-6dd789c8b7-sxjs8\" (UID: \"31f4e8d6-73e9-4c32-aa32-68995576b7c7\") " pod="openshift-console/downloads-6dd789c8b7-sxjs8" Nov 04 02:33:10.851387 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:33:10.851391 4405 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/31f4e8d6-73e9-4c32-aa32-68995576b7c7-tmp\") pod \"downloads-6dd789c8b7-sxjs8\" (UID: \"31f4e8d6-73e9-4c32-aa32-68995576b7c7\") " pod="openshift-console/downloads-6dd789c8b7-sxjs8" Nov 04 02:33:10.851832 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:33:10.851812 4405 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/31f4e8d6-73e9-4c32-aa32-68995576b7c7-tmp\") pod \"downloads-6dd789c8b7-sxjs8\" (UID: \"31f4e8d6-73e9-4c32-aa32-68995576b7c7\") " pod="openshift-console/downloads-6dd789c8b7-sxjs8" Nov 04 02:33:10.915813 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:33:10.915772 4405 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-47rh6\" (UniqueName: \"kubernetes.io/projected/31f4e8d6-73e9-4c32-aa32-68995576b7c7-kube-api-access-47rh6\") pod \"downloads-6dd789c8b7-sxjs8\" (UID: \"31f4e8d6-73e9-4c32-aa32-68995576b7c7\") " pod="openshift-console/downloads-6dd789c8b7-sxjs8" Nov 04 02:33:11.079435 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:33:11.079370 4405 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-image-registry/image-registry-6dcc9f78d9-9t79d" podStartSLOduration=2.079352323 podStartE2EDuration="2.079352323s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 02:33:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 02:33:11.079042632 +0000 UTC m=+37.509742766" watchObservedRunningTime="2026-11-04 02:33:11.079352323 +0000 UTC m=+37.510052444" Nov 04 02:33:11.198801 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:33:11.198707 4405 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-6dd789c8b7-sxjs8" Nov 04 02:33:11.632097 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:33:11.632061 4405 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-6dd789c8b7-sxjs8"] Nov 04 02:33:11.635499 ip-10-0-134-125 kubenswrapper[4405]: W1104 02:33:11.635470 4405 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod31f4e8d6_73e9_4c32_aa32_68995576b7c7.slice/crio-6eac57409f73c6301b4ba63eaea2eba92a77045a8aedb00ac3ccc53e25946595 WatchSource:0}: Error finding container 6eac57409f73c6301b4ba63eaea2eba92a77045a8aedb00ac3ccc53e25946595: Status 404 returned error can't find the container with id 6eac57409f73c6301b4ba63eaea2eba92a77045a8aedb00ac3ccc53e25946595 Nov 04 02:33:11.854843 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:33:11.851160 4405 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-5f2vb" event={"ID":"4acab1fc-c591-475a-a15f-790be8131547","Type":"ContainerStarted","Data":"fdc33ad7b66e45a9d2e9e67f253c8a0dd01ae99871d44d64bf011d7e0ed829af"} Nov 04 02:33:11.854843 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:33:11.853436 4405 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-6dd789c8b7-sxjs8" event={"ID":"31f4e8d6-73e9-4c32-aa32-68995576b7c7","Type":"ContainerStarted","Data":"6eac57409f73c6301b4ba63eaea2eba92a77045a8aedb00ac3ccc53e25946595"} Nov 04 02:33:12.827454 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:33:12.827421 4405 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-9ckn2"] Nov 04 02:33:12.832775 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:33:12.832748 4405 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-9ckn2" Nov 04 02:33:12.887649 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:33:12.887616 4405 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-dockercfg-pltl9\"" Nov 04 02:33:12.887887 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:33:12.887869 4405 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-tls\"" Nov 04 02:33:12.888180 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:33:12.888159 4405 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-9ckn2"] Nov 04 02:33:12.969294 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:33:12.968960 4405 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/514441fd-195f-4e0a-8313-4d1047fe6b02-tls-certificates\") pod \"prometheus-operator-admission-webhook-6489d8bf89-9ckn2\" (UID: \"514441fd-195f-4e0a-8313-4d1047fe6b02\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-9ckn2" Nov 04 02:33:13.070168 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:33:13.070130 4405 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/514441fd-195f-4e0a-8313-4d1047fe6b02-tls-certificates\") pod \"prometheus-operator-admission-webhook-6489d8bf89-9ckn2\" (UID: \"514441fd-195f-4e0a-8313-4d1047fe6b02\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-9ckn2" Nov 04 02:33:13.072938 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:33:13.072900 4405 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/514441fd-195f-4e0a-8313-4d1047fe6b02-tls-certificates\") pod \"prometheus-operator-admission-webhook-6489d8bf89-9ckn2\" (UID: \"514441fd-195f-4e0a-8313-4d1047fe6b02\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-9ckn2" Nov 04 02:33:13.143987 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:33:13.143948 4405 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-9ckn2" Nov 04 02:33:13.511732 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:33:13.511698 4405 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-9ckn2"] Nov 04 02:33:13.514922 ip-10-0-134-125 kubenswrapper[4405]: W1104 02:33:13.514880 4405 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod514441fd_195f_4e0a_8313_4d1047fe6b02.slice/crio-cbb3917112dfa12217bc383c2537d755a0ee1cbd8d85f0a258b458810440cdda WatchSource:0}: Error finding container cbb3917112dfa12217bc383c2537d755a0ee1cbd8d85f0a258b458810440cdda: Status 404 returned error can't find the container with id cbb3917112dfa12217bc383c2537d755a0ee1cbd8d85f0a258b458810440cdda Nov 04 02:33:13.860003 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:33:13.859964 4405 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-9ckn2" event={"ID":"514441fd-195f-4e0a-8313-4d1047fe6b02","Type":"ContainerStarted","Data":"cbb3917112dfa12217bc383c2537d755a0ee1cbd8d85f0a258b458810440cdda"} Nov 04 02:33:16.872395 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:33:16.872351 4405 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-9ckn2" event={"ID":"514441fd-195f-4e0a-8313-4d1047fe6b02","Type":"ContainerStarted","Data":"002c7fffc27392ad17b6e9d8579678c5d1228ec838d99ca83999d4546e9f744a"} Nov 04 02:33:16.872915 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:33:16.872553 4405 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-9ckn2" Nov 04 02:33:16.877063 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:33:16.877042 4405 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-9ckn2" Nov 04 02:33:16.987276 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:33:16.987214 4405 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-9ckn2" podStartSLOduration=2.552895145 podStartE2EDuration="4.987201075s" totalImagesPullingTime="2.43430593s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 02:33:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 02:33:16.986819473 +0000 UTC m=+43.417519594" watchObservedRunningTime="2026-11-04 02:33:16.987201075 +0000 UTC m=+43.417901194" Nov 04 02:33:20.856419 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:33:20.856354 4405 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-wbzfc" Nov 04 02:33:26.115646 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:33:26.115610 4405 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-296tb"] Nov 04 02:33:26.117779 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:33:26.117757 4405 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-296tb" Nov 04 02:33:26.171151 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:33:26.171112 4405 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Nov 04 02:33:26.171151 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:33:26.171154 4405 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Nov 04 02:33:26.171151 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:33:26.171164 4405 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" Nov 04 02:33:26.221120 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:33:26.221079 4405 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-mpgnq\"" Nov 04 02:33:26.221316 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:33:26.221137 4405 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" Nov 04 02:33:26.221316 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:33:26.221140 4405 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" Nov 04 02:33:26.221591 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:33:26.221576 4405 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" Nov 04 02:33:26.273368 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:33:26.273329 4405 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/2226b455-2db8-4e6e-875b-9f3f05947bf9-node-exporter-tls\") pod \"node-exporter-296tb\" (UID: \"2226b455-2db8-4e6e-875b-9f3f05947bf9\") " pod="openshift-monitoring/node-exporter-296tb" Nov 04 02:33:26.273557 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:33:26.273387 4405 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-97jqb\" (UniqueName: \"kubernetes.io/projected/2226b455-2db8-4e6e-875b-9f3f05947bf9-kube-api-access-97jqb\") pod \"node-exporter-296tb\" (UID: \"2226b455-2db8-4e6e-875b-9f3f05947bf9\") " pod="openshift-monitoring/node-exporter-296tb" Nov 04 02:33:26.273557 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:33:26.273442 4405 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/2226b455-2db8-4e6e-875b-9f3f05947bf9-node-exporter-wtmp\") pod \"node-exporter-296tb\" (UID: \"2226b455-2db8-4e6e-875b-9f3f05947bf9\") " pod="openshift-monitoring/node-exporter-296tb" Nov 04 02:33:26.273557 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:33:26.273471 4405 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/2226b455-2db8-4e6e-875b-9f3f05947bf9-node-exporter-accelerators-collector-config\") pod \"node-exporter-296tb\" (UID: \"2226b455-2db8-4e6e-875b-9f3f05947bf9\") " pod="openshift-monitoring/node-exporter-296tb" Nov 04 02:33:26.273557 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:33:26.273500 4405 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/2226b455-2db8-4e6e-875b-9f3f05947bf9-sys\") pod \"node-exporter-296tb\" (UID: \"2226b455-2db8-4e6e-875b-9f3f05947bf9\") " pod="openshift-monitoring/node-exporter-296tb" Nov 04 02:33:26.273557 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:33:26.273517 4405 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/2226b455-2db8-4e6e-875b-9f3f05947bf9-metrics-client-ca\") pod \"node-exporter-296tb\" (UID: \"2226b455-2db8-4e6e-875b-9f3f05947bf9\") " pod="openshift-monitoring/node-exporter-296tb" Nov 04 02:33:26.273557 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:33:26.273534 4405 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/2226b455-2db8-4e6e-875b-9f3f05947bf9-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-296tb\" (UID: \"2226b455-2db8-4e6e-875b-9f3f05947bf9\") " pod="openshift-monitoring/node-exporter-296tb" Nov 04 02:33:26.273874 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:33:26.273561 4405 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/2226b455-2db8-4e6e-875b-9f3f05947bf9-root\") pod \"node-exporter-296tb\" (UID: \"2226b455-2db8-4e6e-875b-9f3f05947bf9\") " pod="openshift-monitoring/node-exporter-296tb" Nov 04 02:33:26.273874 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:33:26.273581 4405 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/2226b455-2db8-4e6e-875b-9f3f05947bf9-node-exporter-textfile\") pod \"node-exporter-296tb\" (UID: \"2226b455-2db8-4e6e-875b-9f3f05947bf9\") " pod="openshift-monitoring/node-exporter-296tb" Nov 04 02:33:26.374152 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:33:26.374060 4405 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/2226b455-2db8-4e6e-875b-9f3f05947bf9-node-exporter-tls\") pod \"node-exporter-296tb\" (UID: \"2226b455-2db8-4e6e-875b-9f3f05947bf9\") " pod="openshift-monitoring/node-exporter-296tb" Nov 04 02:33:26.374152 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:33:26.374117 4405 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-97jqb\" (UniqueName: \"kubernetes.io/projected/2226b455-2db8-4e6e-875b-9f3f05947bf9-kube-api-access-97jqb\") pod \"node-exporter-296tb\" (UID: \"2226b455-2db8-4e6e-875b-9f3f05947bf9\") " pod="openshift-monitoring/node-exporter-296tb" Nov 04 02:33:26.374152 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:33:26.374142 4405 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/2226b455-2db8-4e6e-875b-9f3f05947bf9-node-exporter-wtmp\") pod \"node-exporter-296tb\" (UID: \"2226b455-2db8-4e6e-875b-9f3f05947bf9\") " pod="openshift-monitoring/node-exporter-296tb" Nov 04 02:33:26.374435 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:33:26.374166 4405 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/2226b455-2db8-4e6e-875b-9f3f05947bf9-node-exporter-accelerators-collector-config\") pod \"node-exporter-296tb\" (UID: \"2226b455-2db8-4e6e-875b-9f3f05947bf9\") " pod="openshift-monitoring/node-exporter-296tb" Nov 04 02:33:26.374435 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:33:26.374194 4405 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/2226b455-2db8-4e6e-875b-9f3f05947bf9-sys\") pod \"node-exporter-296tb\" (UID: \"2226b455-2db8-4e6e-875b-9f3f05947bf9\") " pod="openshift-monitoring/node-exporter-296tb" Nov 04 02:33:26.374435 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:33:26.374217 4405 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/2226b455-2db8-4e6e-875b-9f3f05947bf9-metrics-client-ca\") pod \"node-exporter-296tb\" (UID: \"2226b455-2db8-4e6e-875b-9f3f05947bf9\") " pod="openshift-monitoring/node-exporter-296tb" Nov 04 02:33:26.375359 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:33:26.375330 4405 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/2226b455-2db8-4e6e-875b-9f3f05947bf9-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-296tb\" (UID: \"2226b455-2db8-4e6e-875b-9f3f05947bf9\") " pod="openshift-monitoring/node-exporter-296tb" Nov 04 02:33:26.375691 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:33:26.375411 4405 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/2226b455-2db8-4e6e-875b-9f3f05947bf9-root\") pod \"node-exporter-296tb\" (UID: \"2226b455-2db8-4e6e-875b-9f3f05947bf9\") " pod="openshift-monitoring/node-exporter-296tb" Nov 04 02:33:26.375691 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:33:26.375447 4405 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/2226b455-2db8-4e6e-875b-9f3f05947bf9-node-exporter-textfile\") pod \"node-exporter-296tb\" (UID: \"2226b455-2db8-4e6e-875b-9f3f05947bf9\") " pod="openshift-monitoring/node-exporter-296tb" Nov 04 02:33:26.375831 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:33:26.375770 4405 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/2226b455-2db8-4e6e-875b-9f3f05947bf9-root\") pod \"node-exporter-296tb\" (UID: \"2226b455-2db8-4e6e-875b-9f3f05947bf9\") " pod="openshift-monitoring/node-exporter-296tb" Nov 04 02:33:26.375890 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:33:26.375829 4405 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/2226b455-2db8-4e6e-875b-9f3f05947bf9-node-exporter-textfile\") pod \"node-exporter-296tb\" (UID: \"2226b455-2db8-4e6e-875b-9f3f05947bf9\") " pod="openshift-monitoring/node-exporter-296tb" Nov 04 02:33:26.375890 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:33:26.375850 4405 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/2226b455-2db8-4e6e-875b-9f3f05947bf9-sys\") pod \"node-exporter-296tb\" (UID: \"2226b455-2db8-4e6e-875b-9f3f05947bf9\") " pod="openshift-monitoring/node-exporter-296tb" Nov 04 02:33:26.376369 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:33:26.376346 4405 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/2226b455-2db8-4e6e-875b-9f3f05947bf9-node-exporter-accelerators-collector-config\") pod \"node-exporter-296tb\" (UID: \"2226b455-2db8-4e6e-875b-9f3f05947bf9\") " pod="openshift-monitoring/node-exporter-296tb" Nov 04 02:33:26.376473 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:33:26.376463 4405 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/2226b455-2db8-4e6e-875b-9f3f05947bf9-node-exporter-wtmp\") pod \"node-exporter-296tb\" (UID: \"2226b455-2db8-4e6e-875b-9f3f05947bf9\") " pod="openshift-monitoring/node-exporter-296tb" Nov 04 02:33:26.376534 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:33:26.376477 4405 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/2226b455-2db8-4e6e-875b-9f3f05947bf9-metrics-client-ca\") pod \"node-exporter-296tb\" (UID: \"2226b455-2db8-4e6e-875b-9f3f05947bf9\") " pod="openshift-monitoring/node-exporter-296tb" Nov 04 02:33:26.381165 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:33:26.381130 4405 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/2226b455-2db8-4e6e-875b-9f3f05947bf9-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-296tb\" (UID: \"2226b455-2db8-4e6e-875b-9f3f05947bf9\") " pod="openshift-monitoring/node-exporter-296tb" Nov 04 02:33:26.381934 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:33:26.381911 4405 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/2226b455-2db8-4e6e-875b-9f3f05947bf9-node-exporter-tls\") pod \"node-exporter-296tb\" (UID: \"2226b455-2db8-4e6e-875b-9f3f05947bf9\") " pod="openshift-monitoring/node-exporter-296tb" Nov 04 02:33:26.436052 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:33:26.436017 4405 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-97jqb\" (UniqueName: \"kubernetes.io/projected/2226b455-2db8-4e6e-875b-9f3f05947bf9-kube-api-access-97jqb\") pod \"node-exporter-296tb\" (UID: \"2226b455-2db8-4e6e-875b-9f3f05947bf9\") " pod="openshift-monitoring/node-exporter-296tb" Nov 04 02:33:26.730725 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:33:26.730635 4405 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-296tb" Nov 04 02:33:26.740696 ip-10-0-134-125 kubenswrapper[4405]: W1104 02:33:26.740643 4405 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2226b455_2db8_4e6e_875b_9f3f05947bf9.slice/crio-9fbf6ed1bdb1c076a9fd12e9c808279598fdde615d822a691b2eb8392e1222bf WatchSource:0}: Error finding container 9fbf6ed1bdb1c076a9fd12e9c808279598fdde615d822a691b2eb8392e1222bf: Status 404 returned error can't find the container with id 9fbf6ed1bdb1c076a9fd12e9c808279598fdde615d822a691b2eb8392e1222bf Nov 04 02:33:26.911637 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:33:26.911596 4405 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-296tb" event={"ID":"2226b455-2db8-4e6e-875b-9f3f05947bf9","Type":"ContainerStarted","Data":"9fbf6ed1bdb1c076a9fd12e9c808279598fdde615d822a691b2eb8392e1222bf"} Nov 04 02:33:29.513779 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:33:29.513738 4405 patch_prober.go:29] interesting pod/image-registry-6dcc9f78d9-9t79d container/registry namespace/openshift-image-registry: Liveness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]} Nov 04 02:33:29.514215 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:33:29.513819 4405 prober.go:122] "Probe failed" probeType="Liveness" pod="openshift-image-registry/image-registry-6dcc9f78d9-9t79d" podUID="1722b2b5-068d-4327-9eeb-ae14421dab0b" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 04 02:33:29.924869 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:33:29.924830 4405 generic.go:356] "Generic (PLEG): container finished" podID="2226b455-2db8-4e6e-875b-9f3f05947bf9" containerID="6cc57704997cdf2e21f3439be1ad76e652ea5b31a3a155de1b07dca41649b800" exitCode=0 Nov 04 02:33:29.925050 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:33:29.924888 4405 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-296tb" event={"ID":"2226b455-2db8-4e6e-875b-9f3f05947bf9","Type":"ContainerDied","Data":"6cc57704997cdf2e21f3439be1ad76e652ea5b31a3a155de1b07dca41649b800"} Nov 04 02:33:30.931374 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:33:30.931339 4405 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-296tb" event={"ID":"2226b455-2db8-4e6e-875b-9f3f05947bf9","Type":"ContainerStarted","Data":"39f63483e2e2989604a0617c3e04ea8134a6bf8fc2842c5dab8a041603f2d8da"} Nov 04 02:33:30.931876 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:33:30.931402 4405 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-296tb" event={"ID":"2226b455-2db8-4e6e-875b-9f3f05947bf9","Type":"ContainerStarted","Data":"f8e83c0337a5d44cf53a253c06d990aa6227cec00f1be200f84bef55a93adf39"} Nov 04 02:33:31.051432 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:33:31.051365 4405 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-296tb" podStartSLOduration=2.791850777 podStartE2EDuration="5.051350137s" totalImagesPullingTime="2.176389584s" totalInitContainerRuntime="83.109776ms" isStatefulPod=true podCreationTimestamp="2026-11-04 02:33:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 02:33:31.05121002 +0000 UTC m=+57.481910159" watchObservedRunningTime="2026-11-04 02:33:31.051350137 +0000 UTC m=+57.482050249" Nov 04 02:33:31.936818 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:33:31.936769 4405 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-6dd789c8b7-sxjs8" event={"ID":"31f4e8d6-73e9-4c32-aa32-68995576b7c7","Type":"ContainerStarted","Data":"1795d1b9861ccab2470976884c3fbee3f6ffc342c9f7d70df0b84038e1d5b9ce"} Nov 04 02:33:31.937351 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:33:31.937315 4405 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/downloads-6dd789c8b7-sxjs8" Nov 04 02:33:31.963104 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:33:31.963068 4405 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/downloads-6dd789c8b7-sxjs8" Nov 04 02:33:32.053631 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:33:32.053568 4405 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-console/downloads-6dd789c8b7-sxjs8" podStartSLOduration=2.490529795 podStartE2EDuration="22.053552729s" totalImagesPullingTime="19.563022934s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 02:33:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 02:33:32.052898845 +0000 UTC m=+58.483598967" watchObservedRunningTime="2026-11-04 02:33:32.053552729 +0000 UTC m=+58.484252850" Nov 04 02:33:32.861539 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:33:32.861502 4405 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-6dcc9f78d9-9t79d" Nov 04 02:33:35.388407 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:33:35.388369 4405 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Nov 04 02:33:35.423049 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:33:35.423020 4405 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Nov 04 02:33:35.445451 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:33:35.445420 4405 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ff565080-edb4-4987-8bf2-8fd76a3c3696-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"ff565080-edb4-4987-8bf2-8fd76a3c3696\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 04 02:33:35.445586 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:33:35.445464 4405 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/ff565080-edb4-4987-8bf2-8fd76a3c3696-config\") pod \"prometheus-k8s-0\" (UID: \"ff565080-edb4-4987-8bf2-8fd76a3c3696\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 04 02:33:35.445586 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:33:35.445499 4405 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ff565080-edb4-4987-8bf2-8fd76a3c3696-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"ff565080-edb4-4987-8bf2-8fd76a3c3696\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 04 02:33:35.445723 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:33:35.445575 4405 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/ff565080-edb4-4987-8bf2-8fd76a3c3696-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"ff565080-edb4-4987-8bf2-8fd76a3c3696\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 04 02:33:35.445723 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:33:35.445679 4405 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ff565080-edb4-4987-8bf2-8fd76a3c3696-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"ff565080-edb4-4987-8bf2-8fd76a3c3696\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 04 02:33:35.445723 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:33:35.445708 4405 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/ff565080-edb4-4987-8bf2-8fd76a3c3696-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"ff565080-edb4-4987-8bf2-8fd76a3c3696\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 04 02:33:35.445882 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:33:35.445738 4405 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/ff565080-edb4-4987-8bf2-8fd76a3c3696-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"ff565080-edb4-4987-8bf2-8fd76a3c3696\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 04 02:33:35.445882 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:33:35.445762 4405 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/ff565080-edb4-4987-8bf2-8fd76a3c3696-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"ff565080-edb4-4987-8bf2-8fd76a3c3696\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 04 02:33:35.445882 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:33:35.445798 4405 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/ff565080-edb4-4987-8bf2-8fd76a3c3696-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"ff565080-edb4-4987-8bf2-8fd76a3c3696\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 04 02:33:35.445882 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:33:35.445829 4405 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nwzmm\" (UniqueName: \"kubernetes.io/projected/ff565080-edb4-4987-8bf2-8fd76a3c3696-kube-api-access-nwzmm\") pod \"prometheus-k8s-0\" (UID: \"ff565080-edb4-4987-8bf2-8fd76a3c3696\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 04 02:33:35.445882 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:33:35.445850 4405 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/ff565080-edb4-4987-8bf2-8fd76a3c3696-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"ff565080-edb4-4987-8bf2-8fd76a3c3696\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 04 02:33:35.446085 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:33:35.445909 4405 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/ff565080-edb4-4987-8bf2-8fd76a3c3696-config-out\") pod \"prometheus-k8s-0\" (UID: \"ff565080-edb4-4987-8bf2-8fd76a3c3696\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 04 02:33:35.446085 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:33:35.445939 4405 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/ff565080-edb4-4987-8bf2-8fd76a3c3696-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"ff565080-edb4-4987-8bf2-8fd76a3c3696\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 04 02:33:35.446085 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:33:35.445962 4405 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/ff565080-edb4-4987-8bf2-8fd76a3c3696-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"ff565080-edb4-4987-8bf2-8fd76a3c3696\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 04 02:33:35.446085 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:33:35.446004 4405 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/ff565080-edb4-4987-8bf2-8fd76a3c3696-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"ff565080-edb4-4987-8bf2-8fd76a3c3696\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 04 02:33:35.446085 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:33:35.446047 4405 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/ff565080-edb4-4987-8bf2-8fd76a3c3696-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"ff565080-edb4-4987-8bf2-8fd76a3c3696\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 04 02:33:35.446324 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:33:35.446149 4405 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/ff565080-edb4-4987-8bf2-8fd76a3c3696-web-config\") pod \"prometheus-k8s-0\" (UID: \"ff565080-edb4-4987-8bf2-8fd76a3c3696\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 04 02:33:35.446324 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:33:35.446193 4405 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/ff565080-edb4-4987-8bf2-8fd76a3c3696-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"ff565080-edb4-4987-8bf2-8fd76a3c3696\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 04 02:33:35.446324 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:33:35.446213 4405 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/ff565080-edb4-4987-8bf2-8fd76a3c3696-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"ff565080-edb4-4987-8bf2-8fd76a3c3696\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 04 02:33:35.446324 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:33:35.446227 4405 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ff565080-edb4-4987-8bf2-8fd76a3c3696-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"ff565080-edb4-4987-8bf2-8fd76a3c3696\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 04 02:33:35.450319 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:33:35.450291 4405 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Nov 04 02:33:35.478910 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:33:35.478828 4405 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\"" Nov 04 02:33:35.478910 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:33:35.478903 4405 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\"" Nov 04 02:33:35.479148 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:33:35.479003 4405 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\"" Nov 04 02:33:35.498256 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:33:35.498225 4405 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\"" Nov 04 02:33:35.498437 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:33:35.498293 4405 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-f2s9t2mks6q85\"" Nov 04 02:33:35.498437 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:33:35.498310 4405 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\"" Nov 04 02:33:35.499922 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:33:35.499899 4405 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\"" Nov 04 02:33:35.500330 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:33:35.500301 4405 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\"" Nov 04 02:33:35.500432 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:33:35.500368 4405 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\"" Nov 04 02:33:35.500482 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:33:35.500439 4405 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\"" Nov 04 02:33:35.500723 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:33:35.500705 4405 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-7krrq\"" Nov 04 02:33:35.547490 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:33:35.547447 4405 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ff565080-edb4-4987-8bf2-8fd76a3c3696-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"ff565080-edb4-4987-8bf2-8fd76a3c3696\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 04 02:33:35.547490 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:33:35.547497 4405 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/ff565080-edb4-4987-8bf2-8fd76a3c3696-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"ff565080-edb4-4987-8bf2-8fd76a3c3696\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 04 02:33:35.547748 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:33:35.547530 4405 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ff565080-edb4-4987-8bf2-8fd76a3c3696-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"ff565080-edb4-4987-8bf2-8fd76a3c3696\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 04 02:33:35.547748 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:33:35.547548 4405 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/ff565080-edb4-4987-8bf2-8fd76a3c3696-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"ff565080-edb4-4987-8bf2-8fd76a3c3696\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 04 02:33:35.547748 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:33:35.547576 4405 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/ff565080-edb4-4987-8bf2-8fd76a3c3696-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"ff565080-edb4-4987-8bf2-8fd76a3c3696\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 04 02:33:35.547748 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:33:35.547602 4405 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/ff565080-edb4-4987-8bf2-8fd76a3c3696-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"ff565080-edb4-4987-8bf2-8fd76a3c3696\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 04 02:33:35.547748 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:33:35.547719 4405 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/ff565080-edb4-4987-8bf2-8fd76a3c3696-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"ff565080-edb4-4987-8bf2-8fd76a3c3696\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 04 02:33:35.548012 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:33:35.547773 4405 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nwzmm\" (UniqueName: \"kubernetes.io/projected/ff565080-edb4-4987-8bf2-8fd76a3c3696-kube-api-access-nwzmm\") pod \"prometheus-k8s-0\" (UID: \"ff565080-edb4-4987-8bf2-8fd76a3c3696\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 04 02:33:35.548012 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:33:35.547801 4405 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/ff565080-edb4-4987-8bf2-8fd76a3c3696-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"ff565080-edb4-4987-8bf2-8fd76a3c3696\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 04 02:33:35.548012 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:33:35.547832 4405 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/ff565080-edb4-4987-8bf2-8fd76a3c3696-config-out\") pod \"prometheus-k8s-0\" (UID: \"ff565080-edb4-4987-8bf2-8fd76a3c3696\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 04 02:33:35.548012 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:33:35.547865 4405 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/ff565080-edb4-4987-8bf2-8fd76a3c3696-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"ff565080-edb4-4987-8bf2-8fd76a3c3696\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 04 02:33:35.548012 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:33:35.547902 4405 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/ff565080-edb4-4987-8bf2-8fd76a3c3696-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"ff565080-edb4-4987-8bf2-8fd76a3c3696\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 04 02:33:35.548012 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:33:35.547932 4405 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/ff565080-edb4-4987-8bf2-8fd76a3c3696-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"ff565080-edb4-4987-8bf2-8fd76a3c3696\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 04 02:33:35.548012 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:33:35.547954 4405 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/ff565080-edb4-4987-8bf2-8fd76a3c3696-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"ff565080-edb4-4987-8bf2-8fd76a3c3696\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 04 02:33:35.548012 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:33:35.547983 4405 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/ff565080-edb4-4987-8bf2-8fd76a3c3696-web-config\") pod \"prometheus-k8s-0\" (UID: \"ff565080-edb4-4987-8bf2-8fd76a3c3696\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 04 02:33:35.548012 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:33:35.548010 4405 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/ff565080-edb4-4987-8bf2-8fd76a3c3696-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"ff565080-edb4-4987-8bf2-8fd76a3c3696\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 04 02:33:35.548481 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:33:35.548043 4405 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/ff565080-edb4-4987-8bf2-8fd76a3c3696-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"ff565080-edb4-4987-8bf2-8fd76a3c3696\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 04 02:33:35.548481 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:33:35.548073 4405 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ff565080-edb4-4987-8bf2-8fd76a3c3696-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"ff565080-edb4-4987-8bf2-8fd76a3c3696\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 04 02:33:35.548481 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:33:35.548112 4405 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ff565080-edb4-4987-8bf2-8fd76a3c3696-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"ff565080-edb4-4987-8bf2-8fd76a3c3696\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 04 02:33:35.548481 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:33:35.548149 4405 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/ff565080-edb4-4987-8bf2-8fd76a3c3696-config\") pod \"prometheus-k8s-0\" (UID: \"ff565080-edb4-4987-8bf2-8fd76a3c3696\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 04 02:33:35.551775 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:33:35.551730 4405 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ff565080-edb4-4987-8bf2-8fd76a3c3696-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"ff565080-edb4-4987-8bf2-8fd76a3c3696\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 04 02:33:35.551918 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:33:35.551896 4405 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/ff565080-edb4-4987-8bf2-8fd76a3c3696-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"ff565080-edb4-4987-8bf2-8fd76a3c3696\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 04 02:33:35.552316 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:33:35.552292 4405 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/ff565080-edb4-4987-8bf2-8fd76a3c3696-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"ff565080-edb4-4987-8bf2-8fd76a3c3696\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 04 02:33:35.552447 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:33:35.552358 4405 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\"" Nov 04 02:33:35.552447 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:33:35.552383 4405 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\"" Nov 04 02:33:35.552569 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:33:35.552528 4405 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\"" Nov 04 02:33:35.552662 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:33:35.552577 4405 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\"" Nov 04 02:33:35.554172 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:33:35.554147 4405 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/ff565080-edb4-4987-8bf2-8fd76a3c3696-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"ff565080-edb4-4987-8bf2-8fd76a3c3696\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 04 02:33:35.554278 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:33:35.554227 4405 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/ff565080-edb4-4987-8bf2-8fd76a3c3696-web-config\") pod \"prometheus-k8s-0\" (UID: \"ff565080-edb4-4987-8bf2-8fd76a3c3696\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 04 02:33:35.555155 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:33:35.555132 4405 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/ff565080-edb4-4987-8bf2-8fd76a3c3696-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"ff565080-edb4-4987-8bf2-8fd76a3c3696\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 04 02:33:35.556139 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:33:35.556096 4405 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/ff565080-edb4-4987-8bf2-8fd76a3c3696-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"ff565080-edb4-4987-8bf2-8fd76a3c3696\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 04 02:33:35.556419 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:33:35.556393 4405 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/ff565080-edb4-4987-8bf2-8fd76a3c3696-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"ff565080-edb4-4987-8bf2-8fd76a3c3696\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 04 02:33:35.556724 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:33:35.556702 4405 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/ff565080-edb4-4987-8bf2-8fd76a3c3696-config-out\") pod \"prometheus-k8s-0\" (UID: \"ff565080-edb4-4987-8bf2-8fd76a3c3696\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 04 02:33:35.556923 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:33:35.556899 4405 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/ff565080-edb4-4987-8bf2-8fd76a3c3696-config\") pod \"prometheus-k8s-0\" (UID: \"ff565080-edb4-4987-8bf2-8fd76a3c3696\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 04 02:33:35.557171 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:33:35.557151 4405 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/ff565080-edb4-4987-8bf2-8fd76a3c3696-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"ff565080-edb4-4987-8bf2-8fd76a3c3696\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 04 02:33:35.557171 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:33:35.557161 4405 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/ff565080-edb4-4987-8bf2-8fd76a3c3696-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"ff565080-edb4-4987-8bf2-8fd76a3c3696\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 04 02:33:35.558592 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:33:35.558522 4405 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/ff565080-edb4-4987-8bf2-8fd76a3c3696-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"ff565080-edb4-4987-8bf2-8fd76a3c3696\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 04 02:33:35.558737 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:33:35.558567 4405 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/ff565080-edb4-4987-8bf2-8fd76a3c3696-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"ff565080-edb4-4987-8bf2-8fd76a3c3696\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 04 02:33:35.558737 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:33:35.558728 4405 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ff565080-edb4-4987-8bf2-8fd76a3c3696-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"ff565080-edb4-4987-8bf2-8fd76a3c3696\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 04 02:33:35.558863 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:33:35.558837 4405 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ff565080-edb4-4987-8bf2-8fd76a3c3696-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"ff565080-edb4-4987-8bf2-8fd76a3c3696\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 04 02:33:35.563117 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:33:35.563093 4405 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/ff565080-edb4-4987-8bf2-8fd76a3c3696-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"ff565080-edb4-4987-8bf2-8fd76a3c3696\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 04 02:33:35.652124 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:33:35.652007 4405 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\"" Nov 04 02:33:35.652124 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:33:35.652039 4405 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\"" Nov 04 02:33:35.658345 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:33:35.658314 4405 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nwzmm\" (UniqueName: \"kubernetes.io/projected/ff565080-edb4-4987-8bf2-8fd76a3c3696-kube-api-access-nwzmm\") pod \"prometheus-k8s-0\" (UID: \"ff565080-edb4-4987-8bf2-8fd76a3c3696\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 04 02:33:35.662184 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:33:35.662153 4405 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ff565080-edb4-4987-8bf2-8fd76a3c3696-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"ff565080-edb4-4987-8bf2-8fd76a3c3696\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 04 02:33:35.663854 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:33:35.663824 4405 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/ff565080-edb4-4987-8bf2-8fd76a3c3696-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"ff565080-edb4-4987-8bf2-8fd76a3c3696\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 04 02:33:35.903088 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:33:35.902784 4405 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Nov 04 02:33:36.420356 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:33:36.420320 4405 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Nov 04 02:33:36.473509 ip-10-0-134-125 kubenswrapper[4405]: W1104 02:33:36.473471 4405 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podff565080_edb4_4987_8bf2_8fd76a3c3696.slice/crio-8539051e8eef45b4e96bcf48b3d6596802f92b17627843b7a0950151c3e2430d WatchSource:0}: Error finding container 8539051e8eef45b4e96bcf48b3d6596802f92b17627843b7a0950151c3e2430d: Status 404 returned error can't find the container with id 8539051e8eef45b4e96bcf48b3d6596802f92b17627843b7a0950151c3e2430d Nov 04 02:33:36.888040 ip-10-0-134-125 kubenswrapper[4405]: E1104 02:33:36.887998 4405 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5f2vb" containerName="extractor" Nov 04 02:33:36.957608 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:33:36.957571 4405 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"ff565080-edb4-4987-8bf2-8fd76a3c3696","Type":"ContainerStarted","Data":"8539051e8eef45b4e96bcf48b3d6596802f92b17627843b7a0950151c3e2430d"} Nov 04 02:33:39.970573 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:33:39.970533 4405 generic.go:356] "Generic (PLEG): container finished" podID="ff565080-edb4-4987-8bf2-8fd76a3c3696" containerID="1856620b7ed80e6e96e591e7bc8009aa615026ba675e9dbebacc87d052a4d823" exitCode=0 Nov 04 02:33:39.971128 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:33:39.970595 4405 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"ff565080-edb4-4987-8bf2-8fd76a3c3696","Type":"ContainerDied","Data":"1856620b7ed80e6e96e591e7bc8009aa615026ba675e9dbebacc87d052a4d823"} Nov 04 02:33:45.992621 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:33:45.992588 4405 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"ff565080-edb4-4987-8bf2-8fd76a3c3696","Type":"ContainerStarted","Data":"8f7c5c483efcc8a8769e063784aa3352ee565ffcf710265b449178b2e39493c7"} Nov 04 02:33:45.993073 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:33:45.992684 4405 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"ff565080-edb4-4987-8bf2-8fd76a3c3696","Type":"ContainerStarted","Data":"94083bdd106ee1adad62a5140b1b6e76596332a99575ed3a144a8978e6403a33"} Nov 04 02:33:50.010974 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:33:50.010937 4405 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"ff565080-edb4-4987-8bf2-8fd76a3c3696","Type":"ContainerStarted","Data":"34d8acd5c631eb2cfa7515fc3afe152421515f7fc4b84da45d47342b1b979a3c"} Nov 04 02:33:50.011413 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:33:50.011009 4405 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"ff565080-edb4-4987-8bf2-8fd76a3c3696","Type":"ContainerStarted","Data":"e08e18635618341698eb474d35609acf121b19b0d0a643613b256eeddc286a42"} Nov 04 02:33:50.011413 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:33:50.011034 4405 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"ff565080-edb4-4987-8bf2-8fd76a3c3696","Type":"ContainerStarted","Data":"2e72f353444898f97a6408532c2a811cc6e161a970e452952faaa961e55d8251"} Nov 04 02:33:50.011413 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:33:50.011056 4405 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"ff565080-edb4-4987-8bf2-8fd76a3c3696","Type":"ContainerStarted","Data":"03d1a710be2e730d83a9410b6b478034ff1186f04055c3fd7266c55972b4556a"} Nov 04 02:33:50.295953 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:33:50.295851 4405 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=3.380301592 podStartE2EDuration="15.295836226s" totalImagesPullingTime="11.89824131s" totalInitContainerRuntime="17.293324ms" isStatefulPod=false podCreationTimestamp="2026-11-04 02:33:35 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 02:33:50.295340139 +0000 UTC m=+76.726040258" watchObservedRunningTime="2026-11-04 02:33:50.295836226 +0000 UTC m=+76.726536369" Nov 04 02:33:50.903930 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:33:50.903894 4405 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0" Nov 04 02:33:55.501458 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:33:55.501420 4405 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-insights/periodic-gathering-hxwj2-xcshl"] Nov 04 02:33:55.501916 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:33:55.501640 4405 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-insights/periodic-gathering-hxwj2-xcshl" podUID="5a5c4b25-7a35-4290-ad1e-04b4a3617877" containerName="insights-gathering" containerID="cri-o://4aac51ec380c8a9549949608c65315e45b58f4f5f73f99f05954b2c8b6f625ad" gracePeriod=30 Nov 04 02:33:55.750506 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:33:55.750481 4405 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-hxwj2-xcshl" Nov 04 02:33:55.821581 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:33:55.821498 4405 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5a5c4b25-7a35-4290-ad1e-04b4a3617877-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5a5c4b25-7a35-4290-ad1e-04b4a3617877-trusted-ca-bundle\") pod \"5a5c4b25-7a35-4290-ad1e-04b4a3617877\" (UID: \"5a5c4b25-7a35-4290-ad1e-04b4a3617877\") " Nov 04 02:33:55.821581 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:33:55.821541 4405 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5a5c4b25-7a35-4290-ad1e-04b4a3617877-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5a5c4b25-7a35-4290-ad1e-04b4a3617877-service-ca-bundle\") pod \"5a5c4b25-7a35-4290-ad1e-04b4a3617877\" (UID: \"5a5c4b25-7a35-4290-ad1e-04b4a3617877\") " Nov 04 02:33:55.821813 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:33:55.821584 4405 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5a5c4b25-7a35-4290-ad1e-04b4a3617877-archives-path\" (UniqueName: \"kubernetes.io/empty-dir/5a5c4b25-7a35-4290-ad1e-04b4a3617877-archives-path\") pod \"5a5c4b25-7a35-4290-ad1e-04b4a3617877\" (UID: \"5a5c4b25-7a35-4290-ad1e-04b4a3617877\") " Nov 04 02:33:55.821813 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:33:55.821629 4405 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5a5c4b25-7a35-4290-ad1e-04b4a3617877-kube-api-access-h288h\" (UniqueName: \"kubernetes.io/projected/5a5c4b25-7a35-4290-ad1e-04b4a3617877-kube-api-access-h288h\") pod \"5a5c4b25-7a35-4290-ad1e-04b4a3617877\" (UID: \"5a5c4b25-7a35-4290-ad1e-04b4a3617877\") " Nov 04 02:33:55.821913 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:33:55.821898 4405 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5a5c4b25-7a35-4290-ad1e-04b4a3617877-trusted-ca-bundle" pod "5a5c4b25-7a35-4290-ad1e-04b4a3617877" (UID: "5a5c4b25-7a35-4290-ad1e-04b4a3617877"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 02:33:55.822005 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:33:55.821983 4405 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5a5c4b25-7a35-4290-ad1e-04b4a3617877-service-ca-bundle" pod "5a5c4b25-7a35-4290-ad1e-04b4a3617877" (UID: "5a5c4b25-7a35-4290-ad1e-04b4a3617877"). InnerVolumeSpecName "service-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 02:33:55.822042 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:33:55.821983 4405 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5a5c4b25-7a35-4290-ad1e-04b4a3617877-archives-path" pod "5a5c4b25-7a35-4290-ad1e-04b4a3617877" (UID: "5a5c4b25-7a35-4290-ad1e-04b4a3617877"). InnerVolumeSpecName "archives-path". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 02:33:55.824125 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:33:55.824105 4405 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5a5c4b25-7a35-4290-ad1e-04b4a3617877-kube-api-access-h288h" pod "5a5c4b25-7a35-4290-ad1e-04b4a3617877" (UID: "5a5c4b25-7a35-4290-ad1e-04b4a3617877"). InnerVolumeSpecName "kube-api-access-h288h". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 02:33:55.922818 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:33:55.922782 4405 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5a5c4b25-7a35-4290-ad1e-04b4a3617877-trusted-ca-bundle\") on node \"ip-10-0-134-125.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 02:33:55.922818 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:33:55.922809 4405 reconciler_common.go:299] "Volume detached for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5a5c4b25-7a35-4290-ad1e-04b4a3617877-service-ca-bundle\") on node \"ip-10-0-134-125.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 02:33:55.922818 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:33:55.922819 4405 reconciler_common.go:299] "Volume detached for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/5a5c4b25-7a35-4290-ad1e-04b4a3617877-archives-path\") on node \"ip-10-0-134-125.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 02:33:55.922818 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:33:55.922827 4405 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h288h\" (UniqueName: \"kubernetes.io/projected/5a5c4b25-7a35-4290-ad1e-04b4a3617877-kube-api-access-h288h\") on node \"ip-10-0-134-125.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 02:33:56.031545 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:33:56.031513 4405 generic.go:356] "Generic (PLEG): container finished" podID="5a5c4b25-7a35-4290-ad1e-04b4a3617877" containerID="4aac51ec380c8a9549949608c65315e45b58f4f5f73f99f05954b2c8b6f625ad" exitCode=2 Nov 04 02:33:56.031735 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:33:56.031561 4405 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-hxwj2-xcshl" Nov 04 02:33:56.031735 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:33:56.031596 4405 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-hxwj2-xcshl" event={"ID":"5a5c4b25-7a35-4290-ad1e-04b4a3617877","Type":"ContainerDied","Data":"4aac51ec380c8a9549949608c65315e45b58f4f5f73f99f05954b2c8b6f625ad"} Nov 04 02:33:56.031735 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:33:56.031637 4405 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-hxwj2-xcshl" event={"ID":"5a5c4b25-7a35-4290-ad1e-04b4a3617877","Type":"ContainerDied","Data":"7f29a771e6ee4b7e85aa9aa1134769a08873c5be8f3c263500189b2051fd67b8"} Nov 04 02:33:56.031735 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:33:56.031691 4405 scope.go:122] "RemoveContainer" containerID="4aac51ec380c8a9549949608c65315e45b58f4f5f73f99f05954b2c8b6f625ad" Nov 04 02:33:56.040296 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:33:56.040279 4405 scope.go:122] "RemoveContainer" containerID="4aac51ec380c8a9549949608c65315e45b58f4f5f73f99f05954b2c8b6f625ad" Nov 04 02:33:56.040558 ip-10-0-134-125 kubenswrapper[4405]: E1104 02:33:56.040538 4405 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4aac51ec380c8a9549949608c65315e45b58f4f5f73f99f05954b2c8b6f625ad\": container with ID starting with 4aac51ec380c8a9549949608c65315e45b58f4f5f73f99f05954b2c8b6f625ad not found: ID does not exist" containerID="4aac51ec380c8a9549949608c65315e45b58f4f5f73f99f05954b2c8b6f625ad" Nov 04 02:33:56.040621 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:33:56.040569 4405 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4aac51ec380c8a9549949608c65315e45b58f4f5f73f99f05954b2c8b6f625ad"} err="failed to get container status \"4aac51ec380c8a9549949608c65315e45b58f4f5f73f99f05954b2c8b6f625ad\": rpc error: code = NotFound desc = could not find container \"4aac51ec380c8a9549949608c65315e45b58f4f5f73f99f05954b2c8b6f625ad\": container with ID starting with 4aac51ec380c8a9549949608c65315e45b58f4f5f73f99f05954b2c8b6f625ad not found: ID does not exist" Nov 04 02:33:56.194215 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:33:56.194180 4405 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-insights/periodic-gathering-hxwj2-xcshl"] Nov 04 02:33:56.197912 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:33:56.197890 4405 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-insights/periodic-gathering-hxwj2-xcshl"] Nov 04 02:33:57.698177 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:33:57.698143 4405 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5a5c4b25-7a35-4290-ad1e-04b4a3617877" path="/var/lib/kubelet/pods/5a5c4b25-7a35-4290-ad1e-04b4a3617877/volumes" Nov 04 02:34:06.886560 ip-10-0-134-125 kubenswrapper[4405]: E1104 02:34:06.886519 4405 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5f2vb" containerName="extractor" Nov 04 02:34:35.902933 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:34:35.902895 4405 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0" Nov 04 02:34:35.922004 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:34:35.921979 4405 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0" Nov 04 02:34:36.176365 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:34:36.176292 4405 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0" Nov 04 02:34:36.882443 ip-10-0-134-125 kubenswrapper[4405]: E1104 02:34:36.882412 4405 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5f2vb" containerName="extractor" Nov 04 02:34:57.936594 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:34:57.936504 4405 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Nov 04 02:34:57.937216 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:34:57.936892 4405 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="ff565080-edb4-4987-8bf2-8fd76a3c3696" containerName="prometheus" containerID="cri-o://94083bdd106ee1adad62a5140b1b6e76596332a99575ed3a144a8978e6403a33" gracePeriod=600 Nov 04 02:34:57.937216 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:34:57.936939 4405 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="ff565080-edb4-4987-8bf2-8fd76a3c3696" containerName="kube-rbac-proxy-thanos" containerID="cri-o://34d8acd5c631eb2cfa7515fc3afe152421515f7fc4b84da45d47342b1b979a3c" gracePeriod=600 Nov 04 02:34:57.937216 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:34:57.936957 4405 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="ff565080-edb4-4987-8bf2-8fd76a3c3696" containerName="config-reloader" containerID="cri-o://8f7c5c483efcc8a8769e063784aa3352ee565ffcf710265b449178b2e39493c7" gracePeriod=600 Nov 04 02:34:57.937216 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:34:57.936906 4405 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="ff565080-edb4-4987-8bf2-8fd76a3c3696" containerName="kube-rbac-proxy" containerID="cri-o://e08e18635618341698eb474d35609acf121b19b0d0a643613b256eeddc286a42" gracePeriod=600 Nov 04 02:34:57.937216 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:34:57.937059 4405 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="ff565080-edb4-4987-8bf2-8fd76a3c3696" containerName="thanos-sidecar" containerID="cri-o://03d1a710be2e730d83a9410b6b478034ff1186f04055c3fd7266c55972b4556a" gracePeriod=600 Nov 04 02:34:57.937216 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:34:57.937160 4405 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="ff565080-edb4-4987-8bf2-8fd76a3c3696" containerName="kube-rbac-proxy-web" containerID="cri-o://2e72f353444898f97a6408532c2a811cc6e161a970e452952faaa961e55d8251" gracePeriod=600 Nov 04 02:34:58.180348 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:34:58.180324 4405 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Nov 04 02:34:58.238774 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:34:58.238687 4405 generic.go:356] "Generic (PLEG): container finished" podID="ff565080-edb4-4987-8bf2-8fd76a3c3696" containerID="34d8acd5c631eb2cfa7515fc3afe152421515f7fc4b84da45d47342b1b979a3c" exitCode=0 Nov 04 02:34:58.238774 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:34:58.238719 4405 generic.go:356] "Generic (PLEG): container finished" podID="ff565080-edb4-4987-8bf2-8fd76a3c3696" containerID="e08e18635618341698eb474d35609acf121b19b0d0a643613b256eeddc286a42" exitCode=0 Nov 04 02:34:58.238774 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:34:58.238732 4405 generic.go:356] "Generic (PLEG): container finished" podID="ff565080-edb4-4987-8bf2-8fd76a3c3696" containerID="2e72f353444898f97a6408532c2a811cc6e161a970e452952faaa961e55d8251" exitCode=0 Nov 04 02:34:58.238774 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:34:58.238744 4405 generic.go:356] "Generic (PLEG): container finished" podID="ff565080-edb4-4987-8bf2-8fd76a3c3696" containerID="03d1a710be2e730d83a9410b6b478034ff1186f04055c3fd7266c55972b4556a" exitCode=0 Nov 04 02:34:58.238774 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:34:58.238752 4405 generic.go:356] "Generic (PLEG): container finished" podID="ff565080-edb4-4987-8bf2-8fd76a3c3696" containerID="8f7c5c483efcc8a8769e063784aa3352ee565ffcf710265b449178b2e39493c7" exitCode=0 Nov 04 02:34:58.238774 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:34:58.238759 4405 generic.go:356] "Generic (PLEG): container finished" podID="ff565080-edb4-4987-8bf2-8fd76a3c3696" containerID="94083bdd106ee1adad62a5140b1b6e76596332a99575ed3a144a8978e6403a33" exitCode=0 Nov 04 02:34:58.239089 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:34:58.238770 4405 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"ff565080-edb4-4987-8bf2-8fd76a3c3696","Type":"ContainerDied","Data":"34d8acd5c631eb2cfa7515fc3afe152421515f7fc4b84da45d47342b1b979a3c"} Nov 04 02:34:58.239089 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:34:58.238789 4405 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Nov 04 02:34:58.239089 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:34:58.238842 4405 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"ff565080-edb4-4987-8bf2-8fd76a3c3696","Type":"ContainerDied","Data":"e08e18635618341698eb474d35609acf121b19b0d0a643613b256eeddc286a42"} Nov 04 02:34:58.239089 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:34:58.238869 4405 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"ff565080-edb4-4987-8bf2-8fd76a3c3696","Type":"ContainerDied","Data":"2e72f353444898f97a6408532c2a811cc6e161a970e452952faaa961e55d8251"} Nov 04 02:34:58.239089 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:34:58.238892 4405 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"ff565080-edb4-4987-8bf2-8fd76a3c3696","Type":"ContainerDied","Data":"03d1a710be2e730d83a9410b6b478034ff1186f04055c3fd7266c55972b4556a"} Nov 04 02:34:58.239089 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:34:58.238913 4405 scope.go:122] "RemoveContainer" containerID="34d8acd5c631eb2cfa7515fc3afe152421515f7fc4b84da45d47342b1b979a3c" Nov 04 02:34:58.239089 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:34:58.238917 4405 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"ff565080-edb4-4987-8bf2-8fd76a3c3696","Type":"ContainerDied","Data":"8f7c5c483efcc8a8769e063784aa3352ee565ffcf710265b449178b2e39493c7"} Nov 04 02:34:58.239089 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:34:58.238942 4405 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"ff565080-edb4-4987-8bf2-8fd76a3c3696","Type":"ContainerDied","Data":"94083bdd106ee1adad62a5140b1b6e76596332a99575ed3a144a8978e6403a33"} Nov 04 02:34:58.239089 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:34:58.238968 4405 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"ff565080-edb4-4987-8bf2-8fd76a3c3696","Type":"ContainerDied","Data":"8539051e8eef45b4e96bcf48b3d6596802f92b17627843b7a0950151c3e2430d"} Nov 04 02:34:58.246296 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:34:58.246275 4405 scope.go:122] "RemoveContainer" containerID="e08e18635618341698eb474d35609acf121b19b0d0a643613b256eeddc286a42" Nov 04 02:34:58.252844 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:34:58.252823 4405 scope.go:122] "RemoveContainer" containerID="2e72f353444898f97a6408532c2a811cc6e161a970e452952faaa961e55d8251" Nov 04 02:34:58.259292 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:34:58.259273 4405 scope.go:122] "RemoveContainer" containerID="03d1a710be2e730d83a9410b6b478034ff1186f04055c3fd7266c55972b4556a" Nov 04 02:34:58.265749 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:34:58.265734 4405 scope.go:122] "RemoveContainer" containerID="8f7c5c483efcc8a8769e063784aa3352ee565ffcf710265b449178b2e39493c7" Nov 04 02:34:58.272438 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:34:58.272418 4405 scope.go:122] "RemoveContainer" containerID="94083bdd106ee1adad62a5140b1b6e76596332a99575ed3a144a8978e6403a33" Nov 04 02:34:58.279520 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:34:58.279500 4405 scope.go:122] "RemoveContainer" containerID="1856620b7ed80e6e96e591e7bc8009aa615026ba675e9dbebacc87d052a4d823" Nov 04 02:34:58.288121 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:34:58.288102 4405 scope.go:122] "RemoveContainer" containerID="34d8acd5c631eb2cfa7515fc3afe152421515f7fc4b84da45d47342b1b979a3c" Nov 04 02:34:58.288408 ip-10-0-134-125 kubenswrapper[4405]: E1104 02:34:58.288391 4405 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"34d8acd5c631eb2cfa7515fc3afe152421515f7fc4b84da45d47342b1b979a3c\": container with ID starting with 34d8acd5c631eb2cfa7515fc3afe152421515f7fc4b84da45d47342b1b979a3c not found: ID does not exist" containerID="34d8acd5c631eb2cfa7515fc3afe152421515f7fc4b84da45d47342b1b979a3c" Nov 04 02:34:58.288463 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:34:58.288418 4405 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"34d8acd5c631eb2cfa7515fc3afe152421515f7fc4b84da45d47342b1b979a3c"} err="failed to get container status \"34d8acd5c631eb2cfa7515fc3afe152421515f7fc4b84da45d47342b1b979a3c\": rpc error: code = NotFound desc = could not find container \"34d8acd5c631eb2cfa7515fc3afe152421515f7fc4b84da45d47342b1b979a3c\": container with ID starting with 34d8acd5c631eb2cfa7515fc3afe152421515f7fc4b84da45d47342b1b979a3c not found: ID does not exist" Nov 04 02:34:58.288463 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:34:58.288438 4405 scope.go:122] "RemoveContainer" containerID="e08e18635618341698eb474d35609acf121b19b0d0a643613b256eeddc286a42" Nov 04 02:34:58.288656 ip-10-0-134-125 kubenswrapper[4405]: E1104 02:34:58.288641 4405 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e08e18635618341698eb474d35609acf121b19b0d0a643613b256eeddc286a42\": container with ID starting with e08e18635618341698eb474d35609acf121b19b0d0a643613b256eeddc286a42 not found: ID does not exist" containerID="e08e18635618341698eb474d35609acf121b19b0d0a643613b256eeddc286a42" Nov 04 02:34:58.288724 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:34:58.288661 4405 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e08e18635618341698eb474d35609acf121b19b0d0a643613b256eeddc286a42"} err="failed to get container status \"e08e18635618341698eb474d35609acf121b19b0d0a643613b256eeddc286a42\": rpc error: code = NotFound desc = could not find container \"e08e18635618341698eb474d35609acf121b19b0d0a643613b256eeddc286a42\": container with ID starting with e08e18635618341698eb474d35609acf121b19b0d0a643613b256eeddc286a42 not found: ID does not exist" Nov 04 02:34:58.288724 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:34:58.288699 4405 scope.go:122] "RemoveContainer" containerID="2e72f353444898f97a6408532c2a811cc6e161a970e452952faaa961e55d8251" Nov 04 02:34:58.288930 ip-10-0-134-125 kubenswrapper[4405]: E1104 02:34:58.288914 4405 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2e72f353444898f97a6408532c2a811cc6e161a970e452952faaa961e55d8251\": container with ID starting with 2e72f353444898f97a6408532c2a811cc6e161a970e452952faaa961e55d8251 not found: ID does not exist" containerID="2e72f353444898f97a6408532c2a811cc6e161a970e452952faaa961e55d8251" Nov 04 02:34:58.288972 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:34:58.288937 4405 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2e72f353444898f97a6408532c2a811cc6e161a970e452952faaa961e55d8251"} err="failed to get container status \"2e72f353444898f97a6408532c2a811cc6e161a970e452952faaa961e55d8251\": rpc error: code = NotFound desc = could not find container \"2e72f353444898f97a6408532c2a811cc6e161a970e452952faaa961e55d8251\": container with ID starting with 2e72f353444898f97a6408532c2a811cc6e161a970e452952faaa961e55d8251 not found: ID does not exist" Nov 04 02:34:58.288972 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:34:58.288950 4405 scope.go:122] "RemoveContainer" containerID="03d1a710be2e730d83a9410b6b478034ff1186f04055c3fd7266c55972b4556a" Nov 04 02:34:58.289199 ip-10-0-134-125 kubenswrapper[4405]: E1104 02:34:58.289186 4405 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"03d1a710be2e730d83a9410b6b478034ff1186f04055c3fd7266c55972b4556a\": container with ID starting with 03d1a710be2e730d83a9410b6b478034ff1186f04055c3fd7266c55972b4556a not found: ID does not exist" containerID="03d1a710be2e730d83a9410b6b478034ff1186f04055c3fd7266c55972b4556a" Nov 04 02:34:58.289238 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:34:58.289203 4405 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"03d1a710be2e730d83a9410b6b478034ff1186f04055c3fd7266c55972b4556a"} err="failed to get container status \"03d1a710be2e730d83a9410b6b478034ff1186f04055c3fd7266c55972b4556a\": rpc error: code = NotFound desc = could not find container \"03d1a710be2e730d83a9410b6b478034ff1186f04055c3fd7266c55972b4556a\": container with ID starting with 03d1a710be2e730d83a9410b6b478034ff1186f04055c3fd7266c55972b4556a not found: ID does not exist" Nov 04 02:34:58.289238 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:34:58.289214 4405 scope.go:122] "RemoveContainer" containerID="8f7c5c483efcc8a8769e063784aa3352ee565ffcf710265b449178b2e39493c7" Nov 04 02:34:58.289445 ip-10-0-134-125 kubenswrapper[4405]: E1104 02:34:58.289425 4405 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8f7c5c483efcc8a8769e063784aa3352ee565ffcf710265b449178b2e39493c7\": container with ID starting with 8f7c5c483efcc8a8769e063784aa3352ee565ffcf710265b449178b2e39493c7 not found: ID does not exist" containerID="8f7c5c483efcc8a8769e063784aa3352ee565ffcf710265b449178b2e39493c7" Nov 04 02:34:58.289484 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:34:58.289451 4405 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8f7c5c483efcc8a8769e063784aa3352ee565ffcf710265b449178b2e39493c7"} err="failed to get container status \"8f7c5c483efcc8a8769e063784aa3352ee565ffcf710265b449178b2e39493c7\": rpc error: code = NotFound desc = could not find container \"8f7c5c483efcc8a8769e063784aa3352ee565ffcf710265b449178b2e39493c7\": container with ID starting with 8f7c5c483efcc8a8769e063784aa3352ee565ffcf710265b449178b2e39493c7 not found: ID does not exist" Nov 04 02:34:58.289484 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:34:58.289467 4405 scope.go:122] "RemoveContainer" containerID="94083bdd106ee1adad62a5140b1b6e76596332a99575ed3a144a8978e6403a33" Nov 04 02:34:58.289747 ip-10-0-134-125 kubenswrapper[4405]: E1104 02:34:58.289731 4405 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"94083bdd106ee1adad62a5140b1b6e76596332a99575ed3a144a8978e6403a33\": container with ID starting with 94083bdd106ee1adad62a5140b1b6e76596332a99575ed3a144a8978e6403a33 not found: ID does not exist" containerID="94083bdd106ee1adad62a5140b1b6e76596332a99575ed3a144a8978e6403a33" Nov 04 02:34:58.289805 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:34:58.289752 4405 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"94083bdd106ee1adad62a5140b1b6e76596332a99575ed3a144a8978e6403a33"} err="failed to get container status \"94083bdd106ee1adad62a5140b1b6e76596332a99575ed3a144a8978e6403a33\": rpc error: code = NotFound desc = could not find container \"94083bdd106ee1adad62a5140b1b6e76596332a99575ed3a144a8978e6403a33\": container with ID starting with 94083bdd106ee1adad62a5140b1b6e76596332a99575ed3a144a8978e6403a33 not found: ID does not exist" Nov 04 02:34:58.289805 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:34:58.289765 4405 scope.go:122] "RemoveContainer" containerID="1856620b7ed80e6e96e591e7bc8009aa615026ba675e9dbebacc87d052a4d823" Nov 04 02:34:58.289967 ip-10-0-134-125 kubenswrapper[4405]: E1104 02:34:58.289951 4405 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1856620b7ed80e6e96e591e7bc8009aa615026ba675e9dbebacc87d052a4d823\": container with ID starting with 1856620b7ed80e6e96e591e7bc8009aa615026ba675e9dbebacc87d052a4d823 not found: ID does not exist" containerID="1856620b7ed80e6e96e591e7bc8009aa615026ba675e9dbebacc87d052a4d823" Nov 04 02:34:58.290018 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:34:58.289969 4405 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1856620b7ed80e6e96e591e7bc8009aa615026ba675e9dbebacc87d052a4d823"} err="failed to get container status \"1856620b7ed80e6e96e591e7bc8009aa615026ba675e9dbebacc87d052a4d823\": rpc error: code = NotFound desc = could not find container \"1856620b7ed80e6e96e591e7bc8009aa615026ba675e9dbebacc87d052a4d823\": container with ID starting with 1856620b7ed80e6e96e591e7bc8009aa615026ba675e9dbebacc87d052a4d823 not found: ID does not exist" Nov 04 02:34:58.290018 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:34:58.289980 4405 scope.go:122] "RemoveContainer" containerID="34d8acd5c631eb2cfa7515fc3afe152421515f7fc4b84da45d47342b1b979a3c" Nov 04 02:34:58.290174 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:34:58.290154 4405 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"34d8acd5c631eb2cfa7515fc3afe152421515f7fc4b84da45d47342b1b979a3c"} err="failed to get container status \"34d8acd5c631eb2cfa7515fc3afe152421515f7fc4b84da45d47342b1b979a3c\": rpc error: code = NotFound desc = could not find container \"34d8acd5c631eb2cfa7515fc3afe152421515f7fc4b84da45d47342b1b979a3c\": container with ID starting with 34d8acd5c631eb2cfa7515fc3afe152421515f7fc4b84da45d47342b1b979a3c not found: ID does not exist" Nov 04 02:34:58.290247 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:34:58.290176 4405 scope.go:122] "RemoveContainer" containerID="e08e18635618341698eb474d35609acf121b19b0d0a643613b256eeddc286a42" Nov 04 02:34:58.290378 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:34:58.290363 4405 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e08e18635618341698eb474d35609acf121b19b0d0a643613b256eeddc286a42"} err="failed to get container status \"e08e18635618341698eb474d35609acf121b19b0d0a643613b256eeddc286a42\": rpc error: code = NotFound desc = could not find container \"e08e18635618341698eb474d35609acf121b19b0d0a643613b256eeddc286a42\": container with ID starting with e08e18635618341698eb474d35609acf121b19b0d0a643613b256eeddc286a42 not found: ID does not exist" Nov 04 02:34:58.290444 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:34:58.290378 4405 scope.go:122] "RemoveContainer" containerID="2e72f353444898f97a6408532c2a811cc6e161a970e452952faaa961e55d8251" Nov 04 02:34:58.290539 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:34:58.290524 4405 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2e72f353444898f97a6408532c2a811cc6e161a970e452952faaa961e55d8251"} err="failed to get container status \"2e72f353444898f97a6408532c2a811cc6e161a970e452952faaa961e55d8251\": rpc error: code = NotFound desc = could not find container \"2e72f353444898f97a6408532c2a811cc6e161a970e452952faaa961e55d8251\": container with ID starting with 2e72f353444898f97a6408532c2a811cc6e161a970e452952faaa961e55d8251 not found: ID does not exist" Nov 04 02:34:58.290592 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:34:58.290539 4405 scope.go:122] "RemoveContainer" containerID="03d1a710be2e730d83a9410b6b478034ff1186f04055c3fd7266c55972b4556a" Nov 04 02:34:58.290756 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:34:58.290738 4405 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"03d1a710be2e730d83a9410b6b478034ff1186f04055c3fd7266c55972b4556a"} err="failed to get container status \"03d1a710be2e730d83a9410b6b478034ff1186f04055c3fd7266c55972b4556a\": rpc error: code = NotFound desc = could not find container \"03d1a710be2e730d83a9410b6b478034ff1186f04055c3fd7266c55972b4556a\": container with ID starting with 03d1a710be2e730d83a9410b6b478034ff1186f04055c3fd7266c55972b4556a not found: ID does not exist" Nov 04 02:34:58.290806 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:34:58.290757 4405 scope.go:122] "RemoveContainer" containerID="8f7c5c483efcc8a8769e063784aa3352ee565ffcf710265b449178b2e39493c7" Nov 04 02:34:58.290983 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:34:58.290968 4405 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8f7c5c483efcc8a8769e063784aa3352ee565ffcf710265b449178b2e39493c7"} err="failed to get container status \"8f7c5c483efcc8a8769e063784aa3352ee565ffcf710265b449178b2e39493c7\": rpc error: code = NotFound desc = could not find container \"8f7c5c483efcc8a8769e063784aa3352ee565ffcf710265b449178b2e39493c7\": container with ID starting with 8f7c5c483efcc8a8769e063784aa3352ee565ffcf710265b449178b2e39493c7 not found: ID does not exist" Nov 04 02:34:58.291031 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:34:58.290983 4405 scope.go:122] "RemoveContainer" containerID="94083bdd106ee1adad62a5140b1b6e76596332a99575ed3a144a8978e6403a33" Nov 04 02:34:58.291211 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:34:58.291192 4405 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"94083bdd106ee1adad62a5140b1b6e76596332a99575ed3a144a8978e6403a33"} err="failed to get container status \"94083bdd106ee1adad62a5140b1b6e76596332a99575ed3a144a8978e6403a33\": rpc error: code = NotFound desc = could not find container \"94083bdd106ee1adad62a5140b1b6e76596332a99575ed3a144a8978e6403a33\": container with ID starting with 94083bdd106ee1adad62a5140b1b6e76596332a99575ed3a144a8978e6403a33 not found: ID does not exist" Nov 04 02:34:58.291292 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:34:58.291212 4405 scope.go:122] "RemoveContainer" containerID="1856620b7ed80e6e96e591e7bc8009aa615026ba675e9dbebacc87d052a4d823" Nov 04 02:34:58.291466 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:34:58.291446 4405 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1856620b7ed80e6e96e591e7bc8009aa615026ba675e9dbebacc87d052a4d823"} err="failed to get container status \"1856620b7ed80e6e96e591e7bc8009aa615026ba675e9dbebacc87d052a4d823\": rpc error: code = NotFound desc = could not find container \"1856620b7ed80e6e96e591e7bc8009aa615026ba675e9dbebacc87d052a4d823\": container with ID starting with 1856620b7ed80e6e96e591e7bc8009aa615026ba675e9dbebacc87d052a4d823 not found: ID does not exist" Nov 04 02:34:58.291514 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:34:58.291468 4405 scope.go:122] "RemoveContainer" containerID="34d8acd5c631eb2cfa7515fc3afe152421515f7fc4b84da45d47342b1b979a3c" Nov 04 02:34:58.291693 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:34:58.291655 4405 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"34d8acd5c631eb2cfa7515fc3afe152421515f7fc4b84da45d47342b1b979a3c"} err="failed to get container status \"34d8acd5c631eb2cfa7515fc3afe152421515f7fc4b84da45d47342b1b979a3c\": rpc error: code = NotFound desc = could not find container \"34d8acd5c631eb2cfa7515fc3afe152421515f7fc4b84da45d47342b1b979a3c\": container with ID starting with 34d8acd5c631eb2cfa7515fc3afe152421515f7fc4b84da45d47342b1b979a3c not found: ID does not exist" Nov 04 02:34:58.291693 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:34:58.291688 4405 scope.go:122] "RemoveContainer" containerID="e08e18635618341698eb474d35609acf121b19b0d0a643613b256eeddc286a42" Nov 04 02:34:58.291903 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:34:58.291886 4405 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e08e18635618341698eb474d35609acf121b19b0d0a643613b256eeddc286a42"} err="failed to get container status \"e08e18635618341698eb474d35609acf121b19b0d0a643613b256eeddc286a42\": rpc error: code = NotFound desc = could not find container \"e08e18635618341698eb474d35609acf121b19b0d0a643613b256eeddc286a42\": container with ID starting with e08e18635618341698eb474d35609acf121b19b0d0a643613b256eeddc286a42 not found: ID does not exist" Nov 04 02:34:58.291954 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:34:58.291904 4405 scope.go:122] "RemoveContainer" containerID="2e72f353444898f97a6408532c2a811cc6e161a970e452952faaa961e55d8251" Nov 04 02:34:58.292117 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:34:58.292099 4405 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2e72f353444898f97a6408532c2a811cc6e161a970e452952faaa961e55d8251"} err="failed to get container status \"2e72f353444898f97a6408532c2a811cc6e161a970e452952faaa961e55d8251\": rpc error: code = NotFound desc = could not find container \"2e72f353444898f97a6408532c2a811cc6e161a970e452952faaa961e55d8251\": container with ID starting with 2e72f353444898f97a6408532c2a811cc6e161a970e452952faaa961e55d8251 not found: ID does not exist" Nov 04 02:34:58.292161 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:34:58.292117 4405 scope.go:122] "RemoveContainer" containerID="03d1a710be2e730d83a9410b6b478034ff1186f04055c3fd7266c55972b4556a" Nov 04 02:34:58.292320 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:34:58.292303 4405 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"03d1a710be2e730d83a9410b6b478034ff1186f04055c3fd7266c55972b4556a"} err="failed to get container status \"03d1a710be2e730d83a9410b6b478034ff1186f04055c3fd7266c55972b4556a\": rpc error: code = NotFound desc = could not find container \"03d1a710be2e730d83a9410b6b478034ff1186f04055c3fd7266c55972b4556a\": container with ID starting with 03d1a710be2e730d83a9410b6b478034ff1186f04055c3fd7266c55972b4556a not found: ID does not exist" Nov 04 02:34:58.292388 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:34:58.292321 4405 scope.go:122] "RemoveContainer" containerID="8f7c5c483efcc8a8769e063784aa3352ee565ffcf710265b449178b2e39493c7" Nov 04 02:34:58.292533 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:34:58.292517 4405 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8f7c5c483efcc8a8769e063784aa3352ee565ffcf710265b449178b2e39493c7"} err="failed to get container status \"8f7c5c483efcc8a8769e063784aa3352ee565ffcf710265b449178b2e39493c7\": rpc error: code = NotFound desc = could not find container \"8f7c5c483efcc8a8769e063784aa3352ee565ffcf710265b449178b2e39493c7\": container with ID starting with 8f7c5c483efcc8a8769e063784aa3352ee565ffcf710265b449178b2e39493c7 not found: ID does not exist" Nov 04 02:34:58.292580 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:34:58.292533 4405 scope.go:122] "RemoveContainer" containerID="94083bdd106ee1adad62a5140b1b6e76596332a99575ed3a144a8978e6403a33" Nov 04 02:34:58.292750 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:34:58.292735 4405 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"94083bdd106ee1adad62a5140b1b6e76596332a99575ed3a144a8978e6403a33"} err="failed to get container status \"94083bdd106ee1adad62a5140b1b6e76596332a99575ed3a144a8978e6403a33\": rpc error: code = NotFound desc = could not find container \"94083bdd106ee1adad62a5140b1b6e76596332a99575ed3a144a8978e6403a33\": container with ID starting with 94083bdd106ee1adad62a5140b1b6e76596332a99575ed3a144a8978e6403a33 not found: ID does not exist" Nov 04 02:34:58.292750 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:34:58.292750 4405 scope.go:122] "RemoveContainer" containerID="1856620b7ed80e6e96e591e7bc8009aa615026ba675e9dbebacc87d052a4d823" Nov 04 02:34:58.292960 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:34:58.292941 4405 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1856620b7ed80e6e96e591e7bc8009aa615026ba675e9dbebacc87d052a4d823"} err="failed to get container status \"1856620b7ed80e6e96e591e7bc8009aa615026ba675e9dbebacc87d052a4d823\": rpc error: code = NotFound desc = could not find container \"1856620b7ed80e6e96e591e7bc8009aa615026ba675e9dbebacc87d052a4d823\": container with ID starting with 1856620b7ed80e6e96e591e7bc8009aa615026ba675e9dbebacc87d052a4d823 not found: ID does not exist" Nov 04 02:34:58.293023 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:34:58.292961 4405 scope.go:122] "RemoveContainer" containerID="34d8acd5c631eb2cfa7515fc3afe152421515f7fc4b84da45d47342b1b979a3c" Nov 04 02:34:58.293169 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:34:58.293152 4405 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"34d8acd5c631eb2cfa7515fc3afe152421515f7fc4b84da45d47342b1b979a3c"} err="failed to get container status \"34d8acd5c631eb2cfa7515fc3afe152421515f7fc4b84da45d47342b1b979a3c\": rpc error: code = NotFound desc = could not find container \"34d8acd5c631eb2cfa7515fc3afe152421515f7fc4b84da45d47342b1b979a3c\": container with ID starting with 34d8acd5c631eb2cfa7515fc3afe152421515f7fc4b84da45d47342b1b979a3c not found: ID does not exist" Nov 04 02:34:58.293215 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:34:58.293169 4405 scope.go:122] "RemoveContainer" containerID="e08e18635618341698eb474d35609acf121b19b0d0a643613b256eeddc286a42" Nov 04 02:34:58.293350 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:34:58.293335 4405 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e08e18635618341698eb474d35609acf121b19b0d0a643613b256eeddc286a42"} err="failed to get container status \"e08e18635618341698eb474d35609acf121b19b0d0a643613b256eeddc286a42\": rpc error: code = NotFound desc = could not find container \"e08e18635618341698eb474d35609acf121b19b0d0a643613b256eeddc286a42\": container with ID starting with e08e18635618341698eb474d35609acf121b19b0d0a643613b256eeddc286a42 not found: ID does not exist" Nov 04 02:34:58.293350 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:34:58.293349 4405 scope.go:122] "RemoveContainer" containerID="2e72f353444898f97a6408532c2a811cc6e161a970e452952faaa961e55d8251" Nov 04 02:34:58.293526 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:34:58.293511 4405 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2e72f353444898f97a6408532c2a811cc6e161a970e452952faaa961e55d8251"} err="failed to get container status \"2e72f353444898f97a6408532c2a811cc6e161a970e452952faaa961e55d8251\": rpc error: code = NotFound desc = could not find container \"2e72f353444898f97a6408532c2a811cc6e161a970e452952faaa961e55d8251\": container with ID starting with 2e72f353444898f97a6408532c2a811cc6e161a970e452952faaa961e55d8251 not found: ID does not exist" Nov 04 02:34:58.293526 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:34:58.293525 4405 scope.go:122] "RemoveContainer" containerID="03d1a710be2e730d83a9410b6b478034ff1186f04055c3fd7266c55972b4556a" Nov 04 02:34:58.293724 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:34:58.293709 4405 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"03d1a710be2e730d83a9410b6b478034ff1186f04055c3fd7266c55972b4556a"} err="failed to get container status \"03d1a710be2e730d83a9410b6b478034ff1186f04055c3fd7266c55972b4556a\": rpc error: code = NotFound desc = could not find container \"03d1a710be2e730d83a9410b6b478034ff1186f04055c3fd7266c55972b4556a\": container with ID starting with 03d1a710be2e730d83a9410b6b478034ff1186f04055c3fd7266c55972b4556a not found: ID does not exist" Nov 04 02:34:58.293724 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:34:58.293723 4405 scope.go:122] "RemoveContainer" containerID="8f7c5c483efcc8a8769e063784aa3352ee565ffcf710265b449178b2e39493c7" Nov 04 02:34:58.293897 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:34:58.293881 4405 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8f7c5c483efcc8a8769e063784aa3352ee565ffcf710265b449178b2e39493c7"} err="failed to get container status \"8f7c5c483efcc8a8769e063784aa3352ee565ffcf710265b449178b2e39493c7\": rpc error: code = NotFound desc = could not find container \"8f7c5c483efcc8a8769e063784aa3352ee565ffcf710265b449178b2e39493c7\": container with ID starting with 8f7c5c483efcc8a8769e063784aa3352ee565ffcf710265b449178b2e39493c7 not found: ID does not exist" Nov 04 02:34:58.293939 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:34:58.293896 4405 scope.go:122] "RemoveContainer" containerID="94083bdd106ee1adad62a5140b1b6e76596332a99575ed3a144a8978e6403a33" Nov 04 02:34:58.294093 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:34:58.294076 4405 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"94083bdd106ee1adad62a5140b1b6e76596332a99575ed3a144a8978e6403a33"} err="failed to get container status \"94083bdd106ee1adad62a5140b1b6e76596332a99575ed3a144a8978e6403a33\": rpc error: code = NotFound desc = could not find container \"94083bdd106ee1adad62a5140b1b6e76596332a99575ed3a144a8978e6403a33\": container with ID starting with 94083bdd106ee1adad62a5140b1b6e76596332a99575ed3a144a8978e6403a33 not found: ID does not exist" Nov 04 02:34:58.294165 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:34:58.294095 4405 scope.go:122] "RemoveContainer" containerID="1856620b7ed80e6e96e591e7bc8009aa615026ba675e9dbebacc87d052a4d823" Nov 04 02:34:58.294296 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:34:58.294280 4405 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1856620b7ed80e6e96e591e7bc8009aa615026ba675e9dbebacc87d052a4d823"} err="failed to get container status \"1856620b7ed80e6e96e591e7bc8009aa615026ba675e9dbebacc87d052a4d823\": rpc error: code = NotFound desc = could not find container \"1856620b7ed80e6e96e591e7bc8009aa615026ba675e9dbebacc87d052a4d823\": container with ID starting with 1856620b7ed80e6e96e591e7bc8009aa615026ba675e9dbebacc87d052a4d823 not found: ID does not exist" Nov 04 02:34:58.294342 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:34:58.294295 4405 scope.go:122] "RemoveContainer" containerID="34d8acd5c631eb2cfa7515fc3afe152421515f7fc4b84da45d47342b1b979a3c" Nov 04 02:34:58.294474 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:34:58.294460 4405 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"34d8acd5c631eb2cfa7515fc3afe152421515f7fc4b84da45d47342b1b979a3c"} err="failed to get container status \"34d8acd5c631eb2cfa7515fc3afe152421515f7fc4b84da45d47342b1b979a3c\": rpc error: code = NotFound desc = could not find container \"34d8acd5c631eb2cfa7515fc3afe152421515f7fc4b84da45d47342b1b979a3c\": container with ID starting with 34d8acd5c631eb2cfa7515fc3afe152421515f7fc4b84da45d47342b1b979a3c not found: ID does not exist" Nov 04 02:34:58.294474 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:34:58.294473 4405 scope.go:122] "RemoveContainer" containerID="e08e18635618341698eb474d35609acf121b19b0d0a643613b256eeddc286a42" Nov 04 02:34:58.294723 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:34:58.294704 4405 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e08e18635618341698eb474d35609acf121b19b0d0a643613b256eeddc286a42"} err="failed to get container status \"e08e18635618341698eb474d35609acf121b19b0d0a643613b256eeddc286a42\": rpc error: code = NotFound desc = could not find container \"e08e18635618341698eb474d35609acf121b19b0d0a643613b256eeddc286a42\": container with ID starting with e08e18635618341698eb474d35609acf121b19b0d0a643613b256eeddc286a42 not found: ID does not exist" Nov 04 02:34:58.294792 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:34:58.294723 4405 scope.go:122] "RemoveContainer" containerID="2e72f353444898f97a6408532c2a811cc6e161a970e452952faaa961e55d8251" Nov 04 02:34:58.295022 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:34:58.294991 4405 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2e72f353444898f97a6408532c2a811cc6e161a970e452952faaa961e55d8251"} err="failed to get container status \"2e72f353444898f97a6408532c2a811cc6e161a970e452952faaa961e55d8251\": rpc error: code = NotFound desc = could not find container \"2e72f353444898f97a6408532c2a811cc6e161a970e452952faaa961e55d8251\": container with ID starting with 2e72f353444898f97a6408532c2a811cc6e161a970e452952faaa961e55d8251 not found: ID does not exist" Nov 04 02:34:58.295022 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:34:58.295018 4405 scope.go:122] "RemoveContainer" containerID="03d1a710be2e730d83a9410b6b478034ff1186f04055c3fd7266c55972b4556a" Nov 04 02:34:58.295349 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:34:58.295332 4405 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"03d1a710be2e730d83a9410b6b478034ff1186f04055c3fd7266c55972b4556a"} err="failed to get container status \"03d1a710be2e730d83a9410b6b478034ff1186f04055c3fd7266c55972b4556a\": rpc error: code = NotFound desc = could not find container \"03d1a710be2e730d83a9410b6b478034ff1186f04055c3fd7266c55972b4556a\": container with ID starting with 03d1a710be2e730d83a9410b6b478034ff1186f04055c3fd7266c55972b4556a not found: ID does not exist" Nov 04 02:34:58.295349 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:34:58.295349 4405 scope.go:122] "RemoveContainer" containerID="8f7c5c483efcc8a8769e063784aa3352ee565ffcf710265b449178b2e39493c7" Nov 04 02:34:58.295579 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:34:58.295561 4405 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8f7c5c483efcc8a8769e063784aa3352ee565ffcf710265b449178b2e39493c7"} err="failed to get container status \"8f7c5c483efcc8a8769e063784aa3352ee565ffcf710265b449178b2e39493c7\": rpc error: code = NotFound desc = could not find container \"8f7c5c483efcc8a8769e063784aa3352ee565ffcf710265b449178b2e39493c7\": container with ID starting with 8f7c5c483efcc8a8769e063784aa3352ee565ffcf710265b449178b2e39493c7 not found: ID does not exist" Nov 04 02:34:58.295645 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:34:58.295580 4405 scope.go:122] "RemoveContainer" containerID="94083bdd106ee1adad62a5140b1b6e76596332a99575ed3a144a8978e6403a33" Nov 04 02:34:58.295854 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:34:58.295830 4405 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"94083bdd106ee1adad62a5140b1b6e76596332a99575ed3a144a8978e6403a33"} err="failed to get container status \"94083bdd106ee1adad62a5140b1b6e76596332a99575ed3a144a8978e6403a33\": rpc error: code = NotFound desc = could not find container \"94083bdd106ee1adad62a5140b1b6e76596332a99575ed3a144a8978e6403a33\": container with ID starting with 94083bdd106ee1adad62a5140b1b6e76596332a99575ed3a144a8978e6403a33 not found: ID does not exist" Nov 04 02:34:58.295901 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:34:58.295855 4405 scope.go:122] "RemoveContainer" containerID="1856620b7ed80e6e96e591e7bc8009aa615026ba675e9dbebacc87d052a4d823" Nov 04 02:34:58.296073 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:34:58.296056 4405 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1856620b7ed80e6e96e591e7bc8009aa615026ba675e9dbebacc87d052a4d823"} err="failed to get container status \"1856620b7ed80e6e96e591e7bc8009aa615026ba675e9dbebacc87d052a4d823\": rpc error: code = NotFound desc = could not find container \"1856620b7ed80e6e96e591e7bc8009aa615026ba675e9dbebacc87d052a4d823\": container with ID starting with 1856620b7ed80e6e96e591e7bc8009aa615026ba675e9dbebacc87d052a4d823 not found: ID does not exist" Nov 04 02:34:58.296116 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:34:58.296075 4405 scope.go:122] "RemoveContainer" containerID="34d8acd5c631eb2cfa7515fc3afe152421515f7fc4b84da45d47342b1b979a3c" Nov 04 02:34:58.296286 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:34:58.296267 4405 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"34d8acd5c631eb2cfa7515fc3afe152421515f7fc4b84da45d47342b1b979a3c"} err="failed to get container status \"34d8acd5c631eb2cfa7515fc3afe152421515f7fc4b84da45d47342b1b979a3c\": rpc error: code = NotFound desc = could not find container \"34d8acd5c631eb2cfa7515fc3afe152421515f7fc4b84da45d47342b1b979a3c\": container with ID starting with 34d8acd5c631eb2cfa7515fc3afe152421515f7fc4b84da45d47342b1b979a3c not found: ID does not exist" Nov 04 02:34:58.296355 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:34:58.296287 4405 scope.go:122] "RemoveContainer" containerID="e08e18635618341698eb474d35609acf121b19b0d0a643613b256eeddc286a42" Nov 04 02:34:58.296516 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:34:58.296499 4405 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e08e18635618341698eb474d35609acf121b19b0d0a643613b256eeddc286a42"} err="failed to get container status \"e08e18635618341698eb474d35609acf121b19b0d0a643613b256eeddc286a42\": rpc error: code = NotFound desc = could not find container \"e08e18635618341698eb474d35609acf121b19b0d0a643613b256eeddc286a42\": container with ID starting with e08e18635618341698eb474d35609acf121b19b0d0a643613b256eeddc286a42 not found: ID does not exist" Nov 04 02:34:58.296566 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:34:58.296518 4405 scope.go:122] "RemoveContainer" containerID="2e72f353444898f97a6408532c2a811cc6e161a970e452952faaa961e55d8251" Nov 04 02:34:58.296797 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:34:58.296779 4405 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2e72f353444898f97a6408532c2a811cc6e161a970e452952faaa961e55d8251"} err="failed to get container status \"2e72f353444898f97a6408532c2a811cc6e161a970e452952faaa961e55d8251\": rpc error: code = NotFound desc = could not find container \"2e72f353444898f97a6408532c2a811cc6e161a970e452952faaa961e55d8251\": container with ID starting with 2e72f353444898f97a6408532c2a811cc6e161a970e452952faaa961e55d8251 not found: ID does not exist" Nov 04 02:34:58.296797 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:34:58.296797 4405 scope.go:122] "RemoveContainer" containerID="03d1a710be2e730d83a9410b6b478034ff1186f04055c3fd7266c55972b4556a" Nov 04 02:34:58.297036 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:34:58.297019 4405 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"03d1a710be2e730d83a9410b6b478034ff1186f04055c3fd7266c55972b4556a"} err="failed to get container status \"03d1a710be2e730d83a9410b6b478034ff1186f04055c3fd7266c55972b4556a\": rpc error: code = NotFound desc = could not find container \"03d1a710be2e730d83a9410b6b478034ff1186f04055c3fd7266c55972b4556a\": container with ID starting with 03d1a710be2e730d83a9410b6b478034ff1186f04055c3fd7266c55972b4556a not found: ID does not exist" Nov 04 02:34:58.297084 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:34:58.297036 4405 scope.go:122] "RemoveContainer" containerID="8f7c5c483efcc8a8769e063784aa3352ee565ffcf710265b449178b2e39493c7" Nov 04 02:34:58.297263 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:34:58.297244 4405 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8f7c5c483efcc8a8769e063784aa3352ee565ffcf710265b449178b2e39493c7"} err="failed to get container status \"8f7c5c483efcc8a8769e063784aa3352ee565ffcf710265b449178b2e39493c7\": rpc error: code = NotFound desc = could not find container \"8f7c5c483efcc8a8769e063784aa3352ee565ffcf710265b449178b2e39493c7\": container with ID starting with 8f7c5c483efcc8a8769e063784aa3352ee565ffcf710265b449178b2e39493c7 not found: ID does not exist" Nov 04 02:34:58.297303 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:34:58.297263 4405 scope.go:122] "RemoveContainer" containerID="94083bdd106ee1adad62a5140b1b6e76596332a99575ed3a144a8978e6403a33" Nov 04 02:34:58.297463 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:34:58.297449 4405 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"94083bdd106ee1adad62a5140b1b6e76596332a99575ed3a144a8978e6403a33"} err="failed to get container status \"94083bdd106ee1adad62a5140b1b6e76596332a99575ed3a144a8978e6403a33\": rpc error: code = NotFound desc = could not find container \"94083bdd106ee1adad62a5140b1b6e76596332a99575ed3a144a8978e6403a33\": container with ID starting with 94083bdd106ee1adad62a5140b1b6e76596332a99575ed3a144a8978e6403a33 not found: ID does not exist" Nov 04 02:34:58.297507 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:34:58.297464 4405 scope.go:122] "RemoveContainer" containerID="1856620b7ed80e6e96e591e7bc8009aa615026ba675e9dbebacc87d052a4d823" Nov 04 02:34:58.297658 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:34:58.297644 4405 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1856620b7ed80e6e96e591e7bc8009aa615026ba675e9dbebacc87d052a4d823"} err="failed to get container status \"1856620b7ed80e6e96e591e7bc8009aa615026ba675e9dbebacc87d052a4d823\": rpc error: code = NotFound desc = could not find container \"1856620b7ed80e6e96e591e7bc8009aa615026ba675e9dbebacc87d052a4d823\": container with ID starting with 1856620b7ed80e6e96e591e7bc8009aa615026ba675e9dbebacc87d052a4d823 not found: ID does not exist" Nov 04 02:34:58.301869 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:34:58.301853 4405 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ff565080-edb4-4987-8bf2-8fd76a3c3696-secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/ff565080-edb4-4987-8bf2-8fd76a3c3696-secret-prometheus-k8s-tls\") pod \"ff565080-edb4-4987-8bf2-8fd76a3c3696\" (UID: \"ff565080-edb4-4987-8bf2-8fd76a3c3696\") " Nov 04 02:34:58.301921 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:34:58.301880 4405 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ff565080-edb4-4987-8bf2-8fd76a3c3696-prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/ff565080-edb4-4987-8bf2-8fd76a3c3696-prometheus-k8s-rulefiles-1\") pod \"ff565080-edb4-4987-8bf2-8fd76a3c3696\" (UID: \"ff565080-edb4-4987-8bf2-8fd76a3c3696\") " Nov 04 02:34:58.301921 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:34:58.301910 4405 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ff565080-edb4-4987-8bf2-8fd76a3c3696-secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/ff565080-edb4-4987-8bf2-8fd76a3c3696-secret-metrics-client-certs\") pod \"ff565080-edb4-4987-8bf2-8fd76a3c3696\" (UID: \"ff565080-edb4-4987-8bf2-8fd76a3c3696\") " Nov 04 02:34:58.302025 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:34:58.301945 4405 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ff565080-edb4-4987-8bf2-8fd76a3c3696-secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/ff565080-edb4-4987-8bf2-8fd76a3c3696-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"ff565080-edb4-4987-8bf2-8fd76a3c3696\" (UID: \"ff565080-edb4-4987-8bf2-8fd76a3c3696\") " Nov 04 02:34:58.302025 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:34:58.302010 4405 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ff565080-edb4-4987-8bf2-8fd76a3c3696-tls-assets\" (UniqueName: \"kubernetes.io/projected/ff565080-edb4-4987-8bf2-8fd76a3c3696-tls-assets\") pod \"ff565080-edb4-4987-8bf2-8fd76a3c3696\" (UID: \"ff565080-edb4-4987-8bf2-8fd76a3c3696\") " Nov 04 02:34:58.302124 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:34:58.302040 4405 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ff565080-edb4-4987-8bf2-8fd76a3c3696-prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ff565080-edb4-4987-8bf2-8fd76a3c3696-prometheus-trusted-ca-bundle\") pod \"ff565080-edb4-4987-8bf2-8fd76a3c3696\" (UID: \"ff565080-edb4-4987-8bf2-8fd76a3c3696\") " Nov 04 02:34:58.302124 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:34:58.302068 4405 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ff565080-edb4-4987-8bf2-8fd76a3c3696-secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/ff565080-edb4-4987-8bf2-8fd76a3c3696-secret-grpc-tls\") pod \"ff565080-edb4-4987-8bf2-8fd76a3c3696\" (UID: \"ff565080-edb4-4987-8bf2-8fd76a3c3696\") " Nov 04 02:34:58.302124 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:34:58.302096 4405 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ff565080-edb4-4987-8bf2-8fd76a3c3696-configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ff565080-edb4-4987-8bf2-8fd76a3c3696-configmap-metrics-client-ca\") pod \"ff565080-edb4-4987-8bf2-8fd76a3c3696\" (UID: \"ff565080-edb4-4987-8bf2-8fd76a3c3696\") " Nov 04 02:34:58.302272 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:34:58.302122 4405 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ff565080-edb4-4987-8bf2-8fd76a3c3696-config-out\" (UniqueName: \"kubernetes.io/empty-dir/ff565080-edb4-4987-8bf2-8fd76a3c3696-config-out\") pod \"ff565080-edb4-4987-8bf2-8fd76a3c3696\" (UID: \"ff565080-edb4-4987-8bf2-8fd76a3c3696\") " Nov 04 02:34:58.302272 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:34:58.302148 4405 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ff565080-edb4-4987-8bf2-8fd76a3c3696-prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/ff565080-edb4-4987-8bf2-8fd76a3c3696-prometheus-k8s-db\") pod \"ff565080-edb4-4987-8bf2-8fd76a3c3696\" (UID: \"ff565080-edb4-4987-8bf2-8fd76a3c3696\") " Nov 04 02:34:58.302272 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:34:58.302181 4405 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ff565080-edb4-4987-8bf2-8fd76a3c3696-config\" (UniqueName: \"kubernetes.io/secret/ff565080-edb4-4987-8bf2-8fd76a3c3696-config\") pod \"ff565080-edb4-4987-8bf2-8fd76a3c3696\" (UID: \"ff565080-edb4-4987-8bf2-8fd76a3c3696\") " Nov 04 02:34:58.302272 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:34:58.302221 4405 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ff565080-edb4-4987-8bf2-8fd76a3c3696-prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/ff565080-edb4-4987-8bf2-8fd76a3c3696-prometheus-k8s-rulefiles-2\") pod \"ff565080-edb4-4987-8bf2-8fd76a3c3696\" (UID: \"ff565080-edb4-4987-8bf2-8fd76a3c3696\") " Nov 04 02:34:58.302272 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:34:58.302244 4405 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ff565080-edb4-4987-8bf2-8fd76a3c3696-prometheus-k8s-rulefiles-1" pod "ff565080-edb4-4987-8bf2-8fd76a3c3696" (UID: "ff565080-edb4-4987-8bf2-8fd76a3c3696"). InnerVolumeSpecName "prometheus-k8s-rulefiles-1". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 02:34:58.302272 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:34:58.302251 4405 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ff565080-edb4-4987-8bf2-8fd76a3c3696-prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/ff565080-edb4-4987-8bf2-8fd76a3c3696-prometheus-k8s-rulefiles-0\") pod \"ff565080-edb4-4987-8bf2-8fd76a3c3696\" (UID: \"ff565080-edb4-4987-8bf2-8fd76a3c3696\") " Nov 04 02:34:58.302568 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:34:58.302375 4405 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ff565080-edb4-4987-8bf2-8fd76a3c3696-configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ff565080-edb4-4987-8bf2-8fd76a3c3696-configmap-serving-certs-ca-bundle\") pod \"ff565080-edb4-4987-8bf2-8fd76a3c3696\" (UID: \"ff565080-edb4-4987-8bf2-8fd76a3c3696\") " Nov 04 02:34:58.302568 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:34:58.302413 4405 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ff565080-edb4-4987-8bf2-8fd76a3c3696-secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/ff565080-edb4-4987-8bf2-8fd76a3c3696-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"ff565080-edb4-4987-8bf2-8fd76a3c3696\" (UID: \"ff565080-edb4-4987-8bf2-8fd76a3c3696\") " Nov 04 02:34:58.302568 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:34:58.302444 4405 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ff565080-edb4-4987-8bf2-8fd76a3c3696-kube-api-access-nwzmm\" (UniqueName: \"kubernetes.io/projected/ff565080-edb4-4987-8bf2-8fd76a3c3696-kube-api-access-nwzmm\") pod \"ff565080-edb4-4987-8bf2-8fd76a3c3696\" (UID: \"ff565080-edb4-4987-8bf2-8fd76a3c3696\") " Nov 04 02:34:58.302568 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:34:58.302474 4405 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ff565080-edb4-4987-8bf2-8fd76a3c3696-web-config\" (UniqueName: \"kubernetes.io/secret/ff565080-edb4-4987-8bf2-8fd76a3c3696-web-config\") pod \"ff565080-edb4-4987-8bf2-8fd76a3c3696\" (UID: \"ff565080-edb4-4987-8bf2-8fd76a3c3696\") " Nov 04 02:34:58.302568 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:34:58.302512 4405 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ff565080-edb4-4987-8bf2-8fd76a3c3696-configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ff565080-edb4-4987-8bf2-8fd76a3c3696-configmap-kubelet-serving-ca-bundle\") pod \"ff565080-edb4-4987-8bf2-8fd76a3c3696\" (UID: \"ff565080-edb4-4987-8bf2-8fd76a3c3696\") " Nov 04 02:34:58.302568 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:34:58.302510 4405 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ff565080-edb4-4987-8bf2-8fd76a3c3696-prometheus-trusted-ca-bundle" pod "ff565080-edb4-4987-8bf2-8fd76a3c3696" (UID: "ff565080-edb4-4987-8bf2-8fd76a3c3696"). InnerVolumeSpecName "prometheus-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 02:34:58.302568 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:34:58.302561 4405 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ff565080-edb4-4987-8bf2-8fd76a3c3696-thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/ff565080-edb4-4987-8bf2-8fd76a3c3696-thanos-prometheus-http-client-file\") pod \"ff565080-edb4-4987-8bf2-8fd76a3c3696\" (UID: \"ff565080-edb4-4987-8bf2-8fd76a3c3696\") " Nov 04 02:34:58.302919 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:34:58.302600 4405 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ff565080-edb4-4987-8bf2-8fd76a3c3696-secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/ff565080-edb4-4987-8bf2-8fd76a3c3696-secret-kube-rbac-proxy\") pod \"ff565080-edb4-4987-8bf2-8fd76a3c3696\" (UID: \"ff565080-edb4-4987-8bf2-8fd76a3c3696\") " Nov 04 02:34:58.302919 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:34:58.302711 4405 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ff565080-edb4-4987-8bf2-8fd76a3c3696-configmap-metrics-client-ca" pod "ff565080-edb4-4987-8bf2-8fd76a3c3696" (UID: "ff565080-edb4-4987-8bf2-8fd76a3c3696"). InnerVolumeSpecName "configmap-metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 02:34:58.302919 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:34:58.302848 4405 reconciler_common.go:299] "Volume detached for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ff565080-edb4-4987-8bf2-8fd76a3c3696-prometheus-trusted-ca-bundle\") on node \"ip-10-0-134-125.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 02:34:58.302919 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:34:58.302866 4405 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/ff565080-edb4-4987-8bf2-8fd76a3c3696-prometheus-k8s-rulefiles-1\") on node \"ip-10-0-134-125.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 02:34:58.305099 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:34:58.303395 4405 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ff565080-edb4-4987-8bf2-8fd76a3c3696-configmap-kubelet-serving-ca-bundle" pod "ff565080-edb4-4987-8bf2-8fd76a3c3696" (UID: "ff565080-edb4-4987-8bf2-8fd76a3c3696"). InnerVolumeSpecName "configmap-kubelet-serving-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 02:34:58.305099 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:34:58.303869 4405 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ff565080-edb4-4987-8bf2-8fd76a3c3696-configmap-serving-certs-ca-bundle" pod "ff565080-edb4-4987-8bf2-8fd76a3c3696" (UID: "ff565080-edb4-4987-8bf2-8fd76a3c3696"). InnerVolumeSpecName "configmap-serving-certs-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 02:34:58.305099 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:34:58.304186 4405 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ff565080-edb4-4987-8bf2-8fd76a3c3696-prometheus-k8s-rulefiles-0" pod "ff565080-edb4-4987-8bf2-8fd76a3c3696" (UID: "ff565080-edb4-4987-8bf2-8fd76a3c3696"). InnerVolumeSpecName "prometheus-k8s-rulefiles-0". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 02:34:58.305099 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:34:58.304897 4405 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ff565080-edb4-4987-8bf2-8fd76a3c3696-prometheus-k8s-db" pod "ff565080-edb4-4987-8bf2-8fd76a3c3696" (UID: "ff565080-edb4-4987-8bf2-8fd76a3c3696"). InnerVolumeSpecName "prometheus-k8s-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 02:34:58.305099 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:34:58.304963 4405 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ff565080-edb4-4987-8bf2-8fd76a3c3696-prometheus-k8s-rulefiles-2" pod "ff565080-edb4-4987-8bf2-8fd76a3c3696" (UID: "ff565080-edb4-4987-8bf2-8fd76a3c3696"). InnerVolumeSpecName "prometheus-k8s-rulefiles-2". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 02:34:58.305383 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:34:58.305337 4405 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ff565080-edb4-4987-8bf2-8fd76a3c3696-secret-metrics-client-certs" pod "ff565080-edb4-4987-8bf2-8fd76a3c3696" (UID: "ff565080-edb4-4987-8bf2-8fd76a3c3696"). InnerVolumeSpecName "secret-metrics-client-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 04 02:34:58.306156 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:34:58.306086 4405 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ff565080-edb4-4987-8bf2-8fd76a3c3696-tls-assets" pod "ff565080-edb4-4987-8bf2-8fd76a3c3696" (UID: "ff565080-edb4-4987-8bf2-8fd76a3c3696"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 02:34:58.306276 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:34:58.306186 4405 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ff565080-edb4-4987-8bf2-8fd76a3c3696-secret-prometheus-k8s-tls" pod "ff565080-edb4-4987-8bf2-8fd76a3c3696" (UID: "ff565080-edb4-4987-8bf2-8fd76a3c3696"). InnerVolumeSpecName "secret-prometheus-k8s-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 04 02:34:58.306354 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:34:58.306285 4405 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ff565080-edb4-4987-8bf2-8fd76a3c3696-kube-api-access-nwzmm" pod "ff565080-edb4-4987-8bf2-8fd76a3c3696" (UID: "ff565080-edb4-4987-8bf2-8fd76a3c3696"). InnerVolumeSpecName "kube-api-access-nwzmm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 02:34:58.306608 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:34:58.306584 4405 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ff565080-edb4-4987-8bf2-8fd76a3c3696-secret-kube-rbac-proxy" pod "ff565080-edb4-4987-8bf2-8fd76a3c3696" (UID: "ff565080-edb4-4987-8bf2-8fd76a3c3696"). InnerVolumeSpecName "secret-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 04 02:34:58.306745 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:34:58.306713 4405 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ff565080-edb4-4987-8bf2-8fd76a3c3696-secret-prometheus-k8s-thanos-sidecar-tls" pod "ff565080-edb4-4987-8bf2-8fd76a3c3696" (UID: "ff565080-edb4-4987-8bf2-8fd76a3c3696"). InnerVolumeSpecName "secret-prometheus-k8s-thanos-sidecar-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 04 02:34:58.306958 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:34:58.306931 4405 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ff565080-edb4-4987-8bf2-8fd76a3c3696-config" pod "ff565080-edb4-4987-8bf2-8fd76a3c3696" (UID: "ff565080-edb4-4987-8bf2-8fd76a3c3696"). InnerVolumeSpecName "config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 04 02:34:58.307058 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:34:58.307039 4405 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ff565080-edb4-4987-8bf2-8fd76a3c3696-config-out" pod "ff565080-edb4-4987-8bf2-8fd76a3c3696" (UID: "ff565080-edb4-4987-8bf2-8fd76a3c3696"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 02:34:58.307388 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:34:58.307366 4405 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ff565080-edb4-4987-8bf2-8fd76a3c3696-secret-grpc-tls" pod "ff565080-edb4-4987-8bf2-8fd76a3c3696" (UID: "ff565080-edb4-4987-8bf2-8fd76a3c3696"). InnerVolumeSpecName "secret-grpc-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 04 02:34:58.307863 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:34:58.307846 4405 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ff565080-edb4-4987-8bf2-8fd76a3c3696-thanos-prometheus-http-client-file" pod "ff565080-edb4-4987-8bf2-8fd76a3c3696" (UID: "ff565080-edb4-4987-8bf2-8fd76a3c3696"). InnerVolumeSpecName "thanos-prometheus-http-client-file". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 04 02:34:58.308132 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:34:58.308114 4405 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ff565080-edb4-4987-8bf2-8fd76a3c3696-secret-prometheus-k8s-kube-rbac-proxy-web" pod "ff565080-edb4-4987-8bf2-8fd76a3c3696" (UID: "ff565080-edb4-4987-8bf2-8fd76a3c3696"). InnerVolumeSpecName "secret-prometheus-k8s-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 04 02:34:58.317690 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:34:58.317645 4405 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ff565080-edb4-4987-8bf2-8fd76a3c3696-web-config" pod "ff565080-edb4-4987-8bf2-8fd76a3c3696" (UID: "ff565080-edb4-4987-8bf2-8fd76a3c3696"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 04 02:34:58.403711 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:34:58.403655 4405 reconciler_common.go:299] "Volume detached for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/ff565080-edb4-4987-8bf2-8fd76a3c3696-secret-grpc-tls\") on node \"ip-10-0-134-125.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 02:34:58.403711 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:34:58.403702 4405 reconciler_common.go:299] "Volume detached for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ff565080-edb4-4987-8bf2-8fd76a3c3696-configmap-metrics-client-ca\") on node \"ip-10-0-134-125.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 02:34:58.403711 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:34:58.403712 4405 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/ff565080-edb4-4987-8bf2-8fd76a3c3696-config-out\") on node \"ip-10-0-134-125.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 02:34:58.403711 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:34:58.403722 4405 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/ff565080-edb4-4987-8bf2-8fd76a3c3696-prometheus-k8s-db\") on node \"ip-10-0-134-125.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 02:34:58.403986 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:34:58.403731 4405 reconciler_common.go:299] "Volume detached for volume \"config\" (UniqueName: \"kubernetes.io/secret/ff565080-edb4-4987-8bf2-8fd76a3c3696-config\") on node \"ip-10-0-134-125.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 02:34:58.403986 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:34:58.403741 4405 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/ff565080-edb4-4987-8bf2-8fd76a3c3696-prometheus-k8s-rulefiles-2\") on node \"ip-10-0-134-125.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 02:34:58.403986 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:34:58.403749 4405 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/ff565080-edb4-4987-8bf2-8fd76a3c3696-prometheus-k8s-rulefiles-0\") on node \"ip-10-0-134-125.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 02:34:58.403986 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:34:58.403758 4405 reconciler_common.go:299] "Volume detached for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ff565080-edb4-4987-8bf2-8fd76a3c3696-configmap-serving-certs-ca-bundle\") on node \"ip-10-0-134-125.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 02:34:58.403986 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:34:58.403768 4405 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/ff565080-edb4-4987-8bf2-8fd76a3c3696-secret-prometheus-k8s-kube-rbac-proxy-web\") on node \"ip-10-0-134-125.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 02:34:58.403986 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:34:58.403776 4405 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nwzmm\" (UniqueName: \"kubernetes.io/projected/ff565080-edb4-4987-8bf2-8fd76a3c3696-kube-api-access-nwzmm\") on node \"ip-10-0-134-125.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 02:34:58.403986 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:34:58.403785 4405 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/ff565080-edb4-4987-8bf2-8fd76a3c3696-web-config\") on node \"ip-10-0-134-125.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 02:34:58.403986 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:34:58.403793 4405 reconciler_common.go:299] "Volume detached for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ff565080-edb4-4987-8bf2-8fd76a3c3696-configmap-kubelet-serving-ca-bundle\") on node \"ip-10-0-134-125.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 02:34:58.403986 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:34:58.403802 4405 reconciler_common.go:299] "Volume detached for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/ff565080-edb4-4987-8bf2-8fd76a3c3696-thanos-prometheus-http-client-file\") on node \"ip-10-0-134-125.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 02:34:58.403986 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:34:58.403810 4405 reconciler_common.go:299] "Volume detached for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/ff565080-edb4-4987-8bf2-8fd76a3c3696-secret-kube-rbac-proxy\") on node \"ip-10-0-134-125.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 02:34:58.403986 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:34:58.403818 4405 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/ff565080-edb4-4987-8bf2-8fd76a3c3696-secret-prometheus-k8s-tls\") on node \"ip-10-0-134-125.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 02:34:58.403986 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:34:58.403827 4405 reconciler_common.go:299] "Volume detached for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/ff565080-edb4-4987-8bf2-8fd76a3c3696-secret-metrics-client-certs\") on node \"ip-10-0-134-125.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 02:34:58.403986 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:34:58.403836 4405 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/ff565080-edb4-4987-8bf2-8fd76a3c3696-secret-prometheus-k8s-thanos-sidecar-tls\") on node \"ip-10-0-134-125.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 02:34:58.403986 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:34:58.403845 4405 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/ff565080-edb4-4987-8bf2-8fd76a3c3696-tls-assets\") on node \"ip-10-0-134-125.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 02:34:58.660389 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:34:58.660352 4405 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Nov 04 02:34:58.664423 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:34:58.664395 4405 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Nov 04 02:34:58.694193 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:34:58.694166 4405 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Nov 04 02:34:58.694402 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:34:58.694390 4405 cpu_manager.go:414] "containerMap: removing container" podUID="5a5c4b25-7a35-4290-ad1e-04b4a3617877" containerName="insights-gathering" Nov 04 02:34:58.694441 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:34:58.694404 4405 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5a5c4b25-7a35-4290-ad1e-04b4a3617877" containerName="insights-gathering" Nov 04 02:34:58.694441 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:34:58.694420 4405 cpu_manager.go:414] "containerMap: removing container" podUID="ff565080-edb4-4987-8bf2-8fd76a3c3696" containerName="prometheus" Nov 04 02:34:58.694441 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:34:58.694425 4405 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ff565080-edb4-4987-8bf2-8fd76a3c3696" containerName="prometheus" Nov 04 02:34:58.694441 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:34:58.694433 4405 cpu_manager.go:414] "containerMap: removing container" podUID="ff565080-edb4-4987-8bf2-8fd76a3c3696" containerName="thanos-sidecar" Nov 04 02:34:58.694441 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:34:58.694438 4405 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ff565080-edb4-4987-8bf2-8fd76a3c3696" containerName="thanos-sidecar" Nov 04 02:34:58.694590 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:34:58.694444 4405 cpu_manager.go:414] "containerMap: removing container" podUID="ff565080-edb4-4987-8bf2-8fd76a3c3696" containerName="kube-rbac-proxy-web" Nov 04 02:34:58.694590 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:34:58.694449 4405 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ff565080-edb4-4987-8bf2-8fd76a3c3696" containerName="kube-rbac-proxy-web" Nov 04 02:34:58.694590 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:34:58.694455 4405 cpu_manager.go:414] "containerMap: removing container" podUID="ff565080-edb4-4987-8bf2-8fd76a3c3696" containerName="kube-rbac-proxy-thanos" Nov 04 02:34:58.694590 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:34:58.694460 4405 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ff565080-edb4-4987-8bf2-8fd76a3c3696" containerName="kube-rbac-proxy-thanos" Nov 04 02:34:58.694590 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:34:58.694471 4405 cpu_manager.go:414] "containerMap: removing container" podUID="ff565080-edb4-4987-8bf2-8fd76a3c3696" containerName="config-reloader" Nov 04 02:34:58.694590 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:34:58.694476 4405 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ff565080-edb4-4987-8bf2-8fd76a3c3696" containerName="config-reloader" Nov 04 02:34:58.694590 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:34:58.694483 4405 cpu_manager.go:414] "containerMap: removing container" podUID="ff565080-edb4-4987-8bf2-8fd76a3c3696" containerName="init-config-reloader" Nov 04 02:34:58.694590 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:34:58.694488 4405 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ff565080-edb4-4987-8bf2-8fd76a3c3696" containerName="init-config-reloader" Nov 04 02:34:58.694590 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:34:58.694494 4405 cpu_manager.go:414] "containerMap: removing container" podUID="ff565080-edb4-4987-8bf2-8fd76a3c3696" containerName="kube-rbac-proxy" Nov 04 02:34:58.694590 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:34:58.694498 4405 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ff565080-edb4-4987-8bf2-8fd76a3c3696" containerName="kube-rbac-proxy" Nov 04 02:34:58.694590 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:34:58.694536 4405 memory_manager.go:365] "RemoveStaleState removing state" podUID="ff565080-edb4-4987-8bf2-8fd76a3c3696" containerName="prometheus" Nov 04 02:34:58.694590 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:34:58.694542 4405 memory_manager.go:365] "RemoveStaleState removing state" podUID="ff565080-edb4-4987-8bf2-8fd76a3c3696" containerName="kube-rbac-proxy-web" Nov 04 02:34:58.694590 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:34:58.694548 4405 memory_manager.go:365] "RemoveStaleState removing state" podUID="ff565080-edb4-4987-8bf2-8fd76a3c3696" containerName="thanos-sidecar" Nov 04 02:34:58.694590 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:34:58.694554 4405 memory_manager.go:365] "RemoveStaleState removing state" podUID="ff565080-edb4-4987-8bf2-8fd76a3c3696" containerName="kube-rbac-proxy-thanos" Nov 04 02:34:58.694590 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:34:58.694560 4405 memory_manager.go:365] "RemoveStaleState removing state" podUID="ff565080-edb4-4987-8bf2-8fd76a3c3696" containerName="config-reloader" Nov 04 02:34:58.694590 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:34:58.694565 4405 memory_manager.go:365] "RemoveStaleState removing state" podUID="ff565080-edb4-4987-8bf2-8fd76a3c3696" containerName="kube-rbac-proxy" Nov 04 02:34:58.694590 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:34:58.694570 4405 memory_manager.go:365] "RemoveStaleState removing state" podUID="5a5c4b25-7a35-4290-ad1e-04b4a3617877" containerName="insights-gathering" Nov 04 02:34:58.701970 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:34:58.701944 4405 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Nov 04 02:34:58.754064 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:34:58.754034 4405 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\"" Nov 04 02:34:58.760927 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:34:58.760796 4405 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\"" Nov 04 02:34:58.760927 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:34:58.760821 4405 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\"" Nov 04 02:34:58.760927 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:34:58.760844 4405 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\"" Nov 04 02:34:58.760927 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:34:58.760870 4405 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\"" Nov 04 02:34:58.760927 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:34:58.760881 4405 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\"" Nov 04 02:34:58.760927 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:34:58.760901 4405 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\"" Nov 04 02:34:58.760927 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:34:58.760933 4405 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\"" Nov 04 02:34:58.761351 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:34:58.760968 4405 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-7krrq\"" Nov 04 02:34:58.761351 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:34:58.761021 4405 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\"" Nov 04 02:34:58.761351 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:34:58.761093 4405 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\"" Nov 04 02:34:58.761351 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:34:58.761298 4405 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-f2s9t2mks6q85\"" Nov 04 02:34:58.761553 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:34:58.761539 4405 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\"" Nov 04 02:34:58.761870 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:34:58.761856 4405 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\"" Nov 04 02:34:58.763478 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:34:58.763462 4405 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\"" Nov 04 02:34:58.763540 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:34:58.763465 4405 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\"" Nov 04 02:34:58.807058 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:34:58.806984 4405 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/a3c43b2d-590a-4490-b05b-7b30b58592bf-config\") pod \"prometheus-k8s-0\" (UID: \"a3c43b2d-590a-4490-b05b-7b30b58592bf\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 04 02:34:58.807058 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:34:58.807016 4405 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/a3c43b2d-590a-4490-b05b-7b30b58592bf-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"a3c43b2d-590a-4490-b05b-7b30b58592bf\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 04 02:34:58.807058 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:34:58.807043 4405 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a3c43b2d-590a-4490-b05b-7b30b58592bf-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"a3c43b2d-590a-4490-b05b-7b30b58592bf\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 04 02:34:58.807268 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:34:58.807111 4405 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/a3c43b2d-590a-4490-b05b-7b30b58592bf-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"a3c43b2d-590a-4490-b05b-7b30b58592bf\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 04 02:34:58.807268 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:34:58.807144 4405 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/a3c43b2d-590a-4490-b05b-7b30b58592bf-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"a3c43b2d-590a-4490-b05b-7b30b58592bf\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 04 02:34:58.807268 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:34:58.807210 4405 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/a3c43b2d-590a-4490-b05b-7b30b58592bf-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"a3c43b2d-590a-4490-b05b-7b30b58592bf\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 04 02:34:58.807268 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:34:58.807226 4405 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/a3c43b2d-590a-4490-b05b-7b30b58592bf-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"a3c43b2d-590a-4490-b05b-7b30b58592bf\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 04 02:34:58.807415 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:34:58.807281 4405 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/a3c43b2d-590a-4490-b05b-7b30b58592bf-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"a3c43b2d-590a-4490-b05b-7b30b58592bf\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 04 02:34:58.807415 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:34:58.807300 4405 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/a3c43b2d-590a-4490-b05b-7b30b58592bf-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"a3c43b2d-590a-4490-b05b-7b30b58592bf\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 04 02:34:58.807415 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:34:58.807316 4405 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/a3c43b2d-590a-4490-b05b-7b30b58592bf-web-config\") pod \"prometheus-k8s-0\" (UID: \"a3c43b2d-590a-4490-b05b-7b30b58592bf\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 04 02:34:58.807415 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:34:58.807330 4405 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a3c43b2d-590a-4490-b05b-7b30b58592bf-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"a3c43b2d-590a-4490-b05b-7b30b58592bf\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 04 02:34:58.807415 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:34:58.807352 4405 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/a3c43b2d-590a-4490-b05b-7b30b58592bf-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"a3c43b2d-590a-4490-b05b-7b30b58592bf\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 04 02:34:58.807608 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:34:58.807461 4405 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/a3c43b2d-590a-4490-b05b-7b30b58592bf-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"a3c43b2d-590a-4490-b05b-7b30b58592bf\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 04 02:34:58.807608 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:34:58.807511 4405 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a3c43b2d-590a-4490-b05b-7b30b58592bf-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"a3c43b2d-590a-4490-b05b-7b30b58592bf\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 04 02:34:58.807608 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:34:58.807580 4405 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/a3c43b2d-590a-4490-b05b-7b30b58592bf-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"a3c43b2d-590a-4490-b05b-7b30b58592bf\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 04 02:34:58.807608 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:34:58.807601 4405 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/a3c43b2d-590a-4490-b05b-7b30b58592bf-config-out\") pod \"prometheus-k8s-0\" (UID: \"a3c43b2d-590a-4490-b05b-7b30b58592bf\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 04 02:34:58.807789 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:34:58.807621 4405 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a3c43b2d-590a-4490-b05b-7b30b58592bf-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"a3c43b2d-590a-4490-b05b-7b30b58592bf\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 04 02:34:58.807789 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:34:58.807657 4405 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rp92k\" (UniqueName: \"kubernetes.io/projected/a3c43b2d-590a-4490-b05b-7b30b58592bf-kube-api-access-rp92k\") pod \"prometheus-k8s-0\" (UID: \"a3c43b2d-590a-4490-b05b-7b30b58592bf\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 04 02:34:58.807789 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:34:58.807764 4405 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/a3c43b2d-590a-4490-b05b-7b30b58592bf-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"a3c43b2d-590a-4490-b05b-7b30b58592bf\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 04 02:34:58.807789 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:34:58.807784 4405 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/a3c43b2d-590a-4490-b05b-7b30b58592bf-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"a3c43b2d-590a-4490-b05b-7b30b58592bf\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 04 02:34:58.811406 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:34:58.811385 4405 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\"" Nov 04 02:34:58.812250 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:34:58.812231 4405 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Nov 04 02:34:58.909117 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:34:58.909079 4405 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/a3c43b2d-590a-4490-b05b-7b30b58592bf-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"a3c43b2d-590a-4490-b05b-7b30b58592bf\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 04 02:34:58.909117 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:34:58.909122 4405 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/a3c43b2d-590a-4490-b05b-7b30b58592bf-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"a3c43b2d-590a-4490-b05b-7b30b58592bf\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 04 02:34:58.909352 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:34:58.909154 4405 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/a3c43b2d-590a-4490-b05b-7b30b58592bf-config\") pod \"prometheus-k8s-0\" (UID: \"a3c43b2d-590a-4490-b05b-7b30b58592bf\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 04 02:34:58.909352 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:34:58.909272 4405 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/a3c43b2d-590a-4490-b05b-7b30b58592bf-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"a3c43b2d-590a-4490-b05b-7b30b58592bf\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 04 02:34:58.909352 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:34:58.909307 4405 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a3c43b2d-590a-4490-b05b-7b30b58592bf-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"a3c43b2d-590a-4490-b05b-7b30b58592bf\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 04 02:34:58.909352 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:34:58.909336 4405 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/a3c43b2d-590a-4490-b05b-7b30b58592bf-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"a3c43b2d-590a-4490-b05b-7b30b58592bf\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 04 02:34:58.909545 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:34:58.909360 4405 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/a3c43b2d-590a-4490-b05b-7b30b58592bf-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"a3c43b2d-590a-4490-b05b-7b30b58592bf\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 04 02:34:58.909545 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:34:58.909407 4405 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/a3c43b2d-590a-4490-b05b-7b30b58592bf-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"a3c43b2d-590a-4490-b05b-7b30b58592bf\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 04 02:34:58.909545 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:34:58.909430 4405 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/a3c43b2d-590a-4490-b05b-7b30b58592bf-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"a3c43b2d-590a-4490-b05b-7b30b58592bf\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 04 02:34:58.909545 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:34:58.909460 4405 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/a3c43b2d-590a-4490-b05b-7b30b58592bf-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"a3c43b2d-590a-4490-b05b-7b30b58592bf\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 04 02:34:58.909545 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:34:58.909490 4405 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/a3c43b2d-590a-4490-b05b-7b30b58592bf-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"a3c43b2d-590a-4490-b05b-7b30b58592bf\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 04 02:34:58.909545 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:34:58.909510 4405 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/a3c43b2d-590a-4490-b05b-7b30b58592bf-web-config\") pod \"prometheus-k8s-0\" (UID: \"a3c43b2d-590a-4490-b05b-7b30b58592bf\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 04 02:34:58.909545 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:34:58.909535 4405 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a3c43b2d-590a-4490-b05b-7b30b58592bf-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"a3c43b2d-590a-4490-b05b-7b30b58592bf\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 04 02:34:58.910089 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:34:58.909565 4405 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/a3c43b2d-590a-4490-b05b-7b30b58592bf-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"a3c43b2d-590a-4490-b05b-7b30b58592bf\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 04 02:34:58.910089 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:34:58.909600 4405 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/a3c43b2d-590a-4490-b05b-7b30b58592bf-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"a3c43b2d-590a-4490-b05b-7b30b58592bf\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 04 02:34:58.910089 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:34:58.909637 4405 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a3c43b2d-590a-4490-b05b-7b30b58592bf-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"a3c43b2d-590a-4490-b05b-7b30b58592bf\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 04 02:34:58.910089 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:34:58.909692 4405 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/a3c43b2d-590a-4490-b05b-7b30b58592bf-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"a3c43b2d-590a-4490-b05b-7b30b58592bf\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 04 02:34:58.910089 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:34:58.909727 4405 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/a3c43b2d-590a-4490-b05b-7b30b58592bf-config-out\") pod \"prometheus-k8s-0\" (UID: \"a3c43b2d-590a-4490-b05b-7b30b58592bf\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 04 02:34:58.910089 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:34:58.909755 4405 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a3c43b2d-590a-4490-b05b-7b30b58592bf-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"a3c43b2d-590a-4490-b05b-7b30b58592bf\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 04 02:34:58.910089 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:34:58.909779 4405 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rp92k\" (UniqueName: \"kubernetes.io/projected/a3c43b2d-590a-4490-b05b-7b30b58592bf-kube-api-access-rp92k\") pod \"prometheus-k8s-0\" (UID: \"a3c43b2d-590a-4490-b05b-7b30b58592bf\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 04 02:34:58.910558 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:34:58.910422 4405 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a3c43b2d-590a-4490-b05b-7b30b58592bf-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"a3c43b2d-590a-4490-b05b-7b30b58592bf\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 04 02:34:58.910867 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:34:58.910839 4405 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/a3c43b2d-590a-4490-b05b-7b30b58592bf-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"a3c43b2d-590a-4490-b05b-7b30b58592bf\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 04 02:34:58.911185 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:34:58.911159 4405 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/a3c43b2d-590a-4490-b05b-7b30b58592bf-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"a3c43b2d-590a-4490-b05b-7b30b58592bf\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 04 02:34:58.911999 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:34:58.911800 4405 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a3c43b2d-590a-4490-b05b-7b30b58592bf-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"a3c43b2d-590a-4490-b05b-7b30b58592bf\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 04 02:34:58.911999 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:34:58.911810 4405 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a3c43b2d-590a-4490-b05b-7b30b58592bf-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"a3c43b2d-590a-4490-b05b-7b30b58592bf\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 04 02:34:58.912567 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:34:58.912536 4405 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/a3c43b2d-590a-4490-b05b-7b30b58592bf-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"a3c43b2d-590a-4490-b05b-7b30b58592bf\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 04 02:34:58.913249 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:34:58.913191 4405 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a3c43b2d-590a-4490-b05b-7b30b58592bf-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"a3c43b2d-590a-4490-b05b-7b30b58592bf\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 04 02:34:58.913408 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:34:58.913391 4405 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/a3c43b2d-590a-4490-b05b-7b30b58592bf-config\") pod \"prometheus-k8s-0\" (UID: \"a3c43b2d-590a-4490-b05b-7b30b58592bf\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 04 02:34:58.913742 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:34:58.913719 4405 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/a3c43b2d-590a-4490-b05b-7b30b58592bf-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"a3c43b2d-590a-4490-b05b-7b30b58592bf\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 04 02:34:58.914898 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:34:58.914417 4405 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/a3c43b2d-590a-4490-b05b-7b30b58592bf-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"a3c43b2d-590a-4490-b05b-7b30b58592bf\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 04 02:34:58.914898 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:34:58.914645 4405 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/a3c43b2d-590a-4490-b05b-7b30b58592bf-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"a3c43b2d-590a-4490-b05b-7b30b58592bf\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 04 02:34:58.914898 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:34:58.914662 4405 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/a3c43b2d-590a-4490-b05b-7b30b58592bf-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"a3c43b2d-590a-4490-b05b-7b30b58592bf\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 04 02:34:58.914898 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:34:58.914711 4405 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/a3c43b2d-590a-4490-b05b-7b30b58592bf-web-config\") pod \"prometheus-k8s-0\" (UID: \"a3c43b2d-590a-4490-b05b-7b30b58592bf\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 04 02:34:58.914898 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:34:58.914881 4405 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/a3c43b2d-590a-4490-b05b-7b30b58592bf-config-out\") pod \"prometheus-k8s-0\" (UID: \"a3c43b2d-590a-4490-b05b-7b30b58592bf\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 04 02:34:58.915231 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:34:58.914881 4405 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/a3c43b2d-590a-4490-b05b-7b30b58592bf-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"a3c43b2d-590a-4490-b05b-7b30b58592bf\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 04 02:34:58.915231 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:34:58.914881 4405 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/a3c43b2d-590a-4490-b05b-7b30b58592bf-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"a3c43b2d-590a-4490-b05b-7b30b58592bf\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 04 02:34:58.915231 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:34:58.915146 4405 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/a3c43b2d-590a-4490-b05b-7b30b58592bf-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"a3c43b2d-590a-4490-b05b-7b30b58592bf\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 04 02:34:58.915377 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:34:58.915316 4405 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/a3c43b2d-590a-4490-b05b-7b30b58592bf-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"a3c43b2d-590a-4490-b05b-7b30b58592bf\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 04 02:34:58.915826 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:34:58.915811 4405 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/a3c43b2d-590a-4490-b05b-7b30b58592bf-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"a3c43b2d-590a-4490-b05b-7b30b58592bf\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 04 02:34:58.971545 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:34:58.971519 4405 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rp92k\" (UniqueName: \"kubernetes.io/projected/a3c43b2d-590a-4490-b05b-7b30b58592bf-kube-api-access-rp92k\") pod \"prometheus-k8s-0\" (UID: \"a3c43b2d-590a-4490-b05b-7b30b58592bf\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 04 02:34:59.017069 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:34:59.017035 4405 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Nov 04 02:34:59.447495 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:34:59.447465 4405 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Nov 04 02:34:59.450380 ip-10-0-134-125 kubenswrapper[4405]: W1104 02:34:59.450344 4405 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda3c43b2d_590a_4490_b05b_7b30b58592bf.slice/crio-017d5dd982b6e3abe064c3b00ad6439fb212912c1b8d696c42e305e24826fbd8 WatchSource:0}: Error finding container 017d5dd982b6e3abe064c3b00ad6439fb212912c1b8d696c42e305e24826fbd8: Status 404 returned error can't find the container with id 017d5dd982b6e3abe064c3b00ad6439fb212912c1b8d696c42e305e24826fbd8 Nov 04 02:34:59.698679 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:34:59.698588 4405 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ff565080-edb4-4987-8bf2-8fd76a3c3696" path="/var/lib/kubelet/pods/ff565080-edb4-4987-8bf2-8fd76a3c3696/volumes" Nov 04 02:35:00.248466 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:35:00.248433 4405 generic.go:356] "Generic (PLEG): container finished" podID="a3c43b2d-590a-4490-b05b-7b30b58592bf" containerID="5900c35c8e50a92a50a83d936f9ac6116aed03386ffcab12c35a8aacf99b138d" exitCode=0 Nov 04 02:35:00.248988 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:35:00.248484 4405 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"a3c43b2d-590a-4490-b05b-7b30b58592bf","Type":"ContainerDied","Data":"5900c35c8e50a92a50a83d936f9ac6116aed03386ffcab12c35a8aacf99b138d"} Nov 04 02:35:00.248988 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:35:00.248551 4405 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"a3c43b2d-590a-4490-b05b-7b30b58592bf","Type":"ContainerStarted","Data":"017d5dd982b6e3abe064c3b00ad6439fb212912c1b8d696c42e305e24826fbd8"} Nov 04 02:35:01.255032 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:35:01.254994 4405 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"a3c43b2d-590a-4490-b05b-7b30b58592bf","Type":"ContainerStarted","Data":"37c72b1bbf28a5e4bf23951996dd21cd38a0c4cdddc9fc8a2a324063e8f1808f"} Nov 04 02:35:01.255426 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:35:01.255064 4405 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"a3c43b2d-590a-4490-b05b-7b30b58592bf","Type":"ContainerStarted","Data":"b7a9445c4b8c89dfe9c42080e4663707d6d037a1b9909752960075c27fc97d49"} Nov 04 02:35:01.255426 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:35:01.255096 4405 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"a3c43b2d-590a-4490-b05b-7b30b58592bf","Type":"ContainerStarted","Data":"d5e620a971db1e79f9edc7dc3cdc452453518f0c39baee1c79459475d8a3c56e"} Nov 04 02:35:01.255426 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:35:01.255120 4405 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"a3c43b2d-590a-4490-b05b-7b30b58592bf","Type":"ContainerStarted","Data":"ebbe498df656f1981fffe16dcdfd17d09c0acef7ff86fd0990bf2326fe993fb3"} Nov 04 02:35:01.255426 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:35:01.255143 4405 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"a3c43b2d-590a-4490-b05b-7b30b58592bf","Type":"ContainerStarted","Data":"9eca5b4519c161ea08db9e5be0b750075703f6f06c682f2604ca8d36df5ce143"} Nov 04 02:35:01.255426 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:35:01.255166 4405 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"a3c43b2d-590a-4490-b05b-7b30b58592bf","Type":"ContainerStarted","Data":"1ffc72080149b501def66bcfda3b87bd5bcc6258b3fbfa2f75bc65327317d5d7"} Nov 04 02:35:01.436558 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:35:01.436502 4405 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=3.420574956 podStartE2EDuration="3.436482689s" totalImagesPullingTime="0s" totalInitContainerRuntime="15.907733ms" isStatefulPod=false podCreationTimestamp="2026-11-04 02:34:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 02:35:01.434917564 +0000 UTC m=+147.865617722" watchObservedRunningTime="2026-11-04 02:35:01.436482689 +0000 UTC m=+147.867182808" Nov 04 02:35:04.017304 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:35:04.017267 4405 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0" Nov 04 02:35:06.881237 ip-10-0-134-125 kubenswrapper[4405]: E1104 02:35:06.881189 4405 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5f2vb" containerName="extractor" Nov 04 02:35:36.882660 ip-10-0-134-125 kubenswrapper[4405]: E1104 02:35:36.882617 4405 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5f2vb" containerName="extractor" Nov 04 02:35:59.017629 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:35:59.017578 4405 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0" Nov 04 02:35:59.033769 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:35:59.033744 4405 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0" Nov 04 02:35:59.488959 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:35:59.488932 4405 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0" Nov 04 02:36:06.882088 ip-10-0-134-125 kubenswrapper[4405]: E1104 02:36:06.882052 4405 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5f2vb" containerName="extractor" Nov 04 02:36:36.882445 ip-10-0-134-125 kubenswrapper[4405]: E1104 02:36:36.882407 4405 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5f2vb" containerName="extractor" Nov 04 02:37:06.882098 ip-10-0-134-125 kubenswrapper[4405]: E1104 02:37:06.882062 4405 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5f2vb" containerName="extractor" Nov 04 02:37:33.704653 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:37:33.704624 4405 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-vwcx8_ffbb2d83-5a93-4a5b-bce3-e40afb8f58ea/ovn-acl-logging/0.log" Nov 04 02:37:33.706893 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:37:33.706873 4405 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-vwcx8_ffbb2d83-5a93-4a5b-bce3-e40afb8f58ea/ovn-acl-logging/0.log" Nov 04 02:37:36.882748 ip-10-0-134-125 kubenswrapper[4405]: E1104 02:37:36.882710 4405 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5f2vb" containerName="extractor" Nov 04 02:38:06.883237 ip-10-0-134-125 kubenswrapper[4405]: E1104 02:38:06.883148 4405 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5f2vb" containerName="extractor" Nov 04 02:38:12.386456 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:38:12.386418 4405 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-7bfc68bc9f-ztzt4"] Nov 04 02:38:12.389578 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:38:12.389562 4405 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7bfc68bc9f-ztzt4" Nov 04 02:38:12.453947 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:38:12.453905 4405 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/9d709338-5abc-45ea-8f0d-c86f004ca526-metrics-cert\") pod \"opendatahub-operator-controller-manager-7bfc68bc9f-ztzt4\" (UID: \"9d709338-5abc-45ea-8f0d-c86f004ca526\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7bfc68bc9f-ztzt4" Nov 04 02:38:12.454133 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:38:12.453970 4405 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zk2f4\" (UniqueName: \"kubernetes.io/projected/9d709338-5abc-45ea-8f0d-c86f004ca526-kube-api-access-zk2f4\") pod \"opendatahub-operator-controller-manager-7bfc68bc9f-ztzt4\" (UID: \"9d709338-5abc-45ea-8f0d-c86f004ca526\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7bfc68bc9f-ztzt4" Nov 04 02:38:12.454133 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:38:12.454067 4405 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/9d709338-5abc-45ea-8f0d-c86f004ca526-tmp\") pod \"opendatahub-operator-controller-manager-7bfc68bc9f-ztzt4\" (UID: \"9d709338-5abc-45ea-8f0d-c86f004ca526\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7bfc68bc9f-ztzt4" Nov 04 02:38:12.454223 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:38:12.454124 4405 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/9d709338-5abc-45ea-8f0d-c86f004ca526-apiservice-cert\") pod \"opendatahub-operator-controller-manager-7bfc68bc9f-ztzt4\" (UID: \"9d709338-5abc-45ea-8f0d-c86f004ca526\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7bfc68bc9f-ztzt4" Nov 04 02:38:12.454223 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:38:12.454192 4405 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/9d709338-5abc-45ea-8f0d-c86f004ca526-manifests\") pod \"opendatahub-operator-controller-manager-7bfc68bc9f-ztzt4\" (UID: \"9d709338-5abc-45ea-8f0d-c86f004ca526\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7bfc68bc9f-ztzt4" Nov 04 02:38:12.454297 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:38:12.454226 4405 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/9d709338-5abc-45ea-8f0d-c86f004ca526-webhook-cert\") pod \"opendatahub-operator-controller-manager-7bfc68bc9f-ztzt4\" (UID: \"9d709338-5abc-45ea-8f0d-c86f004ca526\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7bfc68bc9f-ztzt4" Nov 04 02:38:12.470109 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:38:12.470080 4405 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub-operator-system\"/\"kube-root-ca.crt\"" Nov 04 02:38:12.491890 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:38:12.491721 4405 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-webhook-cert\"" Nov 04 02:38:12.491890 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:38:12.491732 4405 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub-operator-system\"/\"openshift-service-ca.crt\"" Nov 04 02:38:12.491890 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:38:12.491819 4405 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-manager-service-cert\"" Nov 04 02:38:12.492198 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:38:12.491976 4405 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-manager-dockercfg-8qrsw\"" Nov 04 02:38:12.492198 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:38:12.491983 4405 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-metrics-tls\"" Nov 04 02:38:12.493998 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:38:12.493978 4405 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-7bfc68bc9f-ztzt4"] Nov 04 02:38:12.555563 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:38:12.555528 4405 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zk2f4\" (UniqueName: \"kubernetes.io/projected/9d709338-5abc-45ea-8f0d-c86f004ca526-kube-api-access-zk2f4\") pod \"opendatahub-operator-controller-manager-7bfc68bc9f-ztzt4\" (UID: \"9d709338-5abc-45ea-8f0d-c86f004ca526\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7bfc68bc9f-ztzt4" Nov 04 02:38:12.555563 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:38:12.555563 4405 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/9d709338-5abc-45ea-8f0d-c86f004ca526-tmp\") pod \"opendatahub-operator-controller-manager-7bfc68bc9f-ztzt4\" (UID: \"9d709338-5abc-45ea-8f0d-c86f004ca526\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7bfc68bc9f-ztzt4" Nov 04 02:38:12.555881 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:38:12.555584 4405 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/9d709338-5abc-45ea-8f0d-c86f004ca526-apiservice-cert\") pod \"opendatahub-operator-controller-manager-7bfc68bc9f-ztzt4\" (UID: \"9d709338-5abc-45ea-8f0d-c86f004ca526\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7bfc68bc9f-ztzt4" Nov 04 02:38:12.555881 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:38:12.555614 4405 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/9d709338-5abc-45ea-8f0d-c86f004ca526-manifests\") pod \"opendatahub-operator-controller-manager-7bfc68bc9f-ztzt4\" (UID: \"9d709338-5abc-45ea-8f0d-c86f004ca526\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7bfc68bc9f-ztzt4" Nov 04 02:38:12.555881 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:38:12.555639 4405 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/9d709338-5abc-45ea-8f0d-c86f004ca526-webhook-cert\") pod \"opendatahub-operator-controller-manager-7bfc68bc9f-ztzt4\" (UID: \"9d709338-5abc-45ea-8f0d-c86f004ca526\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7bfc68bc9f-ztzt4" Nov 04 02:38:12.555881 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:38:12.555662 4405 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/9d709338-5abc-45ea-8f0d-c86f004ca526-metrics-cert\") pod \"opendatahub-operator-controller-manager-7bfc68bc9f-ztzt4\" (UID: \"9d709338-5abc-45ea-8f0d-c86f004ca526\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7bfc68bc9f-ztzt4" Nov 04 02:38:12.556113 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:38:12.556091 4405 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/9d709338-5abc-45ea-8f0d-c86f004ca526-manifests\") pod \"opendatahub-operator-controller-manager-7bfc68bc9f-ztzt4\" (UID: \"9d709338-5abc-45ea-8f0d-c86f004ca526\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7bfc68bc9f-ztzt4" Nov 04 02:38:12.558026 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:38:12.557998 4405 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/9d709338-5abc-45ea-8f0d-c86f004ca526-tmp\") pod \"opendatahub-operator-controller-manager-7bfc68bc9f-ztzt4\" (UID: \"9d709338-5abc-45ea-8f0d-c86f004ca526\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7bfc68bc9f-ztzt4" Nov 04 02:38:12.558368 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:38:12.558348 4405 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/9d709338-5abc-45ea-8f0d-c86f004ca526-webhook-cert\") pod \"opendatahub-operator-controller-manager-7bfc68bc9f-ztzt4\" (UID: \"9d709338-5abc-45ea-8f0d-c86f004ca526\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7bfc68bc9f-ztzt4" Nov 04 02:38:12.558368 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:38:12.558356 4405 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/9d709338-5abc-45ea-8f0d-c86f004ca526-apiservice-cert\") pod \"opendatahub-operator-controller-manager-7bfc68bc9f-ztzt4\" (UID: \"9d709338-5abc-45ea-8f0d-c86f004ca526\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7bfc68bc9f-ztzt4" Nov 04 02:38:12.558494 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:38:12.558366 4405 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/9d709338-5abc-45ea-8f0d-c86f004ca526-metrics-cert\") pod \"opendatahub-operator-controller-manager-7bfc68bc9f-ztzt4\" (UID: \"9d709338-5abc-45ea-8f0d-c86f004ca526\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7bfc68bc9f-ztzt4" Nov 04 02:38:12.618762 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:38:12.618730 4405 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zk2f4\" (UniqueName: \"kubernetes.io/projected/9d709338-5abc-45ea-8f0d-c86f004ca526-kube-api-access-zk2f4\") pod \"opendatahub-operator-controller-manager-7bfc68bc9f-ztzt4\" (UID: \"9d709338-5abc-45ea-8f0d-c86f004ca526\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7bfc68bc9f-ztzt4" Nov 04 02:38:12.703567 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:38:12.703490 4405 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7bfc68bc9f-ztzt4" Nov 04 02:38:13.189443 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:38:13.189408 4405 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-7bfc68bc9f-ztzt4"] Nov 04 02:38:13.192773 ip-10-0-134-125 kubenswrapper[4405]: W1104 02:38:13.192747 4405 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9d709338_5abc_45ea_8f0d_c86f004ca526.slice/crio-5c9f36a475a5a6b9191ec7ccb1f234a9a33be8ab1eab3b2de7da891384dd4546 WatchSource:0}: Error finding container 5c9f36a475a5a6b9191ec7ccb1f234a9a33be8ab1eab3b2de7da891384dd4546: Status 404 returned error can't find the container with id 5c9f36a475a5a6b9191ec7ccb1f234a9a33be8ab1eab3b2de7da891384dd4546 Nov 04 02:38:13.194695 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:38:13.194653 4405 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 04 02:38:13.909618 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:38:13.909583 4405 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7bfc68bc9f-ztzt4" event={"ID":"9d709338-5abc-45ea-8f0d-c86f004ca526","Type":"ContainerStarted","Data":"5c9f36a475a5a6b9191ec7ccb1f234a9a33be8ab1eab3b2de7da891384dd4546"} Nov 04 02:38:16.923179 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:38:16.923091 4405 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7bfc68bc9f-ztzt4" event={"ID":"9d709338-5abc-45ea-8f0d-c86f004ca526","Type":"ContainerStarted","Data":"11c2e666b97e4675d463959344f890f9adbb2578e7a3a57a7551d2002eee2c55"} Nov 04 02:38:17.928741 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:38:17.928710 4405 generic.go:356] "Generic (PLEG): container finished" podID="9d709338-5abc-45ea-8f0d-c86f004ca526" containerID="11c2e666b97e4675d463959344f890f9adbb2578e7a3a57a7551d2002eee2c55" exitCode=0 Nov 04 02:38:17.929144 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:38:17.928748 4405 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7bfc68bc9f-ztzt4" event={"ID":"9d709338-5abc-45ea-8f0d-c86f004ca526","Type":"ContainerDied","Data":"11c2e666b97e4675d463959344f890f9adbb2578e7a3a57a7551d2002eee2c55"} Nov 04 02:38:18.934375 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:38:18.934342 4405 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7bfc68bc9f-ztzt4" event={"ID":"9d709338-5abc-45ea-8f0d-c86f004ca526","Type":"ContainerStarted","Data":"4a2879f02f03c3ce935e2cc2f2981e756d3b06e1a4dec5ad7595bbe265f02fcd"} Nov 04 02:38:18.934769 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:38:18.934466 4405 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7bfc68bc9f-ztzt4" Nov 04 02:38:19.170156 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:38:19.170102 4405 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7bfc68bc9f-ztzt4" podStartSLOduration=2.352389073 podStartE2EDuration="7.170086458s" totalImagesPullingTime="4.343233451s" totalInitContainerRuntime="474.463934ms" isStatefulPod=false podCreationTimestamp="2026-11-04 02:38:12 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 02:38:19.121086892 +0000 UTC m=+345.551787049" watchObservedRunningTime="2026-11-04 02:38:19.170086458 +0000 UTC m=+345.600786573" Nov 04 02:38:25.951314 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:38:25.951278 4405 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-webhook-5dbbbf75cc-hhnmx"] Nov 04 02:38:25.959158 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:38:25.959136 4405 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-hhnmx" Nov 04 02:38:26.012029 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:38:26.011989 4405 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-webhook-5dbbbf75cc-hhnmx"] Nov 04 02:38:26.060442 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:38:26.060403 4405 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"kube-root-ca.crt\"" Nov 04 02:38:26.060442 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:38:26.060406 4405 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"openshift-service-ca.crt\"" Nov 04 02:38:26.060442 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:38:26.060419 4405 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-webhook-dockercfg-2jsrj\"" Nov 04 02:38:26.074791 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:38:26.074760 4405 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/73142270-a8e7-4180-bfa5-6f720372736f-bound-sa-token\") pod \"cert-manager-webhook-5dbbbf75cc-hhnmx\" (UID: \"73142270-a8e7-4180-bfa5-6f720372736f\") " pod="cert-manager/cert-manager-webhook-5dbbbf75cc-hhnmx" Nov 04 02:38:26.074877 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:38:26.074817 4405 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nhcwk\" (UniqueName: \"kubernetes.io/projected/73142270-a8e7-4180-bfa5-6f720372736f-kube-api-access-nhcwk\") pod \"cert-manager-webhook-5dbbbf75cc-hhnmx\" (UID: \"73142270-a8e7-4180-bfa5-6f720372736f\") " pod="cert-manager/cert-manager-webhook-5dbbbf75cc-hhnmx" Nov 04 02:38:26.176165 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:38:26.176124 4405 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nhcwk\" (UniqueName: \"kubernetes.io/projected/73142270-a8e7-4180-bfa5-6f720372736f-kube-api-access-nhcwk\") pod \"cert-manager-webhook-5dbbbf75cc-hhnmx\" (UID: \"73142270-a8e7-4180-bfa5-6f720372736f\") " pod="cert-manager/cert-manager-webhook-5dbbbf75cc-hhnmx" Nov 04 02:38:26.176362 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:38:26.176185 4405 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/73142270-a8e7-4180-bfa5-6f720372736f-bound-sa-token\") pod \"cert-manager-webhook-5dbbbf75cc-hhnmx\" (UID: \"73142270-a8e7-4180-bfa5-6f720372736f\") " pod="cert-manager/cert-manager-webhook-5dbbbf75cc-hhnmx" Nov 04 02:38:26.239916 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:38:26.239847 4405 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/73142270-a8e7-4180-bfa5-6f720372736f-bound-sa-token\") pod \"cert-manager-webhook-5dbbbf75cc-hhnmx\" (UID: \"73142270-a8e7-4180-bfa5-6f720372736f\") " pod="cert-manager/cert-manager-webhook-5dbbbf75cc-hhnmx" Nov 04 02:38:26.240146 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:38:26.240121 4405 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nhcwk\" (UniqueName: \"kubernetes.io/projected/73142270-a8e7-4180-bfa5-6f720372736f-kube-api-access-nhcwk\") pod \"cert-manager-webhook-5dbbbf75cc-hhnmx\" (UID: \"73142270-a8e7-4180-bfa5-6f720372736f\") " pod="cert-manager/cert-manager-webhook-5dbbbf75cc-hhnmx" Nov 04 02:38:26.282967 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:38:26.282926 4405 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-hhnmx" Nov 04 02:38:26.656936 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:38:26.656907 4405 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-webhook-5dbbbf75cc-hhnmx"] Nov 04 02:38:26.660749 ip-10-0-134-125 kubenswrapper[4405]: W1104 02:38:26.660715 4405 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod73142270_a8e7_4180_bfa5_6f720372736f.slice/crio-2ea4a4f17e9670aa57effbcc8d63f4a2fb319edb78e490a4d1b2b6d72c4d516f WatchSource:0}: Error finding container 2ea4a4f17e9670aa57effbcc8d63f4a2fb319edb78e490a4d1b2b6d72c4d516f: Status 404 returned error can't find the container with id 2ea4a4f17e9670aa57effbcc8d63f4a2fb319edb78e490a4d1b2b6d72c4d516f Nov 04 02:38:26.964575 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:38:26.964482 4405 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-hhnmx" event={"ID":"73142270-a8e7-4180-bfa5-6f720372736f","Type":"ContainerStarted","Data":"2ea4a4f17e9670aa57effbcc8d63f4a2fb319edb78e490a4d1b2b6d72c4d516f"} Nov 04 02:38:29.941531 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:38:29.941499 4405 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7bfc68bc9f-ztzt4" Nov 04 02:38:31.984075 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:38:31.984032 4405 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-hhnmx" event={"ID":"73142270-a8e7-4180-bfa5-6f720372736f","Type":"ContainerStarted","Data":"d3b74f7b44df54fd34a33886e4c66c883de7ef9108e5b43af1448fe82056e848"} Nov 04 02:38:31.984479 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:38:31.984205 4405 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-hhnmx" Nov 04 02:38:32.104485 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:38:32.104426 4405 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-hhnmx" podStartSLOduration=2.525842815 podStartE2EDuration="7.104410951s" totalImagesPullingTime="4.578568136s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 02:38:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 02:38:32.103564193 +0000 UTC m=+358.534264325" watchObservedRunningTime="2026-11-04 02:38:32.104410951 +0000 UTC m=+358.535111065" Nov 04 02:38:32.836341 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:38:32.836309 4405 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-jobset-operator/jobset-operator-674cd59dc5-fbnfc"] Nov 04 02:38:32.839370 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:38:32.839354 4405 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-jobset-operator/jobset-operator-674cd59dc5-fbnfc" Nov 04 02:38:32.933893 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:38:32.933860 4405 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/c10c7eb2-6472-4daf-923c-8246da9cfc93-tmp\") pod \"jobset-operator-674cd59dc5-fbnfc\" (UID: \"c10c7eb2-6472-4daf-923c-8246da9cfc93\") " pod="openshift-jobset-operator/jobset-operator-674cd59dc5-fbnfc" Nov 04 02:38:32.934062 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:38:32.933912 4405 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vhmlr\" (UniqueName: \"kubernetes.io/projected/c10c7eb2-6472-4daf-923c-8246da9cfc93-kube-api-access-vhmlr\") pod \"jobset-operator-674cd59dc5-fbnfc\" (UID: \"c10c7eb2-6472-4daf-923c-8246da9cfc93\") " pod="openshift-jobset-operator/jobset-operator-674cd59dc5-fbnfc" Nov 04 02:38:32.948304 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:38:32.948269 4405 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-jobset-operator\"/\"jobset-operator-dockercfg-gkl45\"" Nov 04 02:38:32.948304 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:38:32.948287 4405 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-jobset-operator\"/\"openshift-service-ca.crt\"" Nov 04 02:38:32.948490 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:38:32.948323 4405 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-jobset-operator\"/\"kube-root-ca.crt\"" Nov 04 02:38:32.949339 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:38:32.949322 4405 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-jobset-operator/jobset-operator-674cd59dc5-fbnfc"] Nov 04 02:38:33.034375 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:38:33.034341 4405 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/c10c7eb2-6472-4daf-923c-8246da9cfc93-tmp\") pod \"jobset-operator-674cd59dc5-fbnfc\" (UID: \"c10c7eb2-6472-4daf-923c-8246da9cfc93\") " pod="openshift-jobset-operator/jobset-operator-674cd59dc5-fbnfc" Nov 04 02:38:33.034375 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:38:33.034382 4405 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vhmlr\" (UniqueName: \"kubernetes.io/projected/c10c7eb2-6472-4daf-923c-8246da9cfc93-kube-api-access-vhmlr\") pod \"jobset-operator-674cd59dc5-fbnfc\" (UID: \"c10c7eb2-6472-4daf-923c-8246da9cfc93\") " pod="openshift-jobset-operator/jobset-operator-674cd59dc5-fbnfc" Nov 04 02:38:33.034842 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:38:33.034743 4405 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/c10c7eb2-6472-4daf-923c-8246da9cfc93-tmp\") pod \"jobset-operator-674cd59dc5-fbnfc\" (UID: \"c10c7eb2-6472-4daf-923c-8246da9cfc93\") " pod="openshift-jobset-operator/jobset-operator-674cd59dc5-fbnfc" Nov 04 02:38:33.098389 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:38:33.098322 4405 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vhmlr\" (UniqueName: \"kubernetes.io/projected/c10c7eb2-6472-4daf-923c-8246da9cfc93-kube-api-access-vhmlr\") pod \"jobset-operator-674cd59dc5-fbnfc\" (UID: \"c10c7eb2-6472-4daf-923c-8246da9cfc93\") " pod="openshift-jobset-operator/jobset-operator-674cd59dc5-fbnfc" Nov 04 02:38:33.150299 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:38:33.150270 4405 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-jobset-operator/jobset-operator-674cd59dc5-fbnfc" Nov 04 02:38:33.535229 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:38:33.535201 4405 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-jobset-operator/jobset-operator-674cd59dc5-fbnfc"] Nov 04 02:38:33.537621 ip-10-0-134-125 kubenswrapper[4405]: W1104 02:38:33.537560 4405 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podc10c7eb2_6472_4daf_923c_8246da9cfc93.slice/crio-c9c4422484c8b3922a20eae09ad47ab6ebf466a7cfd4d6cd571f9d937e8e7253 WatchSource:0}: Error finding container c9c4422484c8b3922a20eae09ad47ab6ebf466a7cfd4d6cd571f9d937e8e7253: Status 404 returned error can't find the container with id c9c4422484c8b3922a20eae09ad47ab6ebf466a7cfd4d6cd571f9d937e8e7253 Nov 04 02:38:33.992208 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:38:33.992173 4405 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-jobset-operator/jobset-operator-674cd59dc5-fbnfc" event={"ID":"c10c7eb2-6472-4daf-923c-8246da9cfc93","Type":"ContainerStarted","Data":"c9c4422484c8b3922a20eae09ad47ab6ebf466a7cfd4d6cd571f9d937e8e7253"} Nov 04 02:38:36.885952 ip-10-0-134-125 kubenswrapper[4405]: E1104 02:38:36.885914 4405 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5f2vb" containerName="extractor" Nov 04 02:38:37.990122 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:38:37.990095 4405 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-hhnmx" Nov 04 02:38:38.007042 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:38:38.007012 4405 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-jobset-operator/jobset-operator-674cd59dc5-fbnfc" event={"ID":"c10c7eb2-6472-4daf-923c-8246da9cfc93","Type":"ContainerStarted","Data":"f021a4da6b5c4b6c8ec8f4e7471d07e8154abce2d542ff56f340daa2f727a8f1"} Nov 04 02:38:38.256982 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:38:38.256887 4405 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-jobset-operator/jobset-operator-674cd59dc5-fbnfc" podStartSLOduration=2.474487272 podStartE2EDuration="6.256874024s" totalImagesPullingTime="3.782386752s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 02:38:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 02:38:38.256511191 +0000 UTC m=+364.687211310" watchObservedRunningTime="2026-11-04 02:38:38.256874024 +0000 UTC m=+364.687574143" Nov 04 02:38:41.998695 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:38:41.998649 4405 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-operator-7b7bb4bc77-rm4jt"] Nov 04 02:38:42.001984 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:38:42.001952 4405 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-7b7bb4bc77-rm4jt" Nov 04 02:38:42.002168 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:38:42.002139 4405 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cdnkp\" (UniqueName: \"kubernetes.io/projected/f33c29b3-72db-4504-b751-2a2666001277-kube-api-access-cdnkp\") pod \"authorino-operator-7b7bb4bc77-rm4jt\" (UID: \"f33c29b3-72db-4504-b751-2a2666001277\") " pod="kuadrant-system/authorino-operator-7b7bb4bc77-rm4jt" Nov 04 02:38:42.064406 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:38:42.064373 4405 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-7b7bb4bc77-rm4jt"] Nov 04 02:38:42.103056 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:38:42.103022 4405 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cdnkp\" (UniqueName: \"kubernetes.io/projected/f33c29b3-72db-4504-b751-2a2666001277-kube-api-access-cdnkp\") pod \"authorino-operator-7b7bb4bc77-rm4jt\" (UID: \"f33c29b3-72db-4504-b751-2a2666001277\") " pod="kuadrant-system/authorino-operator-7b7bb4bc77-rm4jt" Nov 04 02:38:42.112580 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:38:42.112559 4405 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\"" Nov 04 02:38:42.112765 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:38:42.112741 4405 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\"" Nov 04 02:38:42.148468 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:38:42.148445 4405 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-operator-dockercfg-gnkz7\"" Nov 04 02:38:42.163790 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:38:42.163760 4405 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cdnkp\" (UniqueName: \"kubernetes.io/projected/f33c29b3-72db-4504-b751-2a2666001277-kube-api-access-cdnkp\") pod \"authorino-operator-7b7bb4bc77-rm4jt\" (UID: \"f33c29b3-72db-4504-b751-2a2666001277\") " pod="kuadrant-system/authorino-operator-7b7bb4bc77-rm4jt" Nov 04 02:38:42.313621 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:38:42.313521 4405 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-7b7bb4bc77-rm4jt" Nov 04 02:38:42.712506 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:38:42.712477 4405 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-7b7bb4bc77-rm4jt"] Nov 04 02:38:42.715556 ip-10-0-134-125 kubenswrapper[4405]: W1104 02:38:42.715525 4405 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf33c29b3_72db_4504_b751_2a2666001277.slice/crio-03b0072aedea5b66d5797f6af8e640b58ec357f55433d50d39e889b6f21e80fa WatchSource:0}: Error finding container 03b0072aedea5b66d5797f6af8e640b58ec357f55433d50d39e889b6f21e80fa: Status 404 returned error can't find the container with id 03b0072aedea5b66d5797f6af8e640b58ec357f55433d50d39e889b6f21e80fa Nov 04 02:38:43.027284 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:38:43.027207 4405 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-7b7bb4bc77-rm4jt" event={"ID":"f33c29b3-72db-4504-b751-2a2666001277","Type":"ContainerStarted","Data":"03b0072aedea5b66d5797f6af8e640b58ec357f55433d50d39e889b6f21e80fa"} Nov 04 02:38:43.113360 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:38:43.113327 4405 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/kube-auth-proxy-54497bf445-qvs4m"] Nov 04 02:38:43.117994 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:38:43.117974 4405 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-54497bf445-qvs4m" Nov 04 02:38:43.172463 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:38:43.172425 4405 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\"" Nov 04 02:38:43.172701 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:38:43.172539 4405 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\"" Nov 04 02:38:43.172701 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:38:43.172637 4405 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\"" Nov 04 02:38:43.172701 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:38:43.172661 4405 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\"" Nov 04 02:38:43.172849 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:38:43.172707 4405 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-dockercfg-9vcnt\"" Nov 04 02:38:43.211616 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:38:43.211583 4405 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-24hc4\" (UniqueName: \"kubernetes.io/projected/8f77d20d-3db8-47dc-a6ee-dec3f1b68432-kube-api-access-24hc4\") pod \"kube-auth-proxy-54497bf445-qvs4m\" (UID: \"8f77d20d-3db8-47dc-a6ee-dec3f1b68432\") " pod="openshift-ingress/kube-auth-proxy-54497bf445-qvs4m" Nov 04 02:38:43.211789 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:38:43.211621 4405 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8f77d20d-3db8-47dc-a6ee-dec3f1b68432-tls-certs\") pod \"kube-auth-proxy-54497bf445-qvs4m\" (UID: \"8f77d20d-3db8-47dc-a6ee-dec3f1b68432\") " pod="openshift-ingress/kube-auth-proxy-54497bf445-qvs4m" Nov 04 02:38:43.211789 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:38:43.211717 4405 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/8f77d20d-3db8-47dc-a6ee-dec3f1b68432-tmp\") pod \"kube-auth-proxy-54497bf445-qvs4m\" (UID: \"8f77d20d-3db8-47dc-a6ee-dec3f1b68432\") " pod="openshift-ingress/kube-auth-proxy-54497bf445-qvs4m" Nov 04 02:38:43.223425 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:38:43.223399 4405 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-54497bf445-qvs4m"] Nov 04 02:38:43.312146 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:38:43.312060 4405 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8f77d20d-3db8-47dc-a6ee-dec3f1b68432-tls-certs\") pod \"kube-auth-proxy-54497bf445-qvs4m\" (UID: \"8f77d20d-3db8-47dc-a6ee-dec3f1b68432\") " pod="openshift-ingress/kube-auth-proxy-54497bf445-qvs4m" Nov 04 02:38:43.312146 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:38:43.312099 4405 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/8f77d20d-3db8-47dc-a6ee-dec3f1b68432-tmp\") pod \"kube-auth-proxy-54497bf445-qvs4m\" (UID: \"8f77d20d-3db8-47dc-a6ee-dec3f1b68432\") " pod="openshift-ingress/kube-auth-proxy-54497bf445-qvs4m" Nov 04 02:38:43.312370 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:38:43.312148 4405 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-24hc4\" (UniqueName: \"kubernetes.io/projected/8f77d20d-3db8-47dc-a6ee-dec3f1b68432-kube-api-access-24hc4\") pod \"kube-auth-proxy-54497bf445-qvs4m\" (UID: \"8f77d20d-3db8-47dc-a6ee-dec3f1b68432\") " pod="openshift-ingress/kube-auth-proxy-54497bf445-qvs4m" Nov 04 02:38:43.314894 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:38:43.314871 4405 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/8f77d20d-3db8-47dc-a6ee-dec3f1b68432-tmp\") pod \"kube-auth-proxy-54497bf445-qvs4m\" (UID: \"8f77d20d-3db8-47dc-a6ee-dec3f1b68432\") " pod="openshift-ingress/kube-auth-proxy-54497bf445-qvs4m" Nov 04 02:38:43.315197 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:38:43.315175 4405 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8f77d20d-3db8-47dc-a6ee-dec3f1b68432-tls-certs\") pod \"kube-auth-proxy-54497bf445-qvs4m\" (UID: \"8f77d20d-3db8-47dc-a6ee-dec3f1b68432\") " pod="openshift-ingress/kube-auth-proxy-54497bf445-qvs4m" Nov 04 02:38:43.375600 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:38:43.375573 4405 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-24hc4\" (UniqueName: \"kubernetes.io/projected/8f77d20d-3db8-47dc-a6ee-dec3f1b68432-kube-api-access-24hc4\") pod \"kube-auth-proxy-54497bf445-qvs4m\" (UID: \"8f77d20d-3db8-47dc-a6ee-dec3f1b68432\") " pod="openshift-ingress/kube-auth-proxy-54497bf445-qvs4m" Nov 04 02:38:43.430600 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:38:43.430567 4405 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-54497bf445-qvs4m" Nov 04 02:38:43.802229 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:38:43.802078 4405 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-54497bf445-qvs4m"] Nov 04 02:38:43.804597 ip-10-0-134-125 kubenswrapper[4405]: W1104 02:38:43.804567 4405 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8f77d20d_3db8_47dc_a6ee_dec3f1b68432.slice/crio-6f32935c2073c7a087f79f0f53152918be1672a4f1d9f3d1fcd03f0f443553c9 WatchSource:0}: Error finding container 6f32935c2073c7a087f79f0f53152918be1672a4f1d9f3d1fcd03f0f443553c9: Status 404 returned error can't find the container with id 6f32935c2073c7a087f79f0f53152918be1672a4f1d9f3d1fcd03f0f443553c9 Nov 04 02:38:44.033308 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:38:44.033276 4405 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-54497bf445-qvs4m" event={"ID":"8f77d20d-3db8-47dc-a6ee-dec3f1b68432","Type":"ContainerStarted","Data":"6f32935c2073c7a087f79f0f53152918be1672a4f1d9f3d1fcd03f0f443553c9"} Nov 04 02:38:47.047910 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:38:47.047874 4405 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-54497bf445-qvs4m" event={"ID":"8f77d20d-3db8-47dc-a6ee-dec3f1b68432","Type":"ContainerStarted","Data":"4a0106e74d5093069e53f3aca67c3c8cd70b64a1cf9d9dbef972bb007c8126af"} Nov 04 02:38:47.049694 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:38:47.049646 4405 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-7b7bb4bc77-rm4jt" event={"ID":"f33c29b3-72db-4504-b751-2a2666001277","Type":"ContainerStarted","Data":"d4596fc73c37ba66b0c43b67879d7a7b7210482ae1b78b41c6fe2aff9d6d91b4"} Nov 04 02:38:47.049828 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:38:47.049790 4405 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/authorino-operator-7b7bb4bc77-rm4jt" Nov 04 02:38:47.163848 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:38:47.163794 4405 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/kube-auth-proxy-54497bf445-qvs4m" podStartSLOduration=1.021843121 podStartE2EDuration="4.163778483s" totalImagesPullingTime="3.141935362s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 02:38:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 02:38:47.162976181 +0000 UTC m=+373.593676302" watchObservedRunningTime="2026-11-04 02:38:47.163778483 +0000 UTC m=+373.594478603" Nov 04 02:38:47.281419 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:38:47.281298 4405 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/authorino-operator-7b7bb4bc77-rm4jt" podStartSLOduration=2.525530538 podStartE2EDuration="6.281283753s" totalImagesPullingTime="3.755753215s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 02:38:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 02:38:47.280386866 +0000 UTC m=+373.711086985" watchObservedRunningTime="2026-11-04 02:38:47.281283753 +0000 UTC m=+373.711983942" Nov 04 02:38:54.598062 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:38:54.598027 4405 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-mw848"] Nov 04 02:38:54.601332 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:38:54.601316 4405 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-mw848" Nov 04 02:38:54.695294 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:38:54.695259 4405 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gtf72\" (UniqueName: \"kubernetes.io/projected/f540e538-4c5c-41eb-bb64-0d964c4810ea-kube-api-access-gtf72\") pod \"kuadrant-operator-controller-manager-678d87766f-mw848\" (UID: \"f540e538-4c5c-41eb-bb64-0d964c4810ea\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-mw848" Nov 04 02:38:54.695294 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:38:54.695298 4405 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/f540e538-4c5c-41eb-bb64-0d964c4810ea-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-678d87766f-mw848\" (UID: \"f540e538-4c5c-41eb-bb64-0d964c4810ea\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-mw848" Nov 04 02:38:54.706346 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:38:54.706314 4405 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-pbzpn\"" Nov 04 02:38:54.707432 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:38:54.707414 4405 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-mw848"] Nov 04 02:38:54.796740 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:38:54.796700 4405 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gtf72\" (UniqueName: \"kubernetes.io/projected/f540e538-4c5c-41eb-bb64-0d964c4810ea-kube-api-access-gtf72\") pod \"kuadrant-operator-controller-manager-678d87766f-mw848\" (UID: \"f540e538-4c5c-41eb-bb64-0d964c4810ea\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-mw848" Nov 04 02:38:54.796740 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:38:54.796744 4405 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/f540e538-4c5c-41eb-bb64-0d964c4810ea-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-678d87766f-mw848\" (UID: \"f540e538-4c5c-41eb-bb64-0d964c4810ea\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-mw848" Nov 04 02:38:54.797096 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:38:54.797076 4405 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/f540e538-4c5c-41eb-bb64-0d964c4810ea-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-678d87766f-mw848\" (UID: \"f540e538-4c5c-41eb-bb64-0d964c4810ea\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-mw848" Nov 04 02:38:54.866810 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:38:54.866747 4405 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gtf72\" (UniqueName: \"kubernetes.io/projected/f540e538-4c5c-41eb-bb64-0d964c4810ea-kube-api-access-gtf72\") pod \"kuadrant-operator-controller-manager-678d87766f-mw848\" (UID: \"f540e538-4c5c-41eb-bb64-0d964c4810ea\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-mw848" Nov 04 02:38:54.913741 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:38:54.913714 4405 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-mw848" Nov 04 02:38:55.386051 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:38:55.386019 4405 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-mw848"] Nov 04 02:38:55.424862 ip-10-0-134-125 kubenswrapper[4405]: W1104 02:38:55.424832 4405 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf540e538_4c5c_41eb_bb64_0d964c4810ea.slice/crio-a8b27ca9db13255821ac7b9ecf2ab016c3eafb11d08f35c81ab8ac7bdbfc2af3 WatchSource:0}: Error finding container a8b27ca9db13255821ac7b9ecf2ab016c3eafb11d08f35c81ab8ac7bdbfc2af3: Status 404 returned error can't find the container with id a8b27ca9db13255821ac7b9ecf2ab016c3eafb11d08f35c81ab8ac7bdbfc2af3 Nov 04 02:38:56.084476 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:38:56.084438 4405 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-mw848" event={"ID":"f540e538-4c5c-41eb-bb64-0d964c4810ea","Type":"ContainerStarted","Data":"a8b27ca9db13255821ac7b9ecf2ab016c3eafb11d08f35c81ab8ac7bdbfc2af3"} Nov 04 02:38:58.055772 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:38:58.055741 4405 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/authorino-operator-7b7bb4bc77-rm4jt" Nov 04 02:39:03.116411 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:39:03.116332 4405 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-mw848" event={"ID":"f540e538-4c5c-41eb-bb64-0d964c4810ea","Type":"ContainerStarted","Data":"36a4c11b22e50789d76014dba1026875b618bab5e94cae29a2aff907507b0031"} Nov 04 02:39:03.116863 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:39:03.116460 4405 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-mw848" Nov 04 02:39:03.334836 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:39:03.334780 4405 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-mw848" podStartSLOduration=2.528175463 podStartE2EDuration="9.334762224s" totalImagesPullingTime="6.806586761s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 02:38:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 02:39:03.284106053 +0000 UTC m=+389.714806174" watchObservedRunningTime="2026-11-04 02:39:03.334762224 +0000 UTC m=+389.765462344" Nov 04 02:39:04.059507 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:39:04.059476 4405 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-bl5mg"] Nov 04 02:39:04.062989 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:39:04.062962 4405 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-7c9b87d574-bl5mg" Nov 04 02:39:04.115090 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:39:04.115061 4405 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"limitador-limits-config-limitador\"" Nov 04 02:39:04.115239 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:39:04.115151 4405 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-4fz9l\"" Nov 04 02:39:04.165778 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:39:04.165744 4405 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-bl5mg"] Nov 04 02:39:04.181122 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:39:04.181087 4405 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/464f01e2-36a3-4d4b-b92c-a15b621b76a9-config-file\") pod \"limitador-limitador-7c9b87d574-bl5mg\" (UID: \"464f01e2-36a3-4d4b-b92c-a15b621b76a9\") " pod="kuadrant-system/limitador-limitador-7c9b87d574-bl5mg" Nov 04 02:39:04.181283 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:39:04.181202 4405 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xkfnm\" (UniqueName: \"kubernetes.io/projected/464f01e2-36a3-4d4b-b92c-a15b621b76a9-kube-api-access-xkfnm\") pod \"limitador-limitador-7c9b87d574-bl5mg\" (UID: \"464f01e2-36a3-4d4b-b92c-a15b621b76a9\") " pod="kuadrant-system/limitador-limitador-7c9b87d574-bl5mg" Nov 04 02:39:04.282075 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:39:04.282037 4405 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xkfnm\" (UniqueName: \"kubernetes.io/projected/464f01e2-36a3-4d4b-b92c-a15b621b76a9-kube-api-access-xkfnm\") pod \"limitador-limitador-7c9b87d574-bl5mg\" (UID: \"464f01e2-36a3-4d4b-b92c-a15b621b76a9\") " pod="kuadrant-system/limitador-limitador-7c9b87d574-bl5mg" Nov 04 02:39:04.282247 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:39:04.282086 4405 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/464f01e2-36a3-4d4b-b92c-a15b621b76a9-config-file\") pod \"limitador-limitador-7c9b87d574-bl5mg\" (UID: \"464f01e2-36a3-4d4b-b92c-a15b621b76a9\") " pod="kuadrant-system/limitador-limitador-7c9b87d574-bl5mg" Nov 04 02:39:04.282782 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:39:04.282763 4405 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/464f01e2-36a3-4d4b-b92c-a15b621b76a9-config-file\") pod \"limitador-limitador-7c9b87d574-bl5mg\" (UID: \"464f01e2-36a3-4d4b-b92c-a15b621b76a9\") " pod="kuadrant-system/limitador-limitador-7c9b87d574-bl5mg" Nov 04 02:39:04.342762 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:39:04.342726 4405 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xkfnm\" (UniqueName: \"kubernetes.io/projected/464f01e2-36a3-4d4b-b92c-a15b621b76a9-kube-api-access-xkfnm\") pod \"limitador-limitador-7c9b87d574-bl5mg\" (UID: \"464f01e2-36a3-4d4b-b92c-a15b621b76a9\") " pod="kuadrant-system/limitador-limitador-7c9b87d574-bl5mg" Nov 04 02:39:04.367231 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:39:04.367194 4405 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-bl5mg"] Nov 04 02:39:04.375047 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:39:04.375023 4405 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-7c9b87d574-bl5mg" Nov 04 02:39:04.746937 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:39:04.746912 4405 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-bl5mg"] Nov 04 02:39:04.749190 ip-10-0-134-125 kubenswrapper[4405]: W1104 02:39:04.749162 4405 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod464f01e2_36a3_4d4b_b92c_a15b621b76a9.slice/crio-6526d18ff9a2b7c96a8e056adad6492635ae64856cd39e24b297a13781d2c644 WatchSource:0}: Error finding container 6526d18ff9a2b7c96a8e056adad6492635ae64856cd39e24b297a13781d2c644: Status 404 returned error can't find the container with id 6526d18ff9a2b7c96a8e056adad6492635ae64856cd39e24b297a13781d2c644 Nov 04 02:39:05.126398 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:39:05.126368 4405 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-7c9b87d574-bl5mg" event={"ID":"464f01e2-36a3-4d4b-b92c-a15b621b76a9","Type":"ContainerStarted","Data":"6526d18ff9a2b7c96a8e056adad6492635ae64856cd39e24b297a13781d2c644"} Nov 04 02:39:06.886130 ip-10-0-134-125 kubenswrapper[4405]: E1104 02:39:06.886087 4405 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5f2vb" containerName="extractor" Nov 04 02:39:08.139778 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:39:08.139737 4405 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-7c9b87d574-bl5mg" event={"ID":"464f01e2-36a3-4d4b-b92c-a15b621b76a9","Type":"ContainerStarted","Data":"3121445466a1c3b68a45b601e3f7079efc01b21838496b310e5d05410031a163"} Nov 04 02:39:08.140186 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:39:08.139888 4405 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-7c9b87d574-bl5mg" Nov 04 02:39:08.256141 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:39:08.256090 4405 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-7c9b87d574-bl5mg" podStartSLOduration=1.7569528540000001 podStartE2EDuration="4.25607518s" totalImagesPullingTime="2.499122326s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 02:39:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 02:39:08.254646532 +0000 UTC m=+394.685346653" watchObservedRunningTime="2026-11-04 02:39:08.25607518 +0000 UTC m=+394.686775300" Nov 04 02:39:14.123354 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:39:14.123325 4405 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-mw848" Nov 04 02:39:19.144227 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:39:19.144199 4405 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-7c9b87d574-bl5mg" Nov 04 02:39:36.882725 ip-10-0-134-125 kubenswrapper[4405]: E1104 02:39:36.882659 4405 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5f2vb" containerName="extractor" Nov 04 02:40:06.883140 ip-10-0-134-125 kubenswrapper[4405]: E1104 02:40:06.883098 4405 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5f2vb" containerName="extractor" Nov 04 02:40:10.267364 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:40:10.267324 4405 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-vj798"] Nov 04 02:40:10.269351 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:40:10.269313 4405 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-vj798" Nov 04 02:40:10.321403 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:40:10.321366 4405 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"usage-logs-tenancy-proxy-tls\"" Nov 04 02:40:10.321595 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:40:10.321579 4405 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\"" Nov 04 02:40:10.352029 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:40:10.351996 4405 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/214526eb-504c-474f-8fbb-fcfe9438586a-service-ca\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-vj798\" (UID: \"214526eb-504c-474f-8fbb-fcfe9438586a\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-vj798" Nov 04 02:40:10.352187 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:40:10.352037 4405 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/214526eb-504c-474f-8fbb-fcfe9438586a-tls-cert\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-vj798\" (UID: \"214526eb-504c-474f-8fbb-fcfe9438586a\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-vj798" Nov 04 02:40:10.352187 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:40:10.352063 4405 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-85swv\" (UniqueName: \"kubernetes.io/projected/214526eb-504c-474f-8fbb-fcfe9438586a-kube-api-access-85swv\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-vj798\" (UID: \"214526eb-504c-474f-8fbb-fcfe9438586a\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-vj798" Nov 04 02:40:10.352256 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:40:10.352191 4405 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-source\" (UniqueName: \"kubernetes.io/configmap/214526eb-504c-474f-8fbb-fcfe9438586a-proxy-source\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-vj798\" (UID: \"214526eb-504c-474f-8fbb-fcfe9438586a\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-vj798" Nov 04 02:40:10.371071 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:40:10.371029 4405 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"usage-tenancy-proxy-source\"" Nov 04 02:40:10.371253 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:40:10.371093 4405 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\"" Nov 04 02:40:10.371253 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:40:10.371100 4405 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"usage-logs-tenancy-proxy-ca-bundle\"" Nov 04 02:40:10.371253 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:40:10.371157 4405 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"usage-tenancy-proxy-dockercfg-qqwct\"" Nov 04 02:40:10.372873 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:40:10.372849 4405 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-vj798"] Nov 04 02:40:10.452536 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:40:10.452503 4405 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/214526eb-504c-474f-8fbb-fcfe9438586a-service-ca\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-vj798\" (UID: \"214526eb-504c-474f-8fbb-fcfe9438586a\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-vj798" Nov 04 02:40:10.452731 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:40:10.452543 4405 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/214526eb-504c-474f-8fbb-fcfe9438586a-tls-cert\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-vj798\" (UID: \"214526eb-504c-474f-8fbb-fcfe9438586a\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-vj798" Nov 04 02:40:10.452731 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:40:10.452582 4405 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-85swv\" (UniqueName: \"kubernetes.io/projected/214526eb-504c-474f-8fbb-fcfe9438586a-kube-api-access-85swv\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-vj798\" (UID: \"214526eb-504c-474f-8fbb-fcfe9438586a\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-vj798" Nov 04 02:40:10.452731 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:40:10.452640 4405 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-source\" (UniqueName: \"kubernetes.io/configmap/214526eb-504c-474f-8fbb-fcfe9438586a-proxy-source\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-vj798\" (UID: \"214526eb-504c-474f-8fbb-fcfe9438586a\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-vj798" Nov 04 02:40:10.452904 ip-10-0-134-125 kubenswrapper[4405]: E1104 02:40:10.452726 4405 secret.go:189] Couldn't get secret opendatahub/usage-logs-tenancy-proxy-tls: secret "usage-logs-tenancy-proxy-tls" not found Nov 04 02:40:10.452904 ip-10-0-134-125 kubenswrapper[4405]: E1104 02:40:10.452734 4405 configmap.go:193] Couldn't get configMap opendatahub/usage-tenancy-proxy-source: configmap "usage-tenancy-proxy-source" not found Nov 04 02:40:10.452904 ip-10-0-134-125 kubenswrapper[4405]: E1104 02:40:10.452807 4405 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/214526eb-504c-474f-8fbb-fcfe9438586a-tls-cert podName:214526eb-504c-474f-8fbb-fcfe9438586a nodeName:}" failed. No retries permitted until 2026-11-04 02:40:10.952790524 +0000 UTC m=+457.383490625 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "tls-cert" (UniqueName: "kubernetes.io/secret/214526eb-504c-474f-8fbb-fcfe9438586a-tls-cert") pod "usage-logs-tenancy-proxy-9c5d49bcd-vj798" (UID: "214526eb-504c-474f-8fbb-fcfe9438586a") : secret "usage-logs-tenancy-proxy-tls" not found Nov 04 02:40:10.452904 ip-10-0-134-125 kubenswrapper[4405]: E1104 02:40:10.452827 4405 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/214526eb-504c-474f-8fbb-fcfe9438586a-proxy-source podName:214526eb-504c-474f-8fbb-fcfe9438586a nodeName:}" failed. No retries permitted until 2026-11-04 02:40:10.952820821 +0000 UTC m=+457.383520956 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-source" (UniqueName: "kubernetes.io/configmap/214526eb-504c-474f-8fbb-fcfe9438586a-proxy-source") pod "usage-logs-tenancy-proxy-9c5d49bcd-vj798" (UID: "214526eb-504c-474f-8fbb-fcfe9438586a") : configmap "usage-tenancy-proxy-source" not found Nov 04 02:40:10.453273 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:40:10.453255 4405 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/214526eb-504c-474f-8fbb-fcfe9438586a-service-ca\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-vj798\" (UID: \"214526eb-504c-474f-8fbb-fcfe9438586a\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-vj798" Nov 04 02:40:10.515849 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:40:10.515817 4405 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-85swv\" (UniqueName: \"kubernetes.io/projected/214526eb-504c-474f-8fbb-fcfe9438586a-kube-api-access-85swv\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-vj798\" (UID: \"214526eb-504c-474f-8fbb-fcfe9438586a\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-vj798" Nov 04 02:40:10.957466 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:40:10.957425 4405 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/214526eb-504c-474f-8fbb-fcfe9438586a-tls-cert\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-vj798\" (UID: \"214526eb-504c-474f-8fbb-fcfe9438586a\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-vj798" Nov 04 02:40:10.957723 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:40:10.957496 4405 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-source\" (UniqueName: \"kubernetes.io/configmap/214526eb-504c-474f-8fbb-fcfe9438586a-proxy-source\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-vj798\" (UID: \"214526eb-504c-474f-8fbb-fcfe9438586a\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-vj798" Nov 04 02:40:10.958299 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:40:10.958275 4405 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-source\" (UniqueName: \"kubernetes.io/configmap/214526eb-504c-474f-8fbb-fcfe9438586a-proxy-source\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-vj798\" (UID: \"214526eb-504c-474f-8fbb-fcfe9438586a\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-vj798" Nov 04 02:40:10.960129 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:40:10.960107 4405 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/214526eb-504c-474f-8fbb-fcfe9438586a-tls-cert\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-vj798\" (UID: \"214526eb-504c-474f-8fbb-fcfe9438586a\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-vj798" Nov 04 02:40:11.183753 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:40:11.183699 4405 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-vj798" Nov 04 02:40:11.555234 ip-10-0-134-125 kubenswrapper[4405]: W1104 02:40:11.555199 4405 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod214526eb_504c_474f_8fbb_fcfe9438586a.slice/crio-19f6a3128bf6fcafcceb197c7f86ef98940ae12d31a446c7da63d44ee5dde1a4 WatchSource:0}: Error finding container 19f6a3128bf6fcafcceb197c7f86ef98940ae12d31a446c7da63d44ee5dde1a4: Status 404 returned error can't find the container with id 19f6a3128bf6fcafcceb197c7f86ef98940ae12d31a446c7da63d44ee5dde1a4 Nov 04 02:40:11.598826 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:40:11.598791 4405 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-vj798"] Nov 04 02:40:12.392099 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:40:12.392063 4405 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-vj798" event={"ID":"214526eb-504c-474f-8fbb-fcfe9438586a","Type":"ContainerStarted","Data":"19f6a3128bf6fcafcceb197c7f86ef98940ae12d31a446c7da63d44ee5dde1a4"} Nov 04 02:40:24.448846 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:40:24.448807 4405 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-vj798" event={"ID":"214526eb-504c-474f-8fbb-fcfe9438586a","Type":"ContainerStarted","Data":"a402fb70f5b457eb59f18d61ef8723f3afc0fe9f36a2144b04b4f06c7294ad76"} Nov 04 02:40:24.449325 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:40:24.448958 4405 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-vj798" Nov 04 02:40:24.565454 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:40:24.565345 4405 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-vj798" podStartSLOduration=1.851862606 podStartE2EDuration="14.565329391s" totalImagesPullingTime="12.713466785s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 02:40:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 02:40:24.564091458 +0000 UTC m=+470.994791578" watchObservedRunningTime="2026-11-04 02:40:24.565329391 +0000 UTC m=+470.996029511" Nov 04 02:40:35.459455 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:40:35.459419 4405 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-vj798" Nov 04 02:40:36.883334 ip-10-0-134-125 kubenswrapper[4405]: E1104 02:40:36.883284 4405 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5f2vb" containerName="extractor" Nov 04 02:40:43.263467 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:40:43.263430 4405 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-674d648f7b-tfqdq"] Nov 04 02:40:43.444494 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:40:43.444464 4405 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-674d648f7b-tfqdq"] Nov 04 02:40:43.444695 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:40:43.444590 4405 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-674d648f7b-tfqdq" Nov 04 02:40:43.448603 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:40:43.448574 4405 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0d9cb659-370e-466e-81b5-f7d3b6d181ca-maas-api-tls\") pod \"maas-api-674d648f7b-tfqdq\" (UID: \"0d9cb659-370e-466e-81b5-f7d3b6d181ca\") " pod="odh-ai-gateway-infra/maas-api-674d648f7b-tfqdq" Nov 04 02:40:43.448603 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:40:43.448602 4405 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0d9cb659-370e-466e-81b5-f7d3b6d181ca-metrics-certs\") pod \"maas-api-674d648f7b-tfqdq\" (UID: \"0d9cb659-370e-466e-81b5-f7d3b6d181ca\") " pod="odh-ai-gateway-infra/maas-api-674d648f7b-tfqdq" Nov 04 02:40:43.448775 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:40:43.448626 4405 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jh8kx\" (UniqueName: \"kubernetes.io/projected/0d9cb659-370e-466e-81b5-f7d3b6d181ca-kube-api-access-jh8kx\") pod \"maas-api-674d648f7b-tfqdq\" (UID: \"0d9cb659-370e-466e-81b5-f7d3b6d181ca\") " pod="odh-ai-gateway-infra/maas-api-674d648f7b-tfqdq" Nov 04 02:40:43.448810 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:40:43.448783 4405 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0d9cb659-370e-466e-81b5-f7d3b6d181ca-openshift-service-ca\") pod \"maas-api-674d648f7b-tfqdq\" (UID: \"0d9cb659-370e-466e-81b5-f7d3b6d181ca\") " pod="odh-ai-gateway-infra/maas-api-674d648f7b-tfqdq" Nov 04 02:40:43.546283 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:40:43.546191 4405 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 04 02:40:43.546283 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:40:43.546273 4405 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-qtjr2\"" Nov 04 02:40:43.546477 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:40:43.546299 4405 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 04 02:40:43.546477 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:40:43.546303 4405 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-serving-cert\"" Nov 04 02:40:43.546477 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:40:43.546273 4405 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\"" Nov 04 02:40:43.549712 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:40:43.549693 4405 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0d9cb659-370e-466e-81b5-f7d3b6d181ca-openshift-service-ca\") pod \"maas-api-674d648f7b-tfqdq\" (UID: \"0d9cb659-370e-466e-81b5-f7d3b6d181ca\") " pod="odh-ai-gateway-infra/maas-api-674d648f7b-tfqdq" Nov 04 02:40:43.549765 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:40:43.549739 4405 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0d9cb659-370e-466e-81b5-f7d3b6d181ca-maas-api-tls\") pod \"maas-api-674d648f7b-tfqdq\" (UID: \"0d9cb659-370e-466e-81b5-f7d3b6d181ca\") " pod="odh-ai-gateway-infra/maas-api-674d648f7b-tfqdq" Nov 04 02:40:43.549765 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:40:43.549757 4405 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0d9cb659-370e-466e-81b5-f7d3b6d181ca-metrics-certs\") pod \"maas-api-674d648f7b-tfqdq\" (UID: \"0d9cb659-370e-466e-81b5-f7d3b6d181ca\") " pod="odh-ai-gateway-infra/maas-api-674d648f7b-tfqdq" Nov 04 02:40:43.549836 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:40:43.549777 4405 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jh8kx\" (UniqueName: \"kubernetes.io/projected/0d9cb659-370e-466e-81b5-f7d3b6d181ca-kube-api-access-jh8kx\") pod \"maas-api-674d648f7b-tfqdq\" (UID: \"0d9cb659-370e-466e-81b5-f7d3b6d181ca\") " pod="odh-ai-gateway-infra/maas-api-674d648f7b-tfqdq" Nov 04 02:40:43.550350 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:40:43.550323 4405 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0d9cb659-370e-466e-81b5-f7d3b6d181ca-openshift-service-ca\") pod \"maas-api-674d648f7b-tfqdq\" (UID: \"0d9cb659-370e-466e-81b5-f7d3b6d181ca\") " pod="odh-ai-gateway-infra/maas-api-674d648f7b-tfqdq" Nov 04 02:40:43.552352 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:40:43.552332 4405 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0d9cb659-370e-466e-81b5-f7d3b6d181ca-maas-api-tls\") pod \"maas-api-674d648f7b-tfqdq\" (UID: \"0d9cb659-370e-466e-81b5-f7d3b6d181ca\") " pod="odh-ai-gateway-infra/maas-api-674d648f7b-tfqdq" Nov 04 02:40:43.552438 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:40:43.552396 4405 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0d9cb659-370e-466e-81b5-f7d3b6d181ca-metrics-certs\") pod \"maas-api-674d648f7b-tfqdq\" (UID: \"0d9cb659-370e-466e-81b5-f7d3b6d181ca\") " pod="odh-ai-gateway-infra/maas-api-674d648f7b-tfqdq" Nov 04 02:40:43.610807 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:40:43.610773 4405 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jh8kx\" (UniqueName: \"kubernetes.io/projected/0d9cb659-370e-466e-81b5-f7d3b6d181ca-kube-api-access-jh8kx\") pod \"maas-api-674d648f7b-tfqdq\" (UID: \"0d9cb659-370e-466e-81b5-f7d3b6d181ca\") " pod="odh-ai-gateway-infra/maas-api-674d648f7b-tfqdq" Nov 04 02:40:43.775516 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:40:43.775476 4405 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-674d648f7b-tfqdq" Nov 04 02:40:44.169880 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:40:44.169853 4405 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-674d648f7b-tfqdq"] Nov 04 02:40:44.174873 ip-10-0-134-125 kubenswrapper[4405]: W1104 02:40:44.174843 4405 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0d9cb659_370e_466e_81b5_f7d3b6d181ca.slice/crio-424dd46e854ecdade7380384cc305d4b00eb78b30087afedf7a5fb277b45530d WatchSource:0}: Error finding container 424dd46e854ecdade7380384cc305d4b00eb78b30087afedf7a5fb277b45530d: Status 404 returned error can't find the container with id 424dd46e854ecdade7380384cc305d4b00eb78b30087afedf7a5fb277b45530d Nov 04 02:40:44.528290 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:40:44.528206 4405 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-674d648f7b-tfqdq" event={"ID":"0d9cb659-370e-466e-81b5-f7d3b6d181ca","Type":"ContainerStarted","Data":"424dd46e854ecdade7380384cc305d4b00eb78b30087afedf7a5fb277b45530d"} Nov 04 02:40:48.547915 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:40:48.547880 4405 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-674d648f7b-tfqdq" event={"ID":"0d9cb659-370e-466e-81b5-f7d3b6d181ca","Type":"ContainerStarted","Data":"df26c6770663538af6439c02452608cb46343dc5a1bc1a9dbd3d730aa3173cb5"} Nov 04 02:40:48.548446 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:40:48.547954 4405 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-674d648f7b-tfqdq" Nov 04 02:40:48.665148 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:40:48.665099 4405 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-674d648f7b-tfqdq" podStartSLOduration=2.223124029 podStartE2EDuration="5.665073808s" totalImagesPullingTime="3.441949779s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 02:40:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 02:40:48.663322457 +0000 UTC m=+495.094022576" watchObservedRunningTime="2026-11-04 02:40:48.665073808 +0000 UTC m=+495.095773928" Nov 04 02:40:48.835076 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:40:48.835043 4405 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-bl5mg"] Nov 04 02:40:54.557110 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:40:54.557078 4405 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-674d648f7b-tfqdq" Nov 04 02:41:05.971731 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:41:05.971696 4405 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-674d648f7b-tfqdq"] Nov 04 02:41:05.972186 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:41:05.971906 4405 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-674d648f7b-tfqdq" podUID="0d9cb659-370e-466e-81b5-f7d3b6d181ca" containerName="maas-api" containerID="cri-o://df26c6770663538af6439c02452608cb46343dc5a1bc1a9dbd3d730aa3173cb5" gracePeriod=30 Nov 04 02:41:06.214986 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:41:06.214958 4405 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-674d648f7b-tfqdq" Nov 04 02:41:06.312371 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:41:06.312272 4405 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0d9cb659-370e-466e-81b5-f7d3b6d181ca-metrics-certs\" (UniqueName: \"kubernetes.io/secret/0d9cb659-370e-466e-81b5-f7d3b6d181ca-metrics-certs\") pod \"0d9cb659-370e-466e-81b5-f7d3b6d181ca\" (UID: \"0d9cb659-370e-466e-81b5-f7d3b6d181ca\") " Nov 04 02:41:06.312371 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:41:06.312319 4405 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0d9cb659-370e-466e-81b5-f7d3b6d181ca-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0d9cb659-370e-466e-81b5-f7d3b6d181ca-maas-api-tls\") pod \"0d9cb659-370e-466e-81b5-f7d3b6d181ca\" (UID: \"0d9cb659-370e-466e-81b5-f7d3b6d181ca\") " Nov 04 02:41:06.312608 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:41:06.312371 4405 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0d9cb659-370e-466e-81b5-f7d3b6d181ca-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0d9cb659-370e-466e-81b5-f7d3b6d181ca-openshift-service-ca\") pod \"0d9cb659-370e-466e-81b5-f7d3b6d181ca\" (UID: \"0d9cb659-370e-466e-81b5-f7d3b6d181ca\") " Nov 04 02:41:06.312608 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:41:06.312403 4405 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0d9cb659-370e-466e-81b5-f7d3b6d181ca-kube-api-access-jh8kx\" (UniqueName: \"kubernetes.io/projected/0d9cb659-370e-466e-81b5-f7d3b6d181ca-kube-api-access-jh8kx\") pod \"0d9cb659-370e-466e-81b5-f7d3b6d181ca\" (UID: \"0d9cb659-370e-466e-81b5-f7d3b6d181ca\") " Nov 04 02:41:06.312823 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:41:06.312793 4405 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0d9cb659-370e-466e-81b5-f7d3b6d181ca-openshift-service-ca" pod "0d9cb659-370e-466e-81b5-f7d3b6d181ca" (UID: "0d9cb659-370e-466e-81b5-f7d3b6d181ca"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 02:41:06.314770 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:41:06.314743 4405 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0d9cb659-370e-466e-81b5-f7d3b6d181ca-metrics-certs" pod "0d9cb659-370e-466e-81b5-f7d3b6d181ca" (UID: "0d9cb659-370e-466e-81b5-f7d3b6d181ca"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 04 02:41:06.314855 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:41:06.314767 4405 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0d9cb659-370e-466e-81b5-f7d3b6d181ca-maas-api-tls" pod "0d9cb659-370e-466e-81b5-f7d3b6d181ca" (UID: "0d9cb659-370e-466e-81b5-f7d3b6d181ca"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 04 02:41:06.314855 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:41:06.314796 4405 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0d9cb659-370e-466e-81b5-f7d3b6d181ca-kube-api-access-jh8kx" pod "0d9cb659-370e-466e-81b5-f7d3b6d181ca" (UID: "0d9cb659-370e-466e-81b5-f7d3b6d181ca"). InnerVolumeSpecName "kube-api-access-jh8kx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 02:41:06.413614 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:41:06.413572 4405 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0d9cb659-370e-466e-81b5-f7d3b6d181ca-openshift-service-ca\") on node \"ip-10-0-134-125.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 02:41:06.413614 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:41:06.413607 4405 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jh8kx\" (UniqueName: \"kubernetes.io/projected/0d9cb659-370e-466e-81b5-f7d3b6d181ca-kube-api-access-jh8kx\") on node \"ip-10-0-134-125.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 02:41:06.413614 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:41:06.413619 4405 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0d9cb659-370e-466e-81b5-f7d3b6d181ca-metrics-certs\") on node \"ip-10-0-134-125.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 02:41:06.413886 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:41:06.413628 4405 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0d9cb659-370e-466e-81b5-f7d3b6d181ca-maas-api-tls\") on node \"ip-10-0-134-125.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 02:41:06.622079 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:41:06.622040 4405 generic.go:356] "Generic (PLEG): container finished" podID="0d9cb659-370e-466e-81b5-f7d3b6d181ca" containerID="df26c6770663538af6439c02452608cb46343dc5a1bc1a9dbd3d730aa3173cb5" exitCode=0 Nov 04 02:41:06.622255 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:41:06.622090 4405 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-674d648f7b-tfqdq" Nov 04 02:41:06.622255 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:41:06.622110 4405 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-674d648f7b-tfqdq" event={"ID":"0d9cb659-370e-466e-81b5-f7d3b6d181ca","Type":"ContainerDied","Data":"df26c6770663538af6439c02452608cb46343dc5a1bc1a9dbd3d730aa3173cb5"} Nov 04 02:41:06.622255 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:41:06.622163 4405 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-674d648f7b-tfqdq" event={"ID":"0d9cb659-370e-466e-81b5-f7d3b6d181ca","Type":"ContainerDied","Data":"424dd46e854ecdade7380384cc305d4b00eb78b30087afedf7a5fb277b45530d"} Nov 04 02:41:06.622255 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:41:06.622194 4405 scope.go:122] "RemoveContainer" containerID="df26c6770663538af6439c02452608cb46343dc5a1bc1a9dbd3d730aa3173cb5" Nov 04 02:41:06.630955 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:41:06.630936 4405 scope.go:122] "RemoveContainer" containerID="df26c6770663538af6439c02452608cb46343dc5a1bc1a9dbd3d730aa3173cb5" Nov 04 02:41:06.631245 ip-10-0-134-125 kubenswrapper[4405]: E1104 02:41:06.631226 4405 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"df26c6770663538af6439c02452608cb46343dc5a1bc1a9dbd3d730aa3173cb5\": container with ID starting with df26c6770663538af6439c02452608cb46343dc5a1bc1a9dbd3d730aa3173cb5 not found: ID does not exist" containerID="df26c6770663538af6439c02452608cb46343dc5a1bc1a9dbd3d730aa3173cb5" Nov 04 02:41:06.631293 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:41:06.631253 4405 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"df26c6770663538af6439c02452608cb46343dc5a1bc1a9dbd3d730aa3173cb5"} err="failed to get container status \"df26c6770663538af6439c02452608cb46343dc5a1bc1a9dbd3d730aa3173cb5\": rpc error: code = NotFound desc = could not find container \"df26c6770663538af6439c02452608cb46343dc5a1bc1a9dbd3d730aa3173cb5\": container with ID starting with df26c6770663538af6439c02452608cb46343dc5a1bc1a9dbd3d730aa3173cb5 not found: ID does not exist" Nov 04 02:41:06.787961 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:41:06.787179 4405 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-674d648f7b-tfqdq"] Nov 04 02:41:06.791030 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:41:06.791000 4405 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-674d648f7b-tfqdq"] Nov 04 02:41:06.884300 ip-10-0-134-125 kubenswrapper[4405]: E1104 02:41:06.884207 4405 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5f2vb" containerName="extractor" Nov 04 02:41:07.699030 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:41:07.698996 4405 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0d9cb659-370e-466e-81b5-f7d3b6d181ca" path="/var/lib/kubelet/pods/0d9cb659-370e-466e-81b5-f7d3b6d181ca/volumes" Nov 04 02:41:36.882117 ip-10-0-134-125 kubenswrapper[4405]: E1104 02:41:36.882079 4405 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5f2vb" containerName="extractor" Nov 04 02:42:06.881494 ip-10-0-134-125 kubenswrapper[4405]: E1104 02:42:06.881457 4405 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5f2vb" containerName="extractor" Nov 04 02:42:17.500723 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:42:17.500689 4405 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-ljhhw"] Nov 04 02:42:17.503154 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:42:17.501127 4405 cpu_manager.go:414] "containerMap: removing container" podUID="0d9cb659-370e-466e-81b5-f7d3b6d181ca" containerName="maas-api" Nov 04 02:42:17.503154 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:42:17.501145 4405 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0d9cb659-370e-466e-81b5-f7d3b6d181ca" containerName="maas-api" Nov 04 02:42:17.503154 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:42:17.501233 4405 memory_manager.go:365] "RemoveStaleState removing state" podUID="0d9cb659-370e-466e-81b5-f7d3b6d181ca" containerName="maas-api" Nov 04 02:42:17.504009 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:42:17.503990 4405 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-ljhhw" Nov 04 02:42:17.556085 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:42:17.556059 4405 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\"" Nov 04 02:42:17.591859 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:42:17.591815 4405 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f66b1861-d87b-4d57-ab27-f1a24544a534-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-ljhhw\" (UID: \"f66b1861-d87b-4d57-ab27-f1a24544a534\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-ljhhw" Nov 04 02:42:17.592029 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:42:17.591893 4405 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kvh84\" (UniqueName: \"kubernetes.io/projected/f66b1861-d87b-4d57-ab27-f1a24544a534-kube-api-access-kvh84\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-ljhhw\" (UID: \"f66b1861-d87b-4d57-ab27-f1a24544a534\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-ljhhw" Nov 04 02:42:17.592029 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:42:17.591952 4405 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f66b1861-d87b-4d57-ab27-f1a24544a534-dshm\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-ljhhw\" (UID: \"f66b1861-d87b-4d57-ab27-f1a24544a534\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-ljhhw" Nov 04 02:42:17.592107 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:42:17.592052 4405 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f66b1861-d87b-4d57-ab27-f1a24544a534-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-ljhhw\" (UID: \"f66b1861-d87b-4d57-ab27-f1a24544a534\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-ljhhw" Nov 04 02:42:17.592107 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:42:17.592093 4405 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f66b1861-d87b-4d57-ab27-f1a24544a534-model-cache\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-ljhhw\" (UID: \"f66b1861-d87b-4d57-ab27-f1a24544a534\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-ljhhw" Nov 04 02:42:17.592187 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:42:17.592124 4405 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f66b1861-d87b-4d57-ab27-f1a24544a534-home\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-ljhhw\" (UID: \"f66b1861-d87b-4d57-ab27-f1a24544a534\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-ljhhw" Nov 04 02:42:17.605694 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:42:17.605644 4405 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-w44xl\"" Nov 04 02:42:17.605694 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:42:17.605647 4405 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\"" Nov 04 02:42:17.605898 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:42:17.605701 4405 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"premium-simulated-simulated-premium-kserve-self-signed-certs\"" Nov 04 02:42:17.607052 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:42:17.607031 4405 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-ljhhw"] Nov 04 02:42:17.693344 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:42:17.693307 4405 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f66b1861-d87b-4d57-ab27-f1a24544a534-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-ljhhw\" (UID: \"f66b1861-d87b-4d57-ab27-f1a24544a534\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-ljhhw" Nov 04 02:42:17.693509 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:42:17.693371 4405 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f66b1861-d87b-4d57-ab27-f1a24544a534-model-cache\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-ljhhw\" (UID: \"f66b1861-d87b-4d57-ab27-f1a24544a534\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-ljhhw" Nov 04 02:42:17.693509 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:42:17.693402 4405 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f66b1861-d87b-4d57-ab27-f1a24544a534-home\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-ljhhw\" (UID: \"f66b1861-d87b-4d57-ab27-f1a24544a534\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-ljhhw" Nov 04 02:42:17.693509 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:42:17.693431 4405 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f66b1861-d87b-4d57-ab27-f1a24544a534-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-ljhhw\" (UID: \"f66b1861-d87b-4d57-ab27-f1a24544a534\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-ljhhw" Nov 04 02:42:17.693509 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:42:17.693456 4405 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kvh84\" (UniqueName: \"kubernetes.io/projected/f66b1861-d87b-4d57-ab27-f1a24544a534-kube-api-access-kvh84\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-ljhhw\" (UID: \"f66b1861-d87b-4d57-ab27-f1a24544a534\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-ljhhw" Nov 04 02:42:17.693509 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:42:17.693495 4405 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f66b1861-d87b-4d57-ab27-f1a24544a534-dshm\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-ljhhw\" (UID: \"f66b1861-d87b-4d57-ab27-f1a24544a534\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-ljhhw" Nov 04 02:42:17.693793 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:42:17.693768 4405 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f66b1861-d87b-4d57-ab27-f1a24544a534-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-ljhhw\" (UID: \"f66b1861-d87b-4d57-ab27-f1a24544a534\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-ljhhw" Nov 04 02:42:17.693851 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:42:17.693833 4405 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f66b1861-d87b-4d57-ab27-f1a24544a534-model-cache\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-ljhhw\" (UID: \"f66b1861-d87b-4d57-ab27-f1a24544a534\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-ljhhw" Nov 04 02:42:17.693969 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:42:17.693877 4405 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f66b1861-d87b-4d57-ab27-f1a24544a534-home\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-ljhhw\" (UID: \"f66b1861-d87b-4d57-ab27-f1a24544a534\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-ljhhw" Nov 04 02:42:17.695873 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:42:17.695853 4405 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f66b1861-d87b-4d57-ab27-f1a24544a534-dshm\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-ljhhw\" (UID: \"f66b1861-d87b-4d57-ab27-f1a24544a534\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-ljhhw" Nov 04 02:42:17.696164 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:42:17.696143 4405 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f66b1861-d87b-4d57-ab27-f1a24544a534-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-ljhhw\" (UID: \"f66b1861-d87b-4d57-ab27-f1a24544a534\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-ljhhw" Nov 04 02:42:17.754967 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:42:17.754891 4405 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kvh84\" (UniqueName: \"kubernetes.io/projected/f66b1861-d87b-4d57-ab27-f1a24544a534-kube-api-access-kvh84\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-ljhhw\" (UID: \"f66b1861-d87b-4d57-ab27-f1a24544a534\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-ljhhw" Nov 04 02:42:17.816047 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:42:17.816004 4405 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-ljhhw" Nov 04 02:42:18.189205 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:42:18.189130 4405 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-ljhhw"] Nov 04 02:42:18.191391 ip-10-0-134-125 kubenswrapper[4405]: W1104 02:42:18.191357 4405 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podf66b1861_d87b_4d57_ab27_f1a24544a534.slice/crio-8ba8e5ce74a1c235c02fe3dd58471b67ad28112d8f936f3ab348ab0c31f41c83 WatchSource:0}: Error finding container 8ba8e5ce74a1c235c02fe3dd58471b67ad28112d8f936f3ab348ab0c31f41c83: Status 404 returned error can't find the container with id 8ba8e5ce74a1c235c02fe3dd58471b67ad28112d8f936f3ab348ab0c31f41c83 Nov 04 02:42:18.917731 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:42:18.917694 4405 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-ljhhw" event={"ID":"f66b1861-d87b-4d57-ab27-f1a24544a534","Type":"ContainerStarted","Data":"8ba8e5ce74a1c235c02fe3dd58471b67ad28112d8f936f3ab348ab0c31f41c83"} Nov 04 02:42:20.928182 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:42:20.928093 4405 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-ljhhw" event={"ID":"f66b1861-d87b-4d57-ab27-f1a24544a534","Type":"ContainerStarted","Data":"4c2e9cf7ed5cff557d0b0966565157e787fa89c19a869300d1d643954b2745cc"} Nov 04 02:42:21.047655 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:42:21.047596 4405 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-ljhhw" podStartSLOduration=1.610652405 podStartE2EDuration="4.047581243s" totalImagesPullingTime="2.436928838s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 02:42:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 02:42:21.0465174 +0000 UTC m=+587.477217520" watchObservedRunningTime="2026-11-04 02:42:21.047581243 +0000 UTC m=+587.478281362" Nov 04 02:42:21.087157 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:42:21.087121 4405 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-bl5mg"] Nov 04 02:42:21.400592 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:42:21.400561 4405 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/echo-simulator-kserve-64b6f8f96c-2lg6v"] Nov 04 02:42:21.403912 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:42:21.403892 4405 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/echo-simulator-kserve-64b6f8f96c-2lg6v" Nov 04 02:42:21.456218 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:42:21.456188 4405 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"echo-simulator-kserve-self-signed-certs\"" Nov 04 02:42:21.465058 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:42:21.465024 4405 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/echo-simulator-kserve-64b6f8f96c-2lg6v"] Nov 04 02:42:21.526054 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:42:21.526021 4405 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fe1fd7ca-7aad-4761-abe0-78bbd438ffad-home\") pod \"echo-simulator-kserve-64b6f8f96c-2lg6v\" (UID: \"fe1fd7ca-7aad-4761-abe0-78bbd438ffad\") " pod="llm/echo-simulator-kserve-64b6f8f96c-2lg6v" Nov 04 02:42:21.526245 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:42:21.526074 4405 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fe1fd7ca-7aad-4761-abe0-78bbd438ffad-model-cache\") pod \"echo-simulator-kserve-64b6f8f96c-2lg6v\" (UID: \"fe1fd7ca-7aad-4761-abe0-78bbd438ffad\") " pod="llm/echo-simulator-kserve-64b6f8f96c-2lg6v" Nov 04 02:42:21.526245 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:42:21.526100 4405 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fe1fd7ca-7aad-4761-abe0-78bbd438ffad-tls-certs\") pod \"echo-simulator-kserve-64b6f8f96c-2lg6v\" (UID: \"fe1fd7ca-7aad-4761-abe0-78bbd438ffad\") " pod="llm/echo-simulator-kserve-64b6f8f96c-2lg6v" Nov 04 02:42:21.526245 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:42:21.526231 4405 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fe1fd7ca-7aad-4761-abe0-78bbd438ffad-tmp-dir\") pod \"echo-simulator-kserve-64b6f8f96c-2lg6v\" (UID: \"fe1fd7ca-7aad-4761-abe0-78bbd438ffad\") " pod="llm/echo-simulator-kserve-64b6f8f96c-2lg6v" Nov 04 02:42:21.526378 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:42:21.526275 4405 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fe1fd7ca-7aad-4761-abe0-78bbd438ffad-dshm\") pod \"echo-simulator-kserve-64b6f8f96c-2lg6v\" (UID: \"fe1fd7ca-7aad-4761-abe0-78bbd438ffad\") " pod="llm/echo-simulator-kserve-64b6f8f96c-2lg6v" Nov 04 02:42:21.526378 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:42:21.526362 4405 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7c5v4\" (UniqueName: \"kubernetes.io/projected/fe1fd7ca-7aad-4761-abe0-78bbd438ffad-kube-api-access-7c5v4\") pod \"echo-simulator-kserve-64b6f8f96c-2lg6v\" (UID: \"fe1fd7ca-7aad-4761-abe0-78bbd438ffad\") " pod="llm/echo-simulator-kserve-64b6f8f96c-2lg6v" Nov 04 02:42:21.627318 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:42:21.627281 4405 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fe1fd7ca-7aad-4761-abe0-78bbd438ffad-home\") pod \"echo-simulator-kserve-64b6f8f96c-2lg6v\" (UID: \"fe1fd7ca-7aad-4761-abe0-78bbd438ffad\") " pod="llm/echo-simulator-kserve-64b6f8f96c-2lg6v" Nov 04 02:42:21.627502 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:42:21.627335 4405 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fe1fd7ca-7aad-4761-abe0-78bbd438ffad-model-cache\") pod \"echo-simulator-kserve-64b6f8f96c-2lg6v\" (UID: \"fe1fd7ca-7aad-4761-abe0-78bbd438ffad\") " pod="llm/echo-simulator-kserve-64b6f8f96c-2lg6v" Nov 04 02:42:21.627502 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:42:21.627352 4405 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fe1fd7ca-7aad-4761-abe0-78bbd438ffad-tls-certs\") pod \"echo-simulator-kserve-64b6f8f96c-2lg6v\" (UID: \"fe1fd7ca-7aad-4761-abe0-78bbd438ffad\") " pod="llm/echo-simulator-kserve-64b6f8f96c-2lg6v" Nov 04 02:42:21.627502 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:42:21.627373 4405 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fe1fd7ca-7aad-4761-abe0-78bbd438ffad-tmp-dir\") pod \"echo-simulator-kserve-64b6f8f96c-2lg6v\" (UID: \"fe1fd7ca-7aad-4761-abe0-78bbd438ffad\") " pod="llm/echo-simulator-kserve-64b6f8f96c-2lg6v" Nov 04 02:42:21.627502 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:42:21.627402 4405 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fe1fd7ca-7aad-4761-abe0-78bbd438ffad-dshm\") pod \"echo-simulator-kserve-64b6f8f96c-2lg6v\" (UID: \"fe1fd7ca-7aad-4761-abe0-78bbd438ffad\") " pod="llm/echo-simulator-kserve-64b6f8f96c-2lg6v" Nov 04 02:42:21.627502 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:42:21.627426 4405 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7c5v4\" (UniqueName: \"kubernetes.io/projected/fe1fd7ca-7aad-4761-abe0-78bbd438ffad-kube-api-access-7c5v4\") pod \"echo-simulator-kserve-64b6f8f96c-2lg6v\" (UID: \"fe1fd7ca-7aad-4761-abe0-78bbd438ffad\") " pod="llm/echo-simulator-kserve-64b6f8f96c-2lg6v" Nov 04 02:42:21.627793 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:42:21.627770 4405 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fe1fd7ca-7aad-4761-abe0-78bbd438ffad-home\") pod \"echo-simulator-kserve-64b6f8f96c-2lg6v\" (UID: \"fe1fd7ca-7aad-4761-abe0-78bbd438ffad\") " pod="llm/echo-simulator-kserve-64b6f8f96c-2lg6v" Nov 04 02:42:21.627853 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:42:21.627814 4405 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fe1fd7ca-7aad-4761-abe0-78bbd438ffad-tmp-dir\") pod \"echo-simulator-kserve-64b6f8f96c-2lg6v\" (UID: \"fe1fd7ca-7aad-4761-abe0-78bbd438ffad\") " pod="llm/echo-simulator-kserve-64b6f8f96c-2lg6v" Nov 04 02:42:21.627853 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:42:21.627835 4405 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fe1fd7ca-7aad-4761-abe0-78bbd438ffad-model-cache\") pod \"echo-simulator-kserve-64b6f8f96c-2lg6v\" (UID: \"fe1fd7ca-7aad-4761-abe0-78bbd438ffad\") " pod="llm/echo-simulator-kserve-64b6f8f96c-2lg6v" Nov 04 02:42:21.629799 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:42:21.629776 4405 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fe1fd7ca-7aad-4761-abe0-78bbd438ffad-dshm\") pod \"echo-simulator-kserve-64b6f8f96c-2lg6v\" (UID: \"fe1fd7ca-7aad-4761-abe0-78bbd438ffad\") " pod="llm/echo-simulator-kserve-64b6f8f96c-2lg6v" Nov 04 02:42:21.629946 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:42:21.629929 4405 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fe1fd7ca-7aad-4761-abe0-78bbd438ffad-tls-certs\") pod \"echo-simulator-kserve-64b6f8f96c-2lg6v\" (UID: \"fe1fd7ca-7aad-4761-abe0-78bbd438ffad\") " pod="llm/echo-simulator-kserve-64b6f8f96c-2lg6v" Nov 04 02:42:21.688331 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:42:21.688253 4405 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7c5v4\" (UniqueName: \"kubernetes.io/projected/fe1fd7ca-7aad-4761-abe0-78bbd438ffad-kube-api-access-7c5v4\") pod \"echo-simulator-kserve-64b6f8f96c-2lg6v\" (UID: \"fe1fd7ca-7aad-4761-abe0-78bbd438ffad\") " pod="llm/echo-simulator-kserve-64b6f8f96c-2lg6v" Nov 04 02:42:21.715790 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:42:21.715751 4405 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/echo-simulator-kserve-64b6f8f96c-2lg6v" Nov 04 02:42:21.816440 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:42:21.816408 4405 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-ljhhw" Nov 04 02:42:22.086484 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:42:22.086451 4405 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/echo-simulator-kserve-64b6f8f96c-2lg6v"] Nov 04 02:42:22.089621 ip-10-0-134-125 kubenswrapper[4405]: W1104 02:42:22.089595 4405 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfe1fd7ca_7aad_4761_abe0_78bbd438ffad.slice/crio-2a904fd68c5e6524b80fccfc7484cc2ffffdb5f5b69fd1c4e57a9f9d9c970a7f WatchSource:0}: Error finding container 2a904fd68c5e6524b80fccfc7484cc2ffffdb5f5b69fd1c4e57a9f9d9c970a7f: Status 404 returned error can't find the container with id 2a904fd68c5e6524b80fccfc7484cc2ffffdb5f5b69fd1c4e57a9f9d9c970a7f Nov 04 02:42:22.937831 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:42:22.937797 4405 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/echo-simulator-kserve-64b6f8f96c-2lg6v" event={"ID":"fe1fd7ca-7aad-4761-abe0-78bbd438ffad","Type":"ContainerStarted","Data":"2a904fd68c5e6524b80fccfc7484cc2ffffdb5f5b69fd1c4e57a9f9d9c970a7f"} Nov 04 02:42:24.948379 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:42:24.948340 4405 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/echo-simulator-kserve-64b6f8f96c-2lg6v" event={"ID":"fe1fd7ca-7aad-4761-abe0-78bbd438ffad","Type":"ContainerStarted","Data":"1c1fec2cdbe6579bd0c2a6fc25787abbc9c092f29b38d1b949ac9aa55cbc0aa3"} Nov 04 02:42:25.065022 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:42:25.064959 4405 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/echo-simulator-kserve-64b6f8f96c-2lg6v" podStartSLOduration=1.9231504 podStartE2EDuration="4.064941376s" totalImagesPullingTime="2.141790976s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 02:42:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 02:42:25.064507142 +0000 UTC m=+591.495207263" watchObservedRunningTime="2026-11-04 02:42:25.064941376 +0000 UTC m=+591.495641499" Nov 04 02:42:25.716721 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:42:25.716663 4405 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/echo-simulator-kserve-64b6f8f96c-2lg6v" Nov 04 02:42:27.816208 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:42:27.816170 4405 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-ljhhw" Nov 04 02:42:27.828610 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:42:27.828580 4405 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-ljhhw" Nov 04 02:42:27.972623 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:42:27.972593 4405 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-ljhhw" Nov 04 02:42:29.877331 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:42:29.877294 4405 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-bl5mg"] Nov 04 02:42:31.716727 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:42:31.716691 4405 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/echo-simulator-kserve-64b6f8f96c-2lg6v" Nov 04 02:42:31.729148 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:42:31.729120 4405 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/echo-simulator-kserve-64b6f8f96c-2lg6v" Nov 04 02:42:31.991159 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:42:31.991088 4405 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/echo-simulator-kserve-64b6f8f96c-2lg6v" Nov 04 02:42:33.730109 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:42:33.730079 4405 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-vwcx8_ffbb2d83-5a93-4a5b-bce3-e40afb8f58ea/ovn-acl-logging/0.log" Nov 04 02:42:33.732352 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:42:33.732331 4405 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-vwcx8_ffbb2d83-5a93-4a5b-bce3-e40afb8f58ea/ovn-acl-logging/0.log" Nov 04 02:42:36.882302 ip-10-0-134-125 kubenswrapper[4405]: E1104 02:42:36.882268 4405 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5f2vb" containerName="extractor" Nov 04 02:42:39.491548 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:42:39.491517 4405 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-bl5mg"] Nov 04 02:42:43.095171 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:42:43.095130 4405 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-embedding-simulated-kserve-67bf594cd6-gnjj2"] Nov 04 02:42:43.098769 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:42:43.098747 4405 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-gnjj2" Nov 04 02:42:43.200181 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:42:43.200148 4405 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-embedding-simulated-kserve-self-signed-certs\"" Nov 04 02:42:43.201539 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:42:43.201513 4405 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-embedding-simulated-kserve-67bf594cd6-gnjj2"] Nov 04 02:42:43.210297 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:42:43.210276 4405 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/22d477a6-9ab8-4330-ae26-813caff99cb3-tmp-dir\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-gnjj2\" (UID: \"22d477a6-9ab8-4330-ae26-813caff99cb3\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-gnjj2" Nov 04 02:42:43.210393 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:42:43.210337 4405 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/22d477a6-9ab8-4330-ae26-813caff99cb3-dshm\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-gnjj2\" (UID: \"22d477a6-9ab8-4330-ae26-813caff99cb3\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-gnjj2" Nov 04 02:42:43.210458 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:42:43.210436 4405 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/22d477a6-9ab8-4330-ae26-813caff99cb3-home\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-gnjj2\" (UID: \"22d477a6-9ab8-4330-ae26-813caff99cb3\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-gnjj2" Nov 04 02:42:43.210495 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:42:43.210474 4405 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/22d477a6-9ab8-4330-ae26-813caff99cb3-model-cache\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-gnjj2\" (UID: \"22d477a6-9ab8-4330-ae26-813caff99cb3\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-gnjj2" Nov 04 02:42:43.210541 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:42:43.210491 4405 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-975wm\" (UniqueName: \"kubernetes.io/projected/22d477a6-9ab8-4330-ae26-813caff99cb3-kube-api-access-975wm\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-gnjj2\" (UID: \"22d477a6-9ab8-4330-ae26-813caff99cb3\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-gnjj2" Nov 04 02:42:43.210652 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:42:43.210630 4405 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/22d477a6-9ab8-4330-ae26-813caff99cb3-tls-certs\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-gnjj2\" (UID: \"22d477a6-9ab8-4330-ae26-813caff99cb3\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-gnjj2" Nov 04 02:42:43.311410 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:42:43.311371 4405 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/22d477a6-9ab8-4330-ae26-813caff99cb3-home\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-gnjj2\" (UID: \"22d477a6-9ab8-4330-ae26-813caff99cb3\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-gnjj2" Nov 04 02:42:43.311410 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:42:43.311414 4405 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/22d477a6-9ab8-4330-ae26-813caff99cb3-model-cache\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-gnjj2\" (UID: \"22d477a6-9ab8-4330-ae26-813caff99cb3\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-gnjj2" Nov 04 02:42:43.311638 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:42:43.311432 4405 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-975wm\" (UniqueName: \"kubernetes.io/projected/22d477a6-9ab8-4330-ae26-813caff99cb3-kube-api-access-975wm\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-gnjj2\" (UID: \"22d477a6-9ab8-4330-ae26-813caff99cb3\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-gnjj2" Nov 04 02:42:43.311638 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:42:43.311468 4405 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/22d477a6-9ab8-4330-ae26-813caff99cb3-tls-certs\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-gnjj2\" (UID: \"22d477a6-9ab8-4330-ae26-813caff99cb3\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-gnjj2" Nov 04 02:42:43.311638 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:42:43.311494 4405 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/22d477a6-9ab8-4330-ae26-813caff99cb3-tmp-dir\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-gnjj2\" (UID: \"22d477a6-9ab8-4330-ae26-813caff99cb3\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-gnjj2" Nov 04 02:42:43.311638 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:42:43.311524 4405 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/22d477a6-9ab8-4330-ae26-813caff99cb3-dshm\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-gnjj2\" (UID: \"22d477a6-9ab8-4330-ae26-813caff99cb3\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-gnjj2" Nov 04 02:42:43.311963 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:42:43.311858 4405 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/22d477a6-9ab8-4330-ae26-813caff99cb3-home\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-gnjj2\" (UID: \"22d477a6-9ab8-4330-ae26-813caff99cb3\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-gnjj2" Nov 04 02:42:43.311963 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:42:43.311880 4405 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/22d477a6-9ab8-4330-ae26-813caff99cb3-model-cache\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-gnjj2\" (UID: \"22d477a6-9ab8-4330-ae26-813caff99cb3\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-gnjj2" Nov 04 02:42:43.312061 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:42:43.311988 4405 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/22d477a6-9ab8-4330-ae26-813caff99cb3-tmp-dir\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-gnjj2\" (UID: \"22d477a6-9ab8-4330-ae26-813caff99cb3\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-gnjj2" Nov 04 02:42:43.313959 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:42:43.313938 4405 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/22d477a6-9ab8-4330-ae26-813caff99cb3-dshm\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-gnjj2\" (UID: \"22d477a6-9ab8-4330-ae26-813caff99cb3\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-gnjj2" Nov 04 02:42:43.314199 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:42:43.314181 4405 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/22d477a6-9ab8-4330-ae26-813caff99cb3-tls-certs\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-gnjj2\" (UID: \"22d477a6-9ab8-4330-ae26-813caff99cb3\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-gnjj2" Nov 04 02:42:43.375144 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:42:43.375082 4405 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-975wm\" (UniqueName: \"kubernetes.io/projected/22d477a6-9ab8-4330-ae26-813caff99cb3-kube-api-access-975wm\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-gnjj2\" (UID: \"22d477a6-9ab8-4330-ae26-813caff99cb3\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-gnjj2" Nov 04 02:42:43.411996 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:42:43.411969 4405 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-gnjj2" Nov 04 02:42:43.785470 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:42:43.785445 4405 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-embedding-simulated-kserve-67bf594cd6-gnjj2"] Nov 04 02:42:43.788155 ip-10-0-134-125 kubenswrapper[4405]: W1104 02:42:43.788128 4405 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod22d477a6_9ab8_4330_ae26_813caff99cb3.slice/crio-3935ecf39b899512d9fc83a35344cb4970affebfe024bc21063bd852bb2e4386 WatchSource:0}: Error finding container 3935ecf39b899512d9fc83a35344cb4970affebfe024bc21063bd852bb2e4386: Status 404 returned error can't find the container with id 3935ecf39b899512d9fc83a35344cb4970affebfe024bc21063bd852bb2e4386 Nov 04 02:42:44.030074 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:42:44.030040 4405 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-gnjj2" event={"ID":"22d477a6-9ab8-4330-ae26-813caff99cb3","Type":"ContainerStarted","Data":"3935ecf39b899512d9fc83a35344cb4970affebfe024bc21063bd852bb2e4386"} Nov 04 02:42:45.013211 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:42:45.013180 4405 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-76597d6855-dhvd2"] Nov 04 02:42:45.016577 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:42:45.016557 4405 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-dhvd2" Nov 04 02:42:45.036137 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:42:45.036108 4405 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-gnjj2" event={"ID":"22d477a6-9ab8-4330-ae26-813caff99cb3","Type":"ContainerStarted","Data":"c40ac3d2be2926149f650035f5d749b954d159aff60e478eb8d42e6b3483a6be"} Nov 04 02:42:45.068739 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:42:45.068707 4405 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\"" Nov 04 02:42:45.118993 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:42:45.118922 4405 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-pm6wh\"" Nov 04 02:42:45.121746 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:42:45.121715 4405 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-76597d6855-dhvd2"] Nov 04 02:42:45.126792 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:42:45.126681 4405 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ad17f139-f05a-40fe-8750-b3b5938bfacc-istio-envoy\") pod \"maas-default-gateway-openshift-default-76597d6855-dhvd2\" (UID: \"ad17f139-f05a-40fe-8750-b3b5938bfacc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-dhvd2" Nov 04 02:42:45.126792 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:42:45.126768 4405 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ad17f139-f05a-40fe-8750-b3b5938bfacc-istio-podinfo\") pod \"maas-default-gateway-openshift-default-76597d6855-dhvd2\" (UID: \"ad17f139-f05a-40fe-8750-b3b5938bfacc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-dhvd2" Nov 04 02:42:45.126996 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:42:45.126798 4405 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ad17f139-f05a-40fe-8750-b3b5938bfacc-istio-data\") pod \"maas-default-gateway-openshift-default-76597d6855-dhvd2\" (UID: \"ad17f139-f05a-40fe-8750-b3b5938bfacc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-dhvd2" Nov 04 02:42:45.126996 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:42:45.126847 4405 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ad17f139-f05a-40fe-8750-b3b5938bfacc-istio-token\") pod \"maas-default-gateway-openshift-default-76597d6855-dhvd2\" (UID: \"ad17f139-f05a-40fe-8750-b3b5938bfacc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-dhvd2" Nov 04 02:42:45.127117 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:42:45.127007 4405 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ad17f139-f05a-40fe-8750-b3b5938bfacc-workload-certs\") pod \"maas-default-gateway-openshift-default-76597d6855-dhvd2\" (UID: \"ad17f139-f05a-40fe-8750-b3b5938bfacc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-dhvd2" Nov 04 02:42:45.127117 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:42:45.127079 4405 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ad17f139-f05a-40fe-8750-b3b5938bfacc-workload-socket\") pod \"maas-default-gateway-openshift-default-76597d6855-dhvd2\" (UID: \"ad17f139-f05a-40fe-8750-b3b5938bfacc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-dhvd2" Nov 04 02:42:45.127237 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:42:45.127116 4405 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4x8fr\" (UniqueName: \"kubernetes.io/projected/ad17f139-f05a-40fe-8750-b3b5938bfacc-kube-api-access-4x8fr\") pod \"maas-default-gateway-openshift-default-76597d6855-dhvd2\" (UID: \"ad17f139-f05a-40fe-8750-b3b5938bfacc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-dhvd2" Nov 04 02:42:45.127237 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:42:45.127145 4405 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ad17f139-f05a-40fe-8750-b3b5938bfacc-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-76597d6855-dhvd2\" (UID: \"ad17f139-f05a-40fe-8750-b3b5938bfacc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-dhvd2" Nov 04 02:42:45.127237 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:42:45.127172 4405 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ad17f139-f05a-40fe-8750-b3b5938bfacc-credential-socket\") pod \"maas-default-gateway-openshift-default-76597d6855-dhvd2\" (UID: \"ad17f139-f05a-40fe-8750-b3b5938bfacc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-dhvd2" Nov 04 02:42:45.228217 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:42:45.228173 4405 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ad17f139-f05a-40fe-8750-b3b5938bfacc-istio-podinfo\") pod \"maas-default-gateway-openshift-default-76597d6855-dhvd2\" (UID: \"ad17f139-f05a-40fe-8750-b3b5938bfacc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-dhvd2" Nov 04 02:42:45.228217 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:42:45.228219 4405 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ad17f139-f05a-40fe-8750-b3b5938bfacc-istio-data\") pod \"maas-default-gateway-openshift-default-76597d6855-dhvd2\" (UID: \"ad17f139-f05a-40fe-8750-b3b5938bfacc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-dhvd2" Nov 04 02:42:45.228485 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:42:45.228266 4405 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ad17f139-f05a-40fe-8750-b3b5938bfacc-istio-token\") pod \"maas-default-gateway-openshift-default-76597d6855-dhvd2\" (UID: \"ad17f139-f05a-40fe-8750-b3b5938bfacc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-dhvd2" Nov 04 02:42:45.228485 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:42:45.228294 4405 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ad17f139-f05a-40fe-8750-b3b5938bfacc-workload-certs\") pod \"maas-default-gateway-openshift-default-76597d6855-dhvd2\" (UID: \"ad17f139-f05a-40fe-8750-b3b5938bfacc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-dhvd2" Nov 04 02:42:45.228485 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:42:45.228327 4405 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ad17f139-f05a-40fe-8750-b3b5938bfacc-workload-socket\") pod \"maas-default-gateway-openshift-default-76597d6855-dhvd2\" (UID: \"ad17f139-f05a-40fe-8750-b3b5938bfacc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-dhvd2" Nov 04 02:42:45.228485 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:42:45.228450 4405 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4x8fr\" (UniqueName: \"kubernetes.io/projected/ad17f139-f05a-40fe-8750-b3b5938bfacc-kube-api-access-4x8fr\") pod \"maas-default-gateway-openshift-default-76597d6855-dhvd2\" (UID: \"ad17f139-f05a-40fe-8750-b3b5938bfacc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-dhvd2" Nov 04 02:42:45.228764 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:42:45.228551 4405 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ad17f139-f05a-40fe-8750-b3b5938bfacc-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-76597d6855-dhvd2\" (UID: \"ad17f139-f05a-40fe-8750-b3b5938bfacc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-dhvd2" Nov 04 02:42:45.228764 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:42:45.228708 4405 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ad17f139-f05a-40fe-8750-b3b5938bfacc-istio-data\") pod \"maas-default-gateway-openshift-default-76597d6855-dhvd2\" (UID: \"ad17f139-f05a-40fe-8750-b3b5938bfacc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-dhvd2" Nov 04 02:42:45.228883 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:42:45.228812 4405 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ad17f139-f05a-40fe-8750-b3b5938bfacc-workload-socket\") pod \"maas-default-gateway-openshift-default-76597d6855-dhvd2\" (UID: \"ad17f139-f05a-40fe-8750-b3b5938bfacc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-dhvd2" Nov 04 02:42:45.228954 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:42:45.228933 4405 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ad17f139-f05a-40fe-8750-b3b5938bfacc-credential-socket\") pod \"maas-default-gateway-openshift-default-76597d6855-dhvd2\" (UID: \"ad17f139-f05a-40fe-8750-b3b5938bfacc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-dhvd2" Nov 04 02:42:45.229020 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:42:45.229005 4405 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ad17f139-f05a-40fe-8750-b3b5938bfacc-workload-certs\") pod \"maas-default-gateway-openshift-default-76597d6855-dhvd2\" (UID: \"ad17f139-f05a-40fe-8750-b3b5938bfacc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-dhvd2" Nov 04 02:42:45.229074 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:42:45.229039 4405 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ad17f139-f05a-40fe-8750-b3b5938bfacc-istio-envoy\") pod \"maas-default-gateway-openshift-default-76597d6855-dhvd2\" (UID: \"ad17f139-f05a-40fe-8750-b3b5938bfacc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-dhvd2" Nov 04 02:42:45.229378 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:42:45.229356 4405 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ad17f139-f05a-40fe-8750-b3b5938bfacc-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-76597d6855-dhvd2\" (UID: \"ad17f139-f05a-40fe-8750-b3b5938bfacc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-dhvd2" Nov 04 02:42:45.229470 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:42:45.229451 4405 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ad17f139-f05a-40fe-8750-b3b5938bfacc-credential-socket\") pod \"maas-default-gateway-openshift-default-76597d6855-dhvd2\" (UID: \"ad17f139-f05a-40fe-8750-b3b5938bfacc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-dhvd2" Nov 04 02:42:45.230955 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:42:45.230933 4405 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ad17f139-f05a-40fe-8750-b3b5938bfacc-istio-podinfo\") pod \"maas-default-gateway-openshift-default-76597d6855-dhvd2\" (UID: \"ad17f139-f05a-40fe-8750-b3b5938bfacc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-dhvd2" Nov 04 02:42:45.231388 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:42:45.231369 4405 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ad17f139-f05a-40fe-8750-b3b5938bfacc-istio-envoy\") pod \"maas-default-gateway-openshift-default-76597d6855-dhvd2\" (UID: \"ad17f139-f05a-40fe-8750-b3b5938bfacc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-dhvd2" Nov 04 02:42:45.251573 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:42:45.251511 4405 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-gnjj2" podStartSLOduration=1.656786566 podStartE2EDuration="2.251498309s" totalImagesPullingTime="594.711743ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 02:42:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 02:42:45.250218575 +0000 UTC m=+611.680918698" watchObservedRunningTime="2026-11-04 02:42:45.251498309 +0000 UTC m=+611.682198447" Nov 04 02:42:45.289266 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:42:45.289198 4405 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ad17f139-f05a-40fe-8750-b3b5938bfacc-istio-token\") pod \"maas-default-gateway-openshift-default-76597d6855-dhvd2\" (UID: \"ad17f139-f05a-40fe-8750-b3b5938bfacc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-dhvd2" Nov 04 02:42:45.289266 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:42:45.289241 4405 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4x8fr\" (UniqueName: \"kubernetes.io/projected/ad17f139-f05a-40fe-8750-b3b5938bfacc-kube-api-access-4x8fr\") pod \"maas-default-gateway-openshift-default-76597d6855-dhvd2\" (UID: \"ad17f139-f05a-40fe-8750-b3b5938bfacc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-dhvd2" Nov 04 02:42:45.329151 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:42:45.329122 4405 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-dhvd2" Nov 04 02:42:45.413062 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:42:45.413031 4405 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-gnjj2" Nov 04 02:42:45.760860 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:42:45.760827 4405 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-76597d6855-dhvd2"] Nov 04 02:42:45.764342 ip-10-0-134-125 kubenswrapper[4405]: W1104 02:42:45.764311 4405 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podad17f139_f05a_40fe_8750_b3b5938bfacc.slice/crio-a21bf6cf4f256b1c4872b3602aae804f7ebcb2989144a76b188281d769772d31 WatchSource:0}: Error finding container a21bf6cf4f256b1c4872b3602aae804f7ebcb2989144a76b188281d769772d31: Status 404 returned error can't find the container with id a21bf6cf4f256b1c4872b3602aae804f7ebcb2989144a76b188281d769772d31 Nov 04 02:42:46.040915 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:42:46.040821 4405 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-dhvd2" event={"ID":"ad17f139-f05a-40fe-8750-b3b5938bfacc","Type":"ContainerStarted","Data":"a21bf6cf4f256b1c4872b3602aae804f7ebcb2989144a76b188281d769772d31"} Nov 04 02:42:46.476032 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:42:46.475996 4405 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-69b588897d-fg8cm"] Nov 04 02:42:46.481083 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:42:46.481059 4405 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-fg8cm" Nov 04 02:42:46.539869 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:42:46.539835 4405 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-trlp-test-simulated-kserve-self-signed-certs\"" Nov 04 02:42:46.590865 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:42:46.590833 4405 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-69b588897d-fg8cm"] Nov 04 02:42:46.640854 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:42:46.640819 4405 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f490d2f8-2111-42e4-b438-a1180fcacb45-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-fg8cm\" (UID: \"f490d2f8-2111-42e4-b438-a1180fcacb45\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-fg8cm" Nov 04 02:42:46.641017 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:42:46.640858 4405 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xlh2c\" (UniqueName: \"kubernetes.io/projected/f490d2f8-2111-42e4-b438-a1180fcacb45-kube-api-access-xlh2c\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-fg8cm\" (UID: \"f490d2f8-2111-42e4-b438-a1180fcacb45\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-fg8cm" Nov 04 02:42:46.641017 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:42:46.640964 4405 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f490d2f8-2111-42e4-b438-a1180fcacb45-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-fg8cm\" (UID: \"f490d2f8-2111-42e4-b438-a1180fcacb45\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-fg8cm" Nov 04 02:42:46.641017 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:42:46.640996 4405 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f490d2f8-2111-42e4-b438-a1180fcacb45-home\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-fg8cm\" (UID: \"f490d2f8-2111-42e4-b438-a1180fcacb45\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-fg8cm" Nov 04 02:42:46.641114 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:42:46.641015 4405 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f490d2f8-2111-42e4-b438-a1180fcacb45-dshm\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-fg8cm\" (UID: \"f490d2f8-2111-42e4-b438-a1180fcacb45\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-fg8cm" Nov 04 02:42:46.641152 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:42:46.641129 4405 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f490d2f8-2111-42e4-b438-a1180fcacb45-model-cache\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-fg8cm\" (UID: \"f490d2f8-2111-42e4-b438-a1180fcacb45\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-fg8cm" Nov 04 02:42:46.741629 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:42:46.741550 4405 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f490d2f8-2111-42e4-b438-a1180fcacb45-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-fg8cm\" (UID: \"f490d2f8-2111-42e4-b438-a1180fcacb45\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-fg8cm" Nov 04 02:42:46.741629 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:42:46.741587 4405 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f490d2f8-2111-42e4-b438-a1180fcacb45-home\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-fg8cm\" (UID: \"f490d2f8-2111-42e4-b438-a1180fcacb45\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-fg8cm" Nov 04 02:42:46.741629 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:42:46.741608 4405 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f490d2f8-2111-42e4-b438-a1180fcacb45-dshm\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-fg8cm\" (UID: \"f490d2f8-2111-42e4-b438-a1180fcacb45\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-fg8cm" Nov 04 02:42:46.741868 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:42:46.741698 4405 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f490d2f8-2111-42e4-b438-a1180fcacb45-model-cache\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-fg8cm\" (UID: \"f490d2f8-2111-42e4-b438-a1180fcacb45\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-fg8cm" Nov 04 02:42:46.741868 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:42:46.741847 4405 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f490d2f8-2111-42e4-b438-a1180fcacb45-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-fg8cm\" (UID: \"f490d2f8-2111-42e4-b438-a1180fcacb45\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-fg8cm" Nov 04 02:42:46.741959 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:42:46.741901 4405 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xlh2c\" (UniqueName: \"kubernetes.io/projected/f490d2f8-2111-42e4-b438-a1180fcacb45-kube-api-access-xlh2c\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-fg8cm\" (UID: \"f490d2f8-2111-42e4-b438-a1180fcacb45\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-fg8cm" Nov 04 02:42:46.742140 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:42:46.742115 4405 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f490d2f8-2111-42e4-b438-a1180fcacb45-home\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-fg8cm\" (UID: \"f490d2f8-2111-42e4-b438-a1180fcacb45\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-fg8cm" Nov 04 02:42:46.742140 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:42:46.742134 4405 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f490d2f8-2111-42e4-b438-a1180fcacb45-model-cache\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-fg8cm\" (UID: \"f490d2f8-2111-42e4-b438-a1180fcacb45\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-fg8cm" Nov 04 02:42:46.742335 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:42:46.742252 4405 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f490d2f8-2111-42e4-b438-a1180fcacb45-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-fg8cm\" (UID: \"f490d2f8-2111-42e4-b438-a1180fcacb45\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-fg8cm" Nov 04 02:42:46.744004 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:42:46.743983 4405 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f490d2f8-2111-42e4-b438-a1180fcacb45-dshm\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-fg8cm\" (UID: \"f490d2f8-2111-42e4-b438-a1180fcacb45\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-fg8cm" Nov 04 02:42:46.744213 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:42:46.744195 4405 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f490d2f8-2111-42e4-b438-a1180fcacb45-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-fg8cm\" (UID: \"f490d2f8-2111-42e4-b438-a1180fcacb45\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-fg8cm" Nov 04 02:42:46.806110 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:42:46.806073 4405 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xlh2c\" (UniqueName: \"kubernetes.io/projected/f490d2f8-2111-42e4-b438-a1180fcacb45-kube-api-access-xlh2c\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-fg8cm\" (UID: \"f490d2f8-2111-42e4-b438-a1180fcacb45\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-fg8cm" Nov 04 02:42:47.094443 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:42:47.094410 4405 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-fg8cm" Nov 04 02:42:47.469797 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:42:47.469734 4405 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-69b588897d-fg8cm"] Nov 04 02:42:47.472035 ip-10-0-134-125 kubenswrapper[4405]: W1104 02:42:47.471999 4405 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf490d2f8_2111_42e4_b438_a1180fcacb45.slice/crio-6f553f962d35fe8586166693fa7867037d05adc07adae03fcea20f6146e45a74 WatchSource:0}: Error finding container 6f553f962d35fe8586166693fa7867037d05adc07adae03fcea20f6146e45a74: Status 404 returned error can't find the container with id 6f553f962d35fe8586166693fa7867037d05adc07adae03fcea20f6146e45a74 Nov 04 02:42:48.052636 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:42:48.052602 4405 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-fg8cm" event={"ID":"f490d2f8-2111-42e4-b438-a1180fcacb45","Type":"ContainerStarted","Data":"6f553f962d35fe8586166693fa7867037d05adc07adae03fcea20f6146e45a74"} Nov 04 02:42:49.059787 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:42:49.059751 4405 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-fg8cm" event={"ID":"f490d2f8-2111-42e4-b438-a1180fcacb45","Type":"ContainerStarted","Data":"59bfb04bc8db7f6c7e52b47235ffd201ebe3c12a4ecc1b18b22619110b777bbd"} Nov 04 02:42:49.094737 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:42:49.094702 4405 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-fg8cm" Nov 04 02:42:49.178885 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:42:49.178826 4405 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-fg8cm" podStartSLOduration=2.537345462 podStartE2EDuration="3.178807474s" totalImagesPullingTime="641.462012ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 02:42:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 02:42:49.177486865 +0000 UTC m=+615.608186985" watchObservedRunningTime="2026-11-04 02:42:49.178807474 +0000 UTC m=+615.609507595" Nov 04 02:42:49.982222 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:42:49.982188 4405 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 04 02:42:49.982325 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:42:49.982253 4405 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 04 02:42:49.982325 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:42:49.982281 4405 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 04 02:42:50.066776 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:42:50.066712 4405 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-dhvd2" event={"ID":"ad17f139-f05a-40fe-8750-b3b5938bfacc","Type":"ContainerStarted","Data":"abab077e0b3bacb6976169fa82208abeba6c11efc953d43029100ba49ba15ee4"} Nov 04 02:42:50.223160 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:42:50.223110 4405 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-dhvd2" podStartSLOduration=2.008298419 podStartE2EDuration="6.22309584s" totalImagesPullingTime="4.214797421s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 02:42:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 02:42:50.222800676 +0000 UTC m=+616.653500799" watchObservedRunningTime="2026-11-04 02:42:50.22309584 +0000 UTC m=+616.653795961" Nov 04 02:42:50.329850 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:42:50.329774 4405 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-dhvd2" Nov 04 02:42:51.331898 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:42:51.331860 4405 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76597d6855-dhvd2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.29:15021/healthz/ready\": dial tcp 10.134.0.29:15021: connect: connection refused" start-of-body= Nov 04 02:42:51.332382 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:42:51.331922 4405 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-dhvd2" podUID="ad17f139-f05a-40fe-8750-b3b5938bfacc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.29:15021/healthz/ready\": dial tcp 10.134.0.29:15021: connect: connection refused" Nov 04 02:42:52.330107 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:42:52.330062 4405 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76597d6855-dhvd2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.29:15021/healthz/ready\": dial tcp 10.134.0.29:15021: connect: connection refused" start-of-body= Nov 04 02:42:52.330289 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:42:52.330133 4405 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-dhvd2" podUID="ad17f139-f05a-40fe-8750-b3b5938bfacc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.29:15021/healthz/ready\": dial tcp 10.134.0.29:15021: connect: connection refused" Nov 04 02:42:53.330250 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:42:53.330212 4405 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76597d6855-dhvd2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.29:15021/healthz/ready\": dial tcp 10.134.0.29:15021: connect: connection refused" start-of-body= Nov 04 02:42:53.330660 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:42:53.330274 4405 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-dhvd2" podUID="ad17f139-f05a-40fe-8750-b3b5938bfacc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.29:15021/healthz/ready\": dial tcp 10.134.0.29:15021: connect: connection refused" Nov 04 02:42:53.417874 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:42:53.416832 4405 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-gnjj2" Nov 04 02:42:53.431102 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:42:53.431070 4405 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-gnjj2" Nov 04 02:42:53.653277 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:42:53.653243 4405 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-bl5mg"] Nov 04 02:42:54.117320 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:42:54.117006 4405 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-gnjj2" Nov 04 02:42:54.329799 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:42:54.329768 4405 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76597d6855-dhvd2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.29:15021/healthz/ready\": dial tcp 10.134.0.29:15021: connect: connection refused" start-of-body= Nov 04 02:42:54.329959 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:42:54.329836 4405 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-dhvd2" podUID="ad17f139-f05a-40fe-8750-b3b5938bfacc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.29:15021/healthz/ready\": dial tcp 10.134.0.29:15021: connect: connection refused" Nov 04 02:42:55.330296 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:42:55.330255 4405 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76597d6855-dhvd2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.29:15021/healthz/ready\": dial tcp 10.134.0.29:15021: connect: connection refused" start-of-body= Nov 04 02:42:55.330746 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:42:55.330323 4405 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-dhvd2" podUID="ad17f139-f05a-40fe-8750-b3b5938bfacc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.29:15021/healthz/ready\": dial tcp 10.134.0.29:15021: connect: connection refused" Nov 04 02:42:56.329578 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:42:56.329542 4405 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76597d6855-dhvd2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.29:15021/healthz/ready\": dial tcp 10.134.0.29:15021: connect: connection refused" start-of-body= Nov 04 02:42:56.329825 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:42:56.329598 4405 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-dhvd2" podUID="ad17f139-f05a-40fe-8750-b3b5938bfacc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.29:15021/healthz/ready\": dial tcp 10.134.0.29:15021: connect: connection refused" Nov 04 02:42:57.095475 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:42:57.095431 4405 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-fg8cm" Nov 04 02:42:57.108575 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:42:57.108549 4405 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-fg8cm" Nov 04 02:42:57.330385 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:42:57.330345 4405 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76597d6855-dhvd2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.29:15021/healthz/ready\": dial tcp 10.134.0.29:15021: connect: connection refused" start-of-body= Nov 04 02:42:57.330602 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:42:57.330414 4405 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-dhvd2" podUID="ad17f139-f05a-40fe-8750-b3b5938bfacc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.29:15021/healthz/ready\": dial tcp 10.134.0.29:15021: connect: connection refused" Nov 04 02:42:58.108850 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:42:58.108817 4405 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-fg8cm" Nov 04 02:42:59.370871 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:42:59.370825 4405 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76597d6855-dhvd2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.29:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 04 02:42:59.371361 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:42:59.370882 4405 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-dhvd2" podUID="ad17f139-f05a-40fe-8750-b3b5938bfacc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.29:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 04 02:43:00.154314 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:43:00.154268 4405 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-76597d6855-tlkdl"] Nov 04 02:43:00.162511 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:43:00.162480 4405 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-tlkdl" Nov 04 02:43:00.191323 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:43:00.191284 4405 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/45bb7186-84f1-4586-b4c1-50250365ff2b-istio-podinfo\") pod \"maas-default-gateway-openshift-default-76597d6855-tlkdl\" (UID: \"45bb7186-84f1-4586-b4c1-50250365ff2b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-tlkdl" Nov 04 02:43:00.191495 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:43:00.191355 4405 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/45bb7186-84f1-4586-b4c1-50250365ff2b-istio-token\") pod \"maas-default-gateway-openshift-default-76597d6855-tlkdl\" (UID: \"45bb7186-84f1-4586-b4c1-50250365ff2b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-tlkdl" Nov 04 02:43:00.191495 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:43:00.191444 4405 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/45bb7186-84f1-4586-b4c1-50250365ff2b-credential-socket\") pod \"maas-default-gateway-openshift-default-76597d6855-tlkdl\" (UID: \"45bb7186-84f1-4586-b4c1-50250365ff2b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-tlkdl" Nov 04 02:43:00.191495 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:43:00.191474 4405 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/45bb7186-84f1-4586-b4c1-50250365ff2b-istio-data\") pod \"maas-default-gateway-openshift-default-76597d6855-tlkdl\" (UID: \"45bb7186-84f1-4586-b4c1-50250365ff2b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-tlkdl" Nov 04 02:43:00.191627 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:43:00.191504 4405 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/45bb7186-84f1-4586-b4c1-50250365ff2b-workload-certs\") pod \"maas-default-gateway-openshift-default-76597d6855-tlkdl\" (UID: \"45bb7186-84f1-4586-b4c1-50250365ff2b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-tlkdl" Nov 04 02:43:00.191627 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:43:00.191584 4405 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/45bb7186-84f1-4586-b4c1-50250365ff2b-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-76597d6855-tlkdl\" (UID: \"45bb7186-84f1-4586-b4c1-50250365ff2b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-tlkdl" Nov 04 02:43:00.191627 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:43:00.191614 4405 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/45bb7186-84f1-4586-b4c1-50250365ff2b-workload-socket\") pod \"maas-default-gateway-openshift-default-76597d6855-tlkdl\" (UID: \"45bb7186-84f1-4586-b4c1-50250365ff2b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-tlkdl" Nov 04 02:43:00.191799 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:43:00.191635 4405 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/45bb7186-84f1-4586-b4c1-50250365ff2b-istio-envoy\") pod \"maas-default-gateway-openshift-default-76597d6855-tlkdl\" (UID: \"45bb7186-84f1-4586-b4c1-50250365ff2b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-tlkdl" Nov 04 02:43:00.191799 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:43:00.191692 4405 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2lllb\" (UniqueName: \"kubernetes.io/projected/45bb7186-84f1-4586-b4c1-50250365ff2b-kube-api-access-2lllb\") pod \"maas-default-gateway-openshift-default-76597d6855-tlkdl\" (UID: \"45bb7186-84f1-4586-b4c1-50250365ff2b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-tlkdl" Nov 04 02:43:00.253515 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:43:00.253475 4405 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-76597d6855-tlkdl"] Nov 04 02:43:00.292545 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:43:00.292502 4405 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/45bb7186-84f1-4586-b4c1-50250365ff2b-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-76597d6855-tlkdl\" (UID: \"45bb7186-84f1-4586-b4c1-50250365ff2b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-tlkdl" Nov 04 02:43:00.292778 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:43:00.292553 4405 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/45bb7186-84f1-4586-b4c1-50250365ff2b-workload-socket\") pod \"maas-default-gateway-openshift-default-76597d6855-tlkdl\" (UID: \"45bb7186-84f1-4586-b4c1-50250365ff2b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-tlkdl" Nov 04 02:43:00.292778 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:43:00.292584 4405 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/45bb7186-84f1-4586-b4c1-50250365ff2b-istio-envoy\") pod \"maas-default-gateway-openshift-default-76597d6855-tlkdl\" (UID: \"45bb7186-84f1-4586-b4c1-50250365ff2b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-tlkdl" Nov 04 02:43:00.292778 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:43:00.292624 4405 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2lllb\" (UniqueName: \"kubernetes.io/projected/45bb7186-84f1-4586-b4c1-50250365ff2b-kube-api-access-2lllb\") pod \"maas-default-gateway-openshift-default-76597d6855-tlkdl\" (UID: \"45bb7186-84f1-4586-b4c1-50250365ff2b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-tlkdl" Nov 04 02:43:00.292778 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:43:00.292689 4405 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/45bb7186-84f1-4586-b4c1-50250365ff2b-istio-podinfo\") pod \"maas-default-gateway-openshift-default-76597d6855-tlkdl\" (UID: \"45bb7186-84f1-4586-b4c1-50250365ff2b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-tlkdl" Nov 04 02:43:00.292778 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:43:00.292743 4405 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/45bb7186-84f1-4586-b4c1-50250365ff2b-istio-token\") pod \"maas-default-gateway-openshift-default-76597d6855-tlkdl\" (UID: \"45bb7186-84f1-4586-b4c1-50250365ff2b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-tlkdl" Nov 04 02:43:00.292778 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:43:00.292771 4405 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/45bb7186-84f1-4586-b4c1-50250365ff2b-credential-socket\") pod \"maas-default-gateway-openshift-default-76597d6855-tlkdl\" (UID: \"45bb7186-84f1-4586-b4c1-50250365ff2b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-tlkdl" Nov 04 02:43:00.293100 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:43:00.292796 4405 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/45bb7186-84f1-4586-b4c1-50250365ff2b-istio-data\") pod \"maas-default-gateway-openshift-default-76597d6855-tlkdl\" (UID: \"45bb7186-84f1-4586-b4c1-50250365ff2b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-tlkdl" Nov 04 02:43:00.293100 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:43:00.292838 4405 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/45bb7186-84f1-4586-b4c1-50250365ff2b-workload-certs\") pod \"maas-default-gateway-openshift-default-76597d6855-tlkdl\" (UID: \"45bb7186-84f1-4586-b4c1-50250365ff2b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-tlkdl" Nov 04 02:43:00.293299 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:43:00.293274 4405 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/45bb7186-84f1-4586-b4c1-50250365ff2b-workload-certs\") pod \"maas-default-gateway-openshift-default-76597d6855-tlkdl\" (UID: \"45bb7186-84f1-4586-b4c1-50250365ff2b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-tlkdl" Nov 04 02:43:00.294097 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:43:00.294070 4405 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/45bb7186-84f1-4586-b4c1-50250365ff2b-workload-socket\") pod \"maas-default-gateway-openshift-default-76597d6855-tlkdl\" (UID: \"45bb7186-84f1-4586-b4c1-50250365ff2b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-tlkdl" Nov 04 02:43:00.294236 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:43:00.294147 4405 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/45bb7186-84f1-4586-b4c1-50250365ff2b-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-76597d6855-tlkdl\" (UID: \"45bb7186-84f1-4586-b4c1-50250365ff2b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-tlkdl" Nov 04 02:43:00.294588 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:43:00.294568 4405 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/45bb7186-84f1-4586-b4c1-50250365ff2b-credential-socket\") pod \"maas-default-gateway-openshift-default-76597d6855-tlkdl\" (UID: \"45bb7186-84f1-4586-b4c1-50250365ff2b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-tlkdl" Nov 04 02:43:00.295576 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:43:00.295551 4405 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/45bb7186-84f1-4586-b4c1-50250365ff2b-istio-data\") pod \"maas-default-gateway-openshift-default-76597d6855-tlkdl\" (UID: \"45bb7186-84f1-4586-b4c1-50250365ff2b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-tlkdl" Nov 04 02:43:00.302322 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:43:00.299202 4405 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/45bb7186-84f1-4586-b4c1-50250365ff2b-istio-podinfo\") pod \"maas-default-gateway-openshift-default-76597d6855-tlkdl\" (UID: \"45bb7186-84f1-4586-b4c1-50250365ff2b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-tlkdl" Nov 04 02:43:00.302322 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:43:00.299440 4405 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/45bb7186-84f1-4586-b4c1-50250365ff2b-istio-envoy\") pod \"maas-default-gateway-openshift-default-76597d6855-tlkdl\" (UID: \"45bb7186-84f1-4586-b4c1-50250365ff2b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-tlkdl" Nov 04 02:43:00.330056 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:43:00.330014 4405 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-dhvd2" Nov 04 02:43:00.355594 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:43:00.355510 4405 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2lllb\" (UniqueName: \"kubernetes.io/projected/45bb7186-84f1-4586-b4c1-50250365ff2b-kube-api-access-2lllb\") pod \"maas-default-gateway-openshift-default-76597d6855-tlkdl\" (UID: \"45bb7186-84f1-4586-b4c1-50250365ff2b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-tlkdl" Nov 04 02:43:00.356249 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:43:00.356222 4405 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/45bb7186-84f1-4586-b4c1-50250365ff2b-istio-token\") pod \"maas-default-gateway-openshift-default-76597d6855-tlkdl\" (UID: \"45bb7186-84f1-4586-b4c1-50250365ff2b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-tlkdl" Nov 04 02:43:00.411942 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:43:00.411840 4405 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76597d6855-dhvd2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.29:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 04 02:43:00.411942 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:43:00.411908 4405 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-dhvd2" podUID="ad17f139-f05a-40fe-8750-b3b5938bfacc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.29:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 04 02:43:00.479609 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:43:00.479575 4405 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-tlkdl" Nov 04 02:43:00.634227 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:43:00.634197 4405 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-dhvd2" Nov 04 02:43:00.933159 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:43:00.933132 4405 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-76597d6855-tlkdl"] Nov 04 02:43:00.935326 ip-10-0-134-125 kubenswrapper[4405]: W1104 02:43:00.935298 4405 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod45bb7186_84f1_4586_b4c1_50250365ff2b.slice/crio-8cfe87dfef8cfc9190a002e22ce9bbf669809e1322d60af781ae027c5b1d3bb3 WatchSource:0}: Error finding container 8cfe87dfef8cfc9190a002e22ce9bbf669809e1322d60af781ae027c5b1d3bb3: Status 404 returned error can't find the container with id 8cfe87dfef8cfc9190a002e22ce9bbf669809e1322d60af781ae027c5b1d3bb3 Nov 04 02:43:00.938705 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:43:00.938649 4405 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 04 02:43:00.938812 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:43:00.938774 4405 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 04 02:43:00.938883 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:43:00.938832 4405 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 04 02:43:01.132558 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:43:01.132519 4405 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-tlkdl" event={"ID":"45bb7186-84f1-4586-b4c1-50250365ff2b","Type":"ContainerStarted","Data":"dd73bfbe496cb2981cf8933215430460d8e7d58d9eda1d0d88dc5bdfcb4ceb2e"} Nov 04 02:43:01.132760 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:43:01.132586 4405 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-tlkdl" event={"ID":"45bb7186-84f1-4586-b4c1-50250365ff2b","Type":"ContainerStarted","Data":"8cfe87dfef8cfc9190a002e22ce9bbf669809e1322d60af781ae027c5b1d3bb3"} Nov 04 02:43:01.133612 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:43:01.133591 4405 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-dhvd2" Nov 04 02:43:01.254308 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:43:01.254203 4405 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-tlkdl" podStartSLOduration=1.25418623 podStartE2EDuration="1.25418623s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 02:43:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 02:43:01.25254791 +0000 UTC m=+627.683248021" watchObservedRunningTime="2026-11-04 02:43:01.25418623 +0000 UTC m=+627.684886414" Nov 04 02:43:01.480083 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:43:01.480042 4405 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-tlkdl" Nov 04 02:43:02.482310 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:43:02.482272 4405 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76597d6855-tlkdl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.31:15021/healthz/ready\": dial tcp 10.134.0.31:15021: connect: connection refused" start-of-body= Nov 04 02:43:02.482754 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:43:02.482332 4405 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-tlkdl" podUID="45bb7186-84f1-4586-b4c1-50250365ff2b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.31:15021/healthz/ready\": dial tcp 10.134.0.31:15021: connect: connection refused" Nov 04 02:43:03.480957 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:43:03.480916 4405 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76597d6855-tlkdl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.31:15021/healthz/ready\": dial tcp 10.134.0.31:15021: connect: connection refused" start-of-body= Nov 04 02:43:03.481170 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:43:03.480982 4405 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-tlkdl" podUID="45bb7186-84f1-4586-b4c1-50250365ff2b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.31:15021/healthz/ready\": dial tcp 10.134.0.31:15021: connect: connection refused" Nov 04 02:43:04.480308 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:43:04.480271 4405 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76597d6855-tlkdl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.31:15021/healthz/ready\": dial tcp 10.134.0.31:15021: connect: connection refused" start-of-body= Nov 04 02:43:04.480727 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:43:04.480327 4405 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-tlkdl" podUID="45bb7186-84f1-4586-b4c1-50250365ff2b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.31:15021/healthz/ready\": dial tcp 10.134.0.31:15021: connect: connection refused" Nov 04 02:43:05.480413 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:43:05.480373 4405 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76597d6855-tlkdl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.31:15021/healthz/ready\": dial tcp 10.134.0.31:15021: connect: connection refused" start-of-body= Nov 04 02:43:05.480863 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:43:05.480443 4405 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-tlkdl" podUID="45bb7186-84f1-4586-b4c1-50250365ff2b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.31:15021/healthz/ready\": dial tcp 10.134.0.31:15021: connect: connection refused" Nov 04 02:43:06.480493 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:43:06.480461 4405 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76597d6855-tlkdl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.31:15021/healthz/ready\": dial tcp 10.134.0.31:15021: connect: connection refused" start-of-body= Nov 04 02:43:06.480963 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:43:06.480525 4405 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-tlkdl" podUID="45bb7186-84f1-4586-b4c1-50250365ff2b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.31:15021/healthz/ready\": dial tcp 10.134.0.31:15021: connect: connection refused" Nov 04 02:43:06.883853 ip-10-0-134-125 kubenswrapper[4405]: E1104 02:43:06.883816 4405 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5f2vb" containerName="extractor" Nov 04 02:43:07.480778 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:43:07.480743 4405 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76597d6855-tlkdl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.31:15021/healthz/ready\": dial tcp 10.134.0.31:15021: connect: connection refused" start-of-body= Nov 04 02:43:07.481171 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:43:07.480803 4405 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-tlkdl" podUID="45bb7186-84f1-4586-b4c1-50250365ff2b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.31:15021/healthz/ready\": dial tcp 10.134.0.31:15021: connect: connection refused" Nov 04 02:43:08.480874 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:43:08.480834 4405 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76597d6855-tlkdl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.31:15021/healthz/ready\": dial tcp 10.134.0.31:15021: connect: connection refused" start-of-body= Nov 04 02:43:08.481282 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:43:08.480904 4405 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-tlkdl" podUID="45bb7186-84f1-4586-b4c1-50250365ff2b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.31:15021/healthz/ready\": dial tcp 10.134.0.31:15021: connect: connection refused" Nov 04 02:43:09.611282 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:43:09.611248 4405 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-tlkdl" Nov 04 02:43:09.611769 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:43:09.611559 4405 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-tlkdl" Nov 04 02:43:09.612501 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:43:09.612473 4405 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-tlkdl" Nov 04 02:43:18.537634 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:43:18.537595 4405 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/ai-gateway-controller-69464886d7-2vmms"] Nov 04 02:43:18.541279 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:43:18.541260 4405 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/ai-gateway-controller-69464886d7-2vmms" Nov 04 02:43:18.596110 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:43:18.596072 4405 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"ai-gateway-controller-parameters\"" Nov 04 02:43:18.645679 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:43:18.645633 4405 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"ai-gateway-controller-dockercfg-9d55p\"" Nov 04 02:43:18.645679 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:43:18.645642 4405 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"ai-gateway-controller-metrics-tls\"" Nov 04 02:43:18.648085 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:43:18.648064 4405 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/ai-gateway-controller-69464886d7-2vmms"] Nov 04 02:43:18.683736 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:43:18.683702 4405 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ngr6x\" (UniqueName: \"kubernetes.io/projected/8c1cdb1d-4616-4821-923b-0b791f4e93d1-kube-api-access-ngr6x\") pod \"ai-gateway-controller-69464886d7-2vmms\" (UID: \"8c1cdb1d-4616-4821-923b-0b791f4e93d1\") " pod="opendatahub/ai-gateway-controller-69464886d7-2vmms" Nov 04 02:43:18.683736 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:43:18.683737 4405 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8c1cdb1d-4616-4821-923b-0b791f4e93d1-metrics-certs\") pod \"ai-gateway-controller-69464886d7-2vmms\" (UID: \"8c1cdb1d-4616-4821-923b-0b791f4e93d1\") " pod="opendatahub/ai-gateway-controller-69464886d7-2vmms" Nov 04 02:43:18.785080 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:43:18.785042 4405 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ngr6x\" (UniqueName: \"kubernetes.io/projected/8c1cdb1d-4616-4821-923b-0b791f4e93d1-kube-api-access-ngr6x\") pod \"ai-gateway-controller-69464886d7-2vmms\" (UID: \"8c1cdb1d-4616-4821-923b-0b791f4e93d1\") " pod="opendatahub/ai-gateway-controller-69464886d7-2vmms" Nov 04 02:43:18.785080 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:43:18.785080 4405 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8c1cdb1d-4616-4821-923b-0b791f4e93d1-metrics-certs\") pod \"ai-gateway-controller-69464886d7-2vmms\" (UID: \"8c1cdb1d-4616-4821-923b-0b791f4e93d1\") " pod="opendatahub/ai-gateway-controller-69464886d7-2vmms" Nov 04 02:43:18.787716 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:43:18.787640 4405 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8c1cdb1d-4616-4821-923b-0b791f4e93d1-metrics-certs\") pod \"ai-gateway-controller-69464886d7-2vmms\" (UID: \"8c1cdb1d-4616-4821-923b-0b791f4e93d1\") " pod="opendatahub/ai-gateway-controller-69464886d7-2vmms" Nov 04 02:43:18.849648 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:43:18.849620 4405 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ngr6x\" (UniqueName: \"kubernetes.io/projected/8c1cdb1d-4616-4821-923b-0b791f4e93d1-kube-api-access-ngr6x\") pod \"ai-gateway-controller-69464886d7-2vmms\" (UID: \"8c1cdb1d-4616-4821-923b-0b791f4e93d1\") " pod="opendatahub/ai-gateway-controller-69464886d7-2vmms" Nov 04 02:43:18.854511 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:43:18.854491 4405 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/ai-gateway-controller-69464886d7-2vmms" Nov 04 02:43:19.229663 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:43:19.229632 4405 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/ai-gateway-controller-69464886d7-2vmms"] Nov 04 02:43:19.233204 ip-10-0-134-125 kubenswrapper[4405]: W1104 02:43:19.233178 4405 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8c1cdb1d_4616_4821_923b_0b791f4e93d1.slice/crio-6a4bc9a631f0054c2d82a9f061b13cb49746b11fb52e309f09f1995d44e72fce WatchSource:0}: Error finding container 6a4bc9a631f0054c2d82a9f061b13cb49746b11fb52e309f09f1995d44e72fce: Status 404 returned error can't find the container with id 6a4bc9a631f0054c2d82a9f061b13cb49746b11fb52e309f09f1995d44e72fce Nov 04 02:43:19.234779 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:43:19.234756 4405 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 04 02:43:20.219109 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:43:20.219065 4405 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-69464886d7-2vmms" event={"ID":"8c1cdb1d-4616-4821-923b-0b791f4e93d1","Type":"ContainerStarted","Data":"6a4bc9a631f0054c2d82a9f061b13cb49746b11fb52e309f09f1995d44e72fce"} Nov 04 02:43:24.238738 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:43:24.238618 4405 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-69464886d7-2vmms" event={"ID":"8c1cdb1d-4616-4821-923b-0b791f4e93d1","Type":"ContainerStarted","Data":"132cedb7a1ec0f3f8f501c9ba19586c0d408ad32f9d6bc41439e290a81036e05"} Nov 04 02:43:24.238738 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:43:24.238708 4405 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/ai-gateway-controller-69464886d7-2vmms" Nov 04 02:43:24.354906 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:43:24.354844 4405 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/ai-gateway-controller-69464886d7-2vmms" podStartSLOduration=1.6534486080000002 podStartE2EDuration="6.354827028s" totalImagesPullingTime="4.70137842s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 02:43:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 02:43:24.354066942 +0000 UTC m=+650.784767062" watchObservedRunningTime="2026-11-04 02:43:24.354827028 +0000 UTC m=+650.785527150" Nov 04 02:43:35.245285 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:43:35.245255 4405 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/ai-gateway-controller-69464886d7-2vmms" Nov 04 02:43:36.882172 ip-10-0-134-125 kubenswrapper[4405]: E1104 02:43:36.882127 4405 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5f2vb" containerName="extractor" Nov 04 02:43:49.846466 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:43:49.846389 4405 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/ai-gateway-controller-69464886d7-2vmms"] Nov 04 02:43:49.847002 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:43:49.846587 4405 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/ai-gateway-controller-69464886d7-2vmms" podUID="8c1cdb1d-4616-4821-923b-0b791f4e93d1" containerName="manager" containerID="cri-o://132cedb7a1ec0f3f8f501c9ba19586c0d408ad32f9d6bc41439e290a81036e05" gracePeriod=30 Nov 04 02:43:50.098230 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:43:50.098169 4405 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/ai-gateway-controller-69464886d7-2vmms" Nov 04 02:43:50.172771 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:43:50.172736 4405 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8c1cdb1d-4616-4821-923b-0b791f4e93d1-kube-api-access-ngr6x\" (UniqueName: \"kubernetes.io/projected/8c1cdb1d-4616-4821-923b-0b791f4e93d1-kube-api-access-ngr6x\") pod \"8c1cdb1d-4616-4821-923b-0b791f4e93d1\" (UID: \"8c1cdb1d-4616-4821-923b-0b791f4e93d1\") " Nov 04 02:43:50.172953 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:43:50.172822 4405 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8c1cdb1d-4616-4821-923b-0b791f4e93d1-metrics-certs\" (UniqueName: \"kubernetes.io/secret/8c1cdb1d-4616-4821-923b-0b791f4e93d1-metrics-certs\") pod \"8c1cdb1d-4616-4821-923b-0b791f4e93d1\" (UID: \"8c1cdb1d-4616-4821-923b-0b791f4e93d1\") " Nov 04 02:43:50.175184 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:43:50.175139 4405 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8c1cdb1d-4616-4821-923b-0b791f4e93d1-kube-api-access-ngr6x" pod "8c1cdb1d-4616-4821-923b-0b791f4e93d1" (UID: "8c1cdb1d-4616-4821-923b-0b791f4e93d1"). InnerVolumeSpecName "kube-api-access-ngr6x". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 02:43:50.175184 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:43:50.175165 4405 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8c1cdb1d-4616-4821-923b-0b791f4e93d1-metrics-certs" pod "8c1cdb1d-4616-4821-923b-0b791f4e93d1" (UID: "8c1cdb1d-4616-4821-923b-0b791f4e93d1"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 04 02:43:50.273991 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:43:50.273954 4405 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8c1cdb1d-4616-4821-923b-0b791f4e93d1-metrics-certs\") on node \"ip-10-0-134-125.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 02:43:50.273991 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:43:50.273985 4405 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ngr6x\" (UniqueName: \"kubernetes.io/projected/8c1cdb1d-4616-4821-923b-0b791f4e93d1-kube-api-access-ngr6x\") on node \"ip-10-0-134-125.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 02:43:50.348639 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:43:50.348555 4405 generic.go:356] "Generic (PLEG): container finished" podID="8c1cdb1d-4616-4821-923b-0b791f4e93d1" containerID="132cedb7a1ec0f3f8f501c9ba19586c0d408ad32f9d6bc41439e290a81036e05" exitCode=0 Nov 04 02:43:50.348639 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:43:50.348608 4405 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/ai-gateway-controller-69464886d7-2vmms" Nov 04 02:43:50.348858 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:43:50.348637 4405 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-69464886d7-2vmms" event={"ID":"8c1cdb1d-4616-4821-923b-0b791f4e93d1","Type":"ContainerDied","Data":"132cedb7a1ec0f3f8f501c9ba19586c0d408ad32f9d6bc41439e290a81036e05"} Nov 04 02:43:50.348858 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:43:50.348719 4405 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-69464886d7-2vmms" event={"ID":"8c1cdb1d-4616-4821-923b-0b791f4e93d1","Type":"ContainerDied","Data":"6a4bc9a631f0054c2d82a9f061b13cb49746b11fb52e309f09f1995d44e72fce"} Nov 04 02:43:50.348858 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:43:50.348732 4405 scope.go:122] "RemoveContainer" containerID="132cedb7a1ec0f3f8f501c9ba19586c0d408ad32f9d6bc41439e290a81036e05" Nov 04 02:43:50.357775 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:43:50.357758 4405 scope.go:122] "RemoveContainer" containerID="132cedb7a1ec0f3f8f501c9ba19586c0d408ad32f9d6bc41439e290a81036e05" Nov 04 02:43:50.358041 ip-10-0-134-125 kubenswrapper[4405]: E1104 02:43:50.358023 4405 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"132cedb7a1ec0f3f8f501c9ba19586c0d408ad32f9d6bc41439e290a81036e05\": container with ID starting with 132cedb7a1ec0f3f8f501c9ba19586c0d408ad32f9d6bc41439e290a81036e05 not found: ID does not exist" containerID="132cedb7a1ec0f3f8f501c9ba19586c0d408ad32f9d6bc41439e290a81036e05" Nov 04 02:43:50.358079 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:43:50.358051 4405 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"132cedb7a1ec0f3f8f501c9ba19586c0d408ad32f9d6bc41439e290a81036e05"} err="failed to get container status \"132cedb7a1ec0f3f8f501c9ba19586c0d408ad32f9d6bc41439e290a81036e05\": rpc error: code = NotFound desc = could not find container \"132cedb7a1ec0f3f8f501c9ba19586c0d408ad32f9d6bc41439e290a81036e05\": container with ID starting with 132cedb7a1ec0f3f8f501c9ba19586c0d408ad32f9d6bc41439e290a81036e05 not found: ID does not exist" Nov 04 02:43:50.517057 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:43:50.517022 4405 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/ai-gateway-controller-69464886d7-2vmms"] Nov 04 02:43:50.521069 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:43:50.521044 4405 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/ai-gateway-controller-69464886d7-2vmms"] Nov 04 02:43:51.700439 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:43:51.700404 4405 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8c1cdb1d-4616-4821-923b-0b791f4e93d1" path="/var/lib/kubelet/pods/8c1cdb1d-4616-4821-923b-0b791f4e93d1/volumes" Nov 04 02:44:00.622494 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:44:00.622459 4405 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-76597d6855-4ndqx"] Nov 04 02:44:00.622900 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:44:00.622858 4405 cpu_manager.go:414] "containerMap: removing container" podUID="8c1cdb1d-4616-4821-923b-0b791f4e93d1" containerName="manager" Nov 04 02:44:00.622900 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:44:00.622870 4405 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8c1cdb1d-4616-4821-923b-0b791f4e93d1" containerName="manager" Nov 04 02:44:00.622971 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:44:00.622955 4405 memory_manager.go:365] "RemoveStaleState removing state" podUID="8c1cdb1d-4616-4821-923b-0b791f4e93d1" containerName="manager" Nov 04 02:44:00.628586 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:44:00.628569 4405 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-4ndqx" Nov 04 02:44:00.663101 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:44:00.663061 4405 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a236d45b-5615-4d69-9043-52de98c2ab38-istio-envoy\") pod \"maas-default-gateway-openshift-default-76597d6855-4ndqx\" (UID: \"a236d45b-5615-4d69-9043-52de98c2ab38\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-4ndqx" Nov 04 02:44:00.663257 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:44:00.663113 4405 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l2dr4\" (UniqueName: \"kubernetes.io/projected/a236d45b-5615-4d69-9043-52de98c2ab38-kube-api-access-l2dr4\") pod \"maas-default-gateway-openshift-default-76597d6855-4ndqx\" (UID: \"a236d45b-5615-4d69-9043-52de98c2ab38\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-4ndqx" Nov 04 02:44:00.663257 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:44:00.663192 4405 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a236d45b-5615-4d69-9043-52de98c2ab38-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-76597d6855-4ndqx\" (UID: \"a236d45b-5615-4d69-9043-52de98c2ab38\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-4ndqx" Nov 04 02:44:00.663339 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:44:00.663264 4405 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a236d45b-5615-4d69-9043-52de98c2ab38-istio-token\") pod \"maas-default-gateway-openshift-default-76597d6855-4ndqx\" (UID: \"a236d45b-5615-4d69-9043-52de98c2ab38\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-4ndqx" Nov 04 02:44:00.663339 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:44:00.663311 4405 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a236d45b-5615-4d69-9043-52de98c2ab38-workload-certs\") pod \"maas-default-gateway-openshift-default-76597d6855-4ndqx\" (UID: \"a236d45b-5615-4d69-9043-52de98c2ab38\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-4ndqx" Nov 04 02:44:00.663396 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:44:00.663342 4405 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a236d45b-5615-4d69-9043-52de98c2ab38-credential-socket\") pod \"maas-default-gateway-openshift-default-76597d6855-4ndqx\" (UID: \"a236d45b-5615-4d69-9043-52de98c2ab38\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-4ndqx" Nov 04 02:44:00.663396 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:44:00.663378 4405 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a236d45b-5615-4d69-9043-52de98c2ab38-istio-podinfo\") pod \"maas-default-gateway-openshift-default-76597d6855-4ndqx\" (UID: \"a236d45b-5615-4d69-9043-52de98c2ab38\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-4ndqx" Nov 04 02:44:00.663509 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:44:00.663486 4405 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a236d45b-5615-4d69-9043-52de98c2ab38-workload-socket\") pod \"maas-default-gateway-openshift-default-76597d6855-4ndqx\" (UID: \"a236d45b-5615-4d69-9043-52de98c2ab38\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-4ndqx" Nov 04 02:44:00.663544 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:44:00.663526 4405 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a236d45b-5615-4d69-9043-52de98c2ab38-istio-data\") pod \"maas-default-gateway-openshift-default-76597d6855-4ndqx\" (UID: \"a236d45b-5615-4d69-9043-52de98c2ab38\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-4ndqx" Nov 04 02:44:00.722014 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:44:00.721970 4405 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-76597d6855-4ndqx"] Nov 04 02:44:00.764967 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:44:00.764934 4405 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a236d45b-5615-4d69-9043-52de98c2ab38-istio-token\") pod \"maas-default-gateway-openshift-default-76597d6855-4ndqx\" (UID: \"a236d45b-5615-4d69-9043-52de98c2ab38\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-4ndqx" Nov 04 02:44:00.765153 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:44:00.765026 4405 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a236d45b-5615-4d69-9043-52de98c2ab38-workload-certs\") pod \"maas-default-gateway-openshift-default-76597d6855-4ndqx\" (UID: \"a236d45b-5615-4d69-9043-52de98c2ab38\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-4ndqx" Nov 04 02:44:00.765153 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:44:00.765068 4405 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a236d45b-5615-4d69-9043-52de98c2ab38-credential-socket\") pod \"maas-default-gateway-openshift-default-76597d6855-4ndqx\" (UID: \"a236d45b-5615-4d69-9043-52de98c2ab38\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-4ndqx" Nov 04 02:44:00.765153 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:44:00.765091 4405 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a236d45b-5615-4d69-9043-52de98c2ab38-istio-podinfo\") pod \"maas-default-gateway-openshift-default-76597d6855-4ndqx\" (UID: \"a236d45b-5615-4d69-9043-52de98c2ab38\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-4ndqx" Nov 04 02:44:00.765153 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:44:00.765144 4405 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a236d45b-5615-4d69-9043-52de98c2ab38-workload-socket\") pod \"maas-default-gateway-openshift-default-76597d6855-4ndqx\" (UID: \"a236d45b-5615-4d69-9043-52de98c2ab38\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-4ndqx" Nov 04 02:44:00.765370 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:44:00.765174 4405 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a236d45b-5615-4d69-9043-52de98c2ab38-istio-data\") pod \"maas-default-gateway-openshift-default-76597d6855-4ndqx\" (UID: \"a236d45b-5615-4d69-9043-52de98c2ab38\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-4ndqx" Nov 04 02:44:00.765370 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:44:00.765211 4405 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a236d45b-5615-4d69-9043-52de98c2ab38-istio-envoy\") pod \"maas-default-gateway-openshift-default-76597d6855-4ndqx\" (UID: \"a236d45b-5615-4d69-9043-52de98c2ab38\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-4ndqx" Nov 04 02:44:00.765370 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:44:00.765240 4405 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l2dr4\" (UniqueName: \"kubernetes.io/projected/a236d45b-5615-4d69-9043-52de98c2ab38-kube-api-access-l2dr4\") pod \"maas-default-gateway-openshift-default-76597d6855-4ndqx\" (UID: \"a236d45b-5615-4d69-9043-52de98c2ab38\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-4ndqx" Nov 04 02:44:00.765604 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:44:00.765478 4405 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a236d45b-5615-4d69-9043-52de98c2ab38-workload-certs\") pod \"maas-default-gateway-openshift-default-76597d6855-4ndqx\" (UID: \"a236d45b-5615-4d69-9043-52de98c2ab38\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-4ndqx" Nov 04 02:44:00.765604 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:44:00.765534 4405 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a236d45b-5615-4d69-9043-52de98c2ab38-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-76597d6855-4ndqx\" (UID: \"a236d45b-5615-4d69-9043-52de98c2ab38\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-4ndqx" Nov 04 02:44:00.765736 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:44:00.765597 4405 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a236d45b-5615-4d69-9043-52de98c2ab38-credential-socket\") pod \"maas-default-gateway-openshift-default-76597d6855-4ndqx\" (UID: \"a236d45b-5615-4d69-9043-52de98c2ab38\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-4ndqx" Nov 04 02:44:00.765736 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:44:00.765654 4405 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a236d45b-5615-4d69-9043-52de98c2ab38-workload-socket\") pod \"maas-default-gateway-openshift-default-76597d6855-4ndqx\" (UID: \"a236d45b-5615-4d69-9043-52de98c2ab38\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-4ndqx" Nov 04 02:44:00.765813 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:44:00.765758 4405 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a236d45b-5615-4d69-9043-52de98c2ab38-istio-data\") pod \"maas-default-gateway-openshift-default-76597d6855-4ndqx\" (UID: \"a236d45b-5615-4d69-9043-52de98c2ab38\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-4ndqx" Nov 04 02:44:00.766226 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:44:00.766209 4405 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a236d45b-5615-4d69-9043-52de98c2ab38-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-76597d6855-4ndqx\" (UID: \"a236d45b-5615-4d69-9043-52de98c2ab38\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-4ndqx" Nov 04 02:44:00.767740 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:44:00.767717 4405 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a236d45b-5615-4d69-9043-52de98c2ab38-istio-envoy\") pod \"maas-default-gateway-openshift-default-76597d6855-4ndqx\" (UID: \"a236d45b-5615-4d69-9043-52de98c2ab38\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-4ndqx" Nov 04 02:44:00.767944 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:44:00.767923 4405 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a236d45b-5615-4d69-9043-52de98c2ab38-istio-podinfo\") pod \"maas-default-gateway-openshift-default-76597d6855-4ndqx\" (UID: \"a236d45b-5615-4d69-9043-52de98c2ab38\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-4ndqx" Nov 04 02:44:00.830656 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:44:00.830613 4405 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a236d45b-5615-4d69-9043-52de98c2ab38-istio-token\") pod \"maas-default-gateway-openshift-default-76597d6855-4ndqx\" (UID: \"a236d45b-5615-4d69-9043-52de98c2ab38\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-4ndqx" Nov 04 02:44:00.830898 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:44:00.830876 4405 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l2dr4\" (UniqueName: \"kubernetes.io/projected/a236d45b-5615-4d69-9043-52de98c2ab38-kube-api-access-l2dr4\") pod \"maas-default-gateway-openshift-default-76597d6855-4ndqx\" (UID: \"a236d45b-5615-4d69-9043-52de98c2ab38\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-4ndqx" Nov 04 02:44:00.941622 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:44:00.941546 4405 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-4ndqx" Nov 04 02:44:01.368387 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:44:01.368359 4405 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-76597d6855-4ndqx"] Nov 04 02:44:01.371039 ip-10-0-134-125 kubenswrapper[4405]: W1104 02:44:01.371010 4405 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda236d45b_5615_4d69_9043_52de98c2ab38.slice/crio-96145a2af9a45df2eb9e921bf7bc74b67bac365da8be1af8ff3e2587bedbb58e WatchSource:0}: Error finding container 96145a2af9a45df2eb9e921bf7bc74b67bac365da8be1af8ff3e2587bedbb58e: Status 404 returned error can't find the container with id 96145a2af9a45df2eb9e921bf7bc74b67bac365da8be1af8ff3e2587bedbb58e Nov 04 02:44:01.373292 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:44:01.373259 4405 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 04 02:44:01.373387 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:44:01.373325 4405 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 04 02:44:01.373387 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:44:01.373354 4405 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 04 02:44:01.402853 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:44:01.402827 4405 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-4ndqx" event={"ID":"a236d45b-5615-4d69-9043-52de98c2ab38","Type":"ContainerStarted","Data":"96145a2af9a45df2eb9e921bf7bc74b67bac365da8be1af8ff3e2587bedbb58e"} Nov 04 02:44:02.421609 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:44:02.421571 4405 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-4ndqx" event={"ID":"a236d45b-5615-4d69-9043-52de98c2ab38","Type":"ContainerStarted","Data":"9f736a640f6a81dee067ff6beb11f61696cd002aa50f12332518b080fef961c0"} Nov 04 02:44:02.595696 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:44:02.592076 4405 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-4ndqx" podStartSLOduration=2.5920567 podStartE2EDuration="2.5920567s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 02:44:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 02:44:02.591736237 +0000 UTC m=+689.022436358" watchObservedRunningTime="2026-11-04 02:44:02.5920567 +0000 UTC m=+689.022756831" Nov 04 02:44:02.941974 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:44:02.941922 4405 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-4ndqx" Nov 04 02:44:02.943579 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:44:02.943555 4405 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76597d6855-4ndqx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.33:15021/healthz/ready\": dial tcp 10.134.0.33:15021: connect: connection refused" start-of-body= Nov 04 02:44:02.943726 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:44:02.943603 4405 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-4ndqx" podUID="a236d45b-5615-4d69-9043-52de98c2ab38" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.33:15021/healthz/ready\": dial tcp 10.134.0.33:15021: connect: connection refused" Nov 04 02:44:03.941953 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:44:03.941917 4405 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76597d6855-4ndqx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.33:15021/healthz/ready\": dial tcp 10.134.0.33:15021: connect: connection refused" start-of-body= Nov 04 02:44:03.942414 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:44:03.941996 4405 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-4ndqx" podUID="a236d45b-5615-4d69-9043-52de98c2ab38" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.33:15021/healthz/ready\": dial tcp 10.134.0.33:15021: connect: connection refused" Nov 04 02:44:04.942079 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:44:04.942039 4405 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76597d6855-4ndqx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.33:15021/healthz/ready\": dial tcp 10.134.0.33:15021: connect: connection refused" start-of-body= Nov 04 02:44:04.942496 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:44:04.942117 4405 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-4ndqx" podUID="a236d45b-5615-4d69-9043-52de98c2ab38" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.33:15021/healthz/ready\": dial tcp 10.134.0.33:15021: connect: connection refused" Nov 04 02:44:05.942442 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:44:05.942402 4405 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76597d6855-4ndqx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.33:15021/healthz/ready\": dial tcp 10.134.0.33:15021: connect: connection refused" start-of-body= Nov 04 02:44:05.942893 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:44:05.942490 4405 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-4ndqx" podUID="a236d45b-5615-4d69-9043-52de98c2ab38" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.33:15021/healthz/ready\": dial tcp 10.134.0.33:15021: connect: connection refused" Nov 04 02:44:06.883249 ip-10-0-134-125 kubenswrapper[4405]: E1104 02:44:06.883205 4405 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5f2vb" containerName="extractor" Nov 04 02:44:06.942409 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:44:06.942374 4405 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76597d6855-4ndqx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.33:15021/healthz/ready\": dial tcp 10.134.0.33:15021: connect: connection refused" start-of-body= Nov 04 02:44:06.942586 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:44:06.942438 4405 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-4ndqx" podUID="a236d45b-5615-4d69-9043-52de98c2ab38" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.33:15021/healthz/ready\": dial tcp 10.134.0.33:15021: connect: connection refused" Nov 04 02:44:07.942977 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:44:07.942943 4405 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76597d6855-4ndqx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.33:15021/healthz/ready\": dial tcp 10.134.0.33:15021: connect: connection refused" start-of-body= Nov 04 02:44:07.943400 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:44:07.943001 4405 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-4ndqx" podUID="a236d45b-5615-4d69-9043-52de98c2ab38" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.33:15021/healthz/ready\": dial tcp 10.134.0.33:15021: connect: connection refused" Nov 04 02:44:08.942635 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:44:08.942598 4405 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76597d6855-4ndqx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.33:15021/healthz/ready\": dial tcp 10.134.0.33:15021: connect: connection refused" start-of-body= Nov 04 02:44:08.942842 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:44:08.942661 4405 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-4ndqx" podUID="a236d45b-5615-4d69-9043-52de98c2ab38" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.33:15021/healthz/ready\": dial tcp 10.134.0.33:15021: connect: connection refused" Nov 04 02:44:10.416525 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:44:10.416494 4405 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-4ndqx" Nov 04 02:44:10.416993 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:44:10.416808 4405 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-4ndqx" Nov 04 02:44:10.417759 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:44:10.417738 4405 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-4ndqx" Nov 04 02:44:29.278407 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:44:29.278372 4405 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-cdcf67-openshift-default-6ff8f5cbc7-rwgvq"] Nov 04 02:44:29.282046 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:44:29.282015 4405 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-cdcf67-openshift-default-6ff8f5cbc7-rwgvq" Nov 04 02:44:29.332878 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:44:29.332835 4405 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dad304e5-de0a-44b2-a137-8139a2aa2f52-istio-data\") pod \"e2e-worker-w3-cdcf67-openshift-default-6ff8f5cbc7-rwgvq\" (UID: \"dad304e5-de0a-44b2-a137-8139a2aa2f52\") " pod="openshift-ingress/e2e-worker-w3-cdcf67-openshift-default-6ff8f5cbc7-rwgvq" Nov 04 02:44:29.332878 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:44:29.332888 4405 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dad304e5-de0a-44b2-a137-8139a2aa2f52-istiod-ca-cert\") pod \"e2e-worker-w3-cdcf67-openshift-default-6ff8f5cbc7-rwgvq\" (UID: \"dad304e5-de0a-44b2-a137-8139a2aa2f52\") " pod="openshift-ingress/e2e-worker-w3-cdcf67-openshift-default-6ff8f5cbc7-rwgvq" Nov 04 02:44:29.333124 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:44:29.332979 4405 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dad304e5-de0a-44b2-a137-8139a2aa2f52-credential-socket\") pod \"e2e-worker-w3-cdcf67-openshift-default-6ff8f5cbc7-rwgvq\" (UID: \"dad304e5-de0a-44b2-a137-8139a2aa2f52\") " pod="openshift-ingress/e2e-worker-w3-cdcf67-openshift-default-6ff8f5cbc7-rwgvq" Nov 04 02:44:29.333124 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:44:29.333022 4405 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-htpww\" (UniqueName: \"kubernetes.io/projected/dad304e5-de0a-44b2-a137-8139a2aa2f52-kube-api-access-htpww\") pod \"e2e-worker-w3-cdcf67-openshift-default-6ff8f5cbc7-rwgvq\" (UID: \"dad304e5-de0a-44b2-a137-8139a2aa2f52\") " pod="openshift-ingress/e2e-worker-w3-cdcf67-openshift-default-6ff8f5cbc7-rwgvq" Nov 04 02:44:29.333192 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:44:29.333162 4405 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dad304e5-de0a-44b2-a137-8139a2aa2f52-istio-envoy\") pod \"e2e-worker-w3-cdcf67-openshift-default-6ff8f5cbc7-rwgvq\" (UID: \"dad304e5-de0a-44b2-a137-8139a2aa2f52\") " pod="openshift-ingress/e2e-worker-w3-cdcf67-openshift-default-6ff8f5cbc7-rwgvq" Nov 04 02:44:29.333236 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:44:29.333207 4405 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dad304e5-de0a-44b2-a137-8139a2aa2f52-istio-podinfo\") pod \"e2e-worker-w3-cdcf67-openshift-default-6ff8f5cbc7-rwgvq\" (UID: \"dad304e5-de0a-44b2-a137-8139a2aa2f52\") " pod="openshift-ingress/e2e-worker-w3-cdcf67-openshift-default-6ff8f5cbc7-rwgvq" Nov 04 02:44:29.333312 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:44:29.333292 4405 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dad304e5-de0a-44b2-a137-8139a2aa2f52-istio-token\") pod \"e2e-worker-w3-cdcf67-openshift-default-6ff8f5cbc7-rwgvq\" (UID: \"dad304e5-de0a-44b2-a137-8139a2aa2f52\") " pod="openshift-ingress/e2e-worker-w3-cdcf67-openshift-default-6ff8f5cbc7-rwgvq" Nov 04 02:44:29.333366 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:44:29.333330 4405 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dad304e5-de0a-44b2-a137-8139a2aa2f52-workload-certs\") pod \"e2e-worker-w3-cdcf67-openshift-default-6ff8f5cbc7-rwgvq\" (UID: \"dad304e5-de0a-44b2-a137-8139a2aa2f52\") " pod="openshift-ingress/e2e-worker-w3-cdcf67-openshift-default-6ff8f5cbc7-rwgvq" Nov 04 02:44:29.333417 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:44:29.333404 4405 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dad304e5-de0a-44b2-a137-8139a2aa2f52-workload-socket\") pod \"e2e-worker-w3-cdcf67-openshift-default-6ff8f5cbc7-rwgvq\" (UID: \"dad304e5-de0a-44b2-a137-8139a2aa2f52\") " pod="openshift-ingress/e2e-worker-w3-cdcf67-openshift-default-6ff8f5cbc7-rwgvq" Nov 04 02:44:29.342610 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:44:29.342576 4405 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-cdcf67-openshift-default-6ff8f5cbc7-rwgvq"] Nov 04 02:44:29.390881 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:44:29.390849 4405 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w3-cdcf67-openshift-default-dockercfg-pbxqp\"" Nov 04 02:44:29.433911 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:44:29.433880 4405 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dad304e5-de0a-44b2-a137-8139a2aa2f52-istio-data\") pod \"e2e-worker-w3-cdcf67-openshift-default-6ff8f5cbc7-rwgvq\" (UID: \"dad304e5-de0a-44b2-a137-8139a2aa2f52\") " pod="openshift-ingress/e2e-worker-w3-cdcf67-openshift-default-6ff8f5cbc7-rwgvq" Nov 04 02:44:29.433911 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:44:29.433912 4405 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dad304e5-de0a-44b2-a137-8139a2aa2f52-istiod-ca-cert\") pod \"e2e-worker-w3-cdcf67-openshift-default-6ff8f5cbc7-rwgvq\" (UID: \"dad304e5-de0a-44b2-a137-8139a2aa2f52\") " pod="openshift-ingress/e2e-worker-w3-cdcf67-openshift-default-6ff8f5cbc7-rwgvq" Nov 04 02:44:29.434139 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:44:29.433934 4405 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dad304e5-de0a-44b2-a137-8139a2aa2f52-credential-socket\") pod \"e2e-worker-w3-cdcf67-openshift-default-6ff8f5cbc7-rwgvq\" (UID: \"dad304e5-de0a-44b2-a137-8139a2aa2f52\") " pod="openshift-ingress/e2e-worker-w3-cdcf67-openshift-default-6ff8f5cbc7-rwgvq" Nov 04 02:44:29.434139 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:44:29.433954 4405 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-htpww\" (UniqueName: \"kubernetes.io/projected/dad304e5-de0a-44b2-a137-8139a2aa2f52-kube-api-access-htpww\") pod \"e2e-worker-w3-cdcf67-openshift-default-6ff8f5cbc7-rwgvq\" (UID: \"dad304e5-de0a-44b2-a137-8139a2aa2f52\") " pod="openshift-ingress/e2e-worker-w3-cdcf67-openshift-default-6ff8f5cbc7-rwgvq" Nov 04 02:44:29.434139 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:44:29.433998 4405 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dad304e5-de0a-44b2-a137-8139a2aa2f52-istio-envoy\") pod \"e2e-worker-w3-cdcf67-openshift-default-6ff8f5cbc7-rwgvq\" (UID: \"dad304e5-de0a-44b2-a137-8139a2aa2f52\") " pod="openshift-ingress/e2e-worker-w3-cdcf67-openshift-default-6ff8f5cbc7-rwgvq" Nov 04 02:44:29.434139 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:44:29.434016 4405 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dad304e5-de0a-44b2-a137-8139a2aa2f52-istio-podinfo\") pod \"e2e-worker-w3-cdcf67-openshift-default-6ff8f5cbc7-rwgvq\" (UID: \"dad304e5-de0a-44b2-a137-8139a2aa2f52\") " pod="openshift-ingress/e2e-worker-w3-cdcf67-openshift-default-6ff8f5cbc7-rwgvq" Nov 04 02:44:29.434139 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:44:29.434042 4405 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dad304e5-de0a-44b2-a137-8139a2aa2f52-istio-token\") pod \"e2e-worker-w3-cdcf67-openshift-default-6ff8f5cbc7-rwgvq\" (UID: \"dad304e5-de0a-44b2-a137-8139a2aa2f52\") " pod="openshift-ingress/e2e-worker-w3-cdcf67-openshift-default-6ff8f5cbc7-rwgvq" Nov 04 02:44:29.434139 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:44:29.434062 4405 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dad304e5-de0a-44b2-a137-8139a2aa2f52-workload-certs\") pod \"e2e-worker-w3-cdcf67-openshift-default-6ff8f5cbc7-rwgvq\" (UID: \"dad304e5-de0a-44b2-a137-8139a2aa2f52\") " pod="openshift-ingress/e2e-worker-w3-cdcf67-openshift-default-6ff8f5cbc7-rwgvq" Nov 04 02:44:29.434139 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:44:29.434092 4405 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dad304e5-de0a-44b2-a137-8139a2aa2f52-workload-socket\") pod \"e2e-worker-w3-cdcf67-openshift-default-6ff8f5cbc7-rwgvq\" (UID: \"dad304e5-de0a-44b2-a137-8139a2aa2f52\") " pod="openshift-ingress/e2e-worker-w3-cdcf67-openshift-default-6ff8f5cbc7-rwgvq" Nov 04 02:44:29.434470 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:44:29.434331 4405 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dad304e5-de0a-44b2-a137-8139a2aa2f52-istio-data\") pod \"e2e-worker-w3-cdcf67-openshift-default-6ff8f5cbc7-rwgvq\" (UID: \"dad304e5-de0a-44b2-a137-8139a2aa2f52\") " pod="openshift-ingress/e2e-worker-w3-cdcf67-openshift-default-6ff8f5cbc7-rwgvq" Nov 04 02:44:29.434470 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:44:29.434366 4405 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dad304e5-de0a-44b2-a137-8139a2aa2f52-credential-socket\") pod \"e2e-worker-w3-cdcf67-openshift-default-6ff8f5cbc7-rwgvq\" (UID: \"dad304e5-de0a-44b2-a137-8139a2aa2f52\") " pod="openshift-ingress/e2e-worker-w3-cdcf67-openshift-default-6ff8f5cbc7-rwgvq" Nov 04 02:44:29.434577 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:44:29.434538 4405 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dad304e5-de0a-44b2-a137-8139a2aa2f52-workload-socket\") pod \"e2e-worker-w3-cdcf67-openshift-default-6ff8f5cbc7-rwgvq\" (UID: \"dad304e5-de0a-44b2-a137-8139a2aa2f52\") " pod="openshift-ingress/e2e-worker-w3-cdcf67-openshift-default-6ff8f5cbc7-rwgvq" Nov 04 02:44:29.434635 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:44:29.434573 4405 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dad304e5-de0a-44b2-a137-8139a2aa2f52-workload-certs\") pod \"e2e-worker-w3-cdcf67-openshift-default-6ff8f5cbc7-rwgvq\" (UID: \"dad304e5-de0a-44b2-a137-8139a2aa2f52\") " pod="openshift-ingress/e2e-worker-w3-cdcf67-openshift-default-6ff8f5cbc7-rwgvq" Nov 04 02:44:29.434963 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:44:29.434938 4405 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dad304e5-de0a-44b2-a137-8139a2aa2f52-istiod-ca-cert\") pod \"e2e-worker-w3-cdcf67-openshift-default-6ff8f5cbc7-rwgvq\" (UID: \"dad304e5-de0a-44b2-a137-8139a2aa2f52\") " pod="openshift-ingress/e2e-worker-w3-cdcf67-openshift-default-6ff8f5cbc7-rwgvq" Nov 04 02:44:29.436504 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:44:29.436474 4405 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dad304e5-de0a-44b2-a137-8139a2aa2f52-istio-envoy\") pod \"e2e-worker-w3-cdcf67-openshift-default-6ff8f5cbc7-rwgvq\" (UID: \"dad304e5-de0a-44b2-a137-8139a2aa2f52\") " pod="openshift-ingress/e2e-worker-w3-cdcf67-openshift-default-6ff8f5cbc7-rwgvq" Nov 04 02:44:29.436895 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:44:29.436878 4405 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dad304e5-de0a-44b2-a137-8139a2aa2f52-istio-podinfo\") pod \"e2e-worker-w3-cdcf67-openshift-default-6ff8f5cbc7-rwgvq\" (UID: \"dad304e5-de0a-44b2-a137-8139a2aa2f52\") " pod="openshift-ingress/e2e-worker-w3-cdcf67-openshift-default-6ff8f5cbc7-rwgvq" Nov 04 02:44:29.496161 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:44:29.496123 4405 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dad304e5-de0a-44b2-a137-8139a2aa2f52-istio-token\") pod \"e2e-worker-w3-cdcf67-openshift-default-6ff8f5cbc7-rwgvq\" (UID: \"dad304e5-de0a-44b2-a137-8139a2aa2f52\") " pod="openshift-ingress/e2e-worker-w3-cdcf67-openshift-default-6ff8f5cbc7-rwgvq" Nov 04 02:44:29.496315 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:44:29.496238 4405 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-htpww\" (UniqueName: \"kubernetes.io/projected/dad304e5-de0a-44b2-a137-8139a2aa2f52-kube-api-access-htpww\") pod \"e2e-worker-w3-cdcf67-openshift-default-6ff8f5cbc7-rwgvq\" (UID: \"dad304e5-de0a-44b2-a137-8139a2aa2f52\") " pod="openshift-ingress/e2e-worker-w3-cdcf67-openshift-default-6ff8f5cbc7-rwgvq" Nov 04 02:44:29.595697 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:44:29.595637 4405 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-cdcf67-openshift-default-6ff8f5cbc7-rwgvq" Nov 04 02:44:29.938551 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:44:29.938468 4405 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-e516ff-openshift-default-6f4b957485-bjmn9"] Nov 04 02:44:29.942241 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:44:29.942219 4405 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-e516ff-openshift-default-6f4b957485-bjmn9" Nov 04 02:44:29.994800 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:44:29.994771 4405 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w1-e516ff-openshift-default-dockercfg-5x2gt\"" Nov 04 02:44:30.003507 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:44:30.003479 4405 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-e516ff-openshift-default-6f4b957485-bjmn9"] Nov 04 02:44:30.030078 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:44:30.030055 4405 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-cdcf67-openshift-default-6ff8f5cbc7-rwgvq"] Nov 04 02:44:30.032695 ip-10-0-134-125 kubenswrapper[4405]: W1104 02:44:30.032643 4405 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddad304e5_de0a_44b2_a137_8139a2aa2f52.slice/crio-8c7750471f5b1c23493d386a91a2700251a5b0c68dc090adb4b7c64b96cd81d5 WatchSource:0}: Error finding container 8c7750471f5b1c23493d386a91a2700251a5b0c68dc090adb4b7c64b96cd81d5: Status 404 returned error can't find the container with id 8c7750471f5b1c23493d386a91a2700251a5b0c68dc090adb4b7c64b96cd81d5 Nov 04 02:44:30.034918 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:44:30.034881 4405 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 04 02:44:30.035045 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:44:30.034957 4405 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 04 02:44:30.035045 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:44:30.034989 4405 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 04 02:44:30.038505 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:44:30.038487 4405 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3fd4ae35-84c1-45c1-a49c-14f2a8ed1c82-istio-podinfo\") pod \"e2e-worker-w1-e516ff-openshift-default-6f4b957485-bjmn9\" (UID: \"3fd4ae35-84c1-45c1-a49c-14f2a8ed1c82\") " pod="openshift-ingress/e2e-worker-w1-e516ff-openshift-default-6f4b957485-bjmn9" Nov 04 02:44:30.038583 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:44:30.038527 4405 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3fd4ae35-84c1-45c1-a49c-14f2a8ed1c82-workload-certs\") pod \"e2e-worker-w1-e516ff-openshift-default-6f4b957485-bjmn9\" (UID: \"3fd4ae35-84c1-45c1-a49c-14f2a8ed1c82\") " pod="openshift-ingress/e2e-worker-w1-e516ff-openshift-default-6f4b957485-bjmn9" Nov 04 02:44:30.038630 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:44:30.038612 4405 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v5n9f\" (UniqueName: \"kubernetes.io/projected/3fd4ae35-84c1-45c1-a49c-14f2a8ed1c82-kube-api-access-v5n9f\") pod \"e2e-worker-w1-e516ff-openshift-default-6f4b957485-bjmn9\" (UID: \"3fd4ae35-84c1-45c1-a49c-14f2a8ed1c82\") " pod="openshift-ingress/e2e-worker-w1-e516ff-openshift-default-6f4b957485-bjmn9" Nov 04 02:44:30.038700 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:44:30.038647 4405 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3fd4ae35-84c1-45c1-a49c-14f2a8ed1c82-istiod-ca-cert\") pod \"e2e-worker-w1-e516ff-openshift-default-6f4b957485-bjmn9\" (UID: \"3fd4ae35-84c1-45c1-a49c-14f2a8ed1c82\") " pod="openshift-ingress/e2e-worker-w1-e516ff-openshift-default-6f4b957485-bjmn9" Nov 04 02:44:30.038700 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:44:30.038691 4405 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3fd4ae35-84c1-45c1-a49c-14f2a8ed1c82-istio-envoy\") pod \"e2e-worker-w1-e516ff-openshift-default-6f4b957485-bjmn9\" (UID: \"3fd4ae35-84c1-45c1-a49c-14f2a8ed1c82\") " pod="openshift-ingress/e2e-worker-w1-e516ff-openshift-default-6f4b957485-bjmn9" Nov 04 02:44:30.038838 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:44:30.038824 4405 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3fd4ae35-84c1-45c1-a49c-14f2a8ed1c82-istio-token\") pod \"e2e-worker-w1-e516ff-openshift-default-6f4b957485-bjmn9\" (UID: \"3fd4ae35-84c1-45c1-a49c-14f2a8ed1c82\") " pod="openshift-ingress/e2e-worker-w1-e516ff-openshift-default-6f4b957485-bjmn9" Nov 04 02:44:30.038877 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:44:30.038851 4405 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3fd4ae35-84c1-45c1-a49c-14f2a8ed1c82-istio-data\") pod \"e2e-worker-w1-e516ff-openshift-default-6f4b957485-bjmn9\" (UID: \"3fd4ae35-84c1-45c1-a49c-14f2a8ed1c82\") " pod="openshift-ingress/e2e-worker-w1-e516ff-openshift-default-6f4b957485-bjmn9" Nov 04 02:44:30.038937 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:44:30.038880 4405 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3fd4ae35-84c1-45c1-a49c-14f2a8ed1c82-workload-socket\") pod \"e2e-worker-w1-e516ff-openshift-default-6f4b957485-bjmn9\" (UID: \"3fd4ae35-84c1-45c1-a49c-14f2a8ed1c82\") " pod="openshift-ingress/e2e-worker-w1-e516ff-openshift-default-6f4b957485-bjmn9" Nov 04 02:44:30.038984 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:44:30.038964 4405 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3fd4ae35-84c1-45c1-a49c-14f2a8ed1c82-credential-socket\") pod \"e2e-worker-w1-e516ff-openshift-default-6f4b957485-bjmn9\" (UID: \"3fd4ae35-84c1-45c1-a49c-14f2a8ed1c82\") " pod="openshift-ingress/e2e-worker-w1-e516ff-openshift-default-6f4b957485-bjmn9" Nov 04 02:44:30.140044 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:44:30.140000 4405 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3fd4ae35-84c1-45c1-a49c-14f2a8ed1c82-credential-socket\") pod \"e2e-worker-w1-e516ff-openshift-default-6f4b957485-bjmn9\" (UID: \"3fd4ae35-84c1-45c1-a49c-14f2a8ed1c82\") " pod="openshift-ingress/e2e-worker-w1-e516ff-openshift-default-6f4b957485-bjmn9" Nov 04 02:44:30.140044 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:44:30.140040 4405 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3fd4ae35-84c1-45c1-a49c-14f2a8ed1c82-istio-podinfo\") pod \"e2e-worker-w1-e516ff-openshift-default-6f4b957485-bjmn9\" (UID: \"3fd4ae35-84c1-45c1-a49c-14f2a8ed1c82\") " pod="openshift-ingress/e2e-worker-w1-e516ff-openshift-default-6f4b957485-bjmn9" Nov 04 02:44:30.140302 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:44:30.140092 4405 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3fd4ae35-84c1-45c1-a49c-14f2a8ed1c82-workload-certs\") pod \"e2e-worker-w1-e516ff-openshift-default-6f4b957485-bjmn9\" (UID: \"3fd4ae35-84c1-45c1-a49c-14f2a8ed1c82\") " pod="openshift-ingress/e2e-worker-w1-e516ff-openshift-default-6f4b957485-bjmn9" Nov 04 02:44:30.140302 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:44:30.140152 4405 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v5n9f\" (UniqueName: \"kubernetes.io/projected/3fd4ae35-84c1-45c1-a49c-14f2a8ed1c82-kube-api-access-v5n9f\") pod \"e2e-worker-w1-e516ff-openshift-default-6f4b957485-bjmn9\" (UID: \"3fd4ae35-84c1-45c1-a49c-14f2a8ed1c82\") " pod="openshift-ingress/e2e-worker-w1-e516ff-openshift-default-6f4b957485-bjmn9" Nov 04 02:44:30.140302 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:44:30.140185 4405 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3fd4ae35-84c1-45c1-a49c-14f2a8ed1c82-istiod-ca-cert\") pod \"e2e-worker-w1-e516ff-openshift-default-6f4b957485-bjmn9\" (UID: \"3fd4ae35-84c1-45c1-a49c-14f2a8ed1c82\") " pod="openshift-ingress/e2e-worker-w1-e516ff-openshift-default-6f4b957485-bjmn9" Nov 04 02:44:30.140302 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:44:30.140226 4405 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3fd4ae35-84c1-45c1-a49c-14f2a8ed1c82-istio-envoy\") pod \"e2e-worker-w1-e516ff-openshift-default-6f4b957485-bjmn9\" (UID: \"3fd4ae35-84c1-45c1-a49c-14f2a8ed1c82\") " pod="openshift-ingress/e2e-worker-w1-e516ff-openshift-default-6f4b957485-bjmn9" Nov 04 02:44:30.140302 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:44:30.140286 4405 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3fd4ae35-84c1-45c1-a49c-14f2a8ed1c82-istio-token\") pod \"e2e-worker-w1-e516ff-openshift-default-6f4b957485-bjmn9\" (UID: \"3fd4ae35-84c1-45c1-a49c-14f2a8ed1c82\") " pod="openshift-ingress/e2e-worker-w1-e516ff-openshift-default-6f4b957485-bjmn9" Nov 04 02:44:30.140571 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:44:30.140320 4405 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3fd4ae35-84c1-45c1-a49c-14f2a8ed1c82-istio-data\") pod \"e2e-worker-w1-e516ff-openshift-default-6f4b957485-bjmn9\" (UID: \"3fd4ae35-84c1-45c1-a49c-14f2a8ed1c82\") " pod="openshift-ingress/e2e-worker-w1-e516ff-openshift-default-6f4b957485-bjmn9" Nov 04 02:44:30.140571 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:44:30.140359 4405 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3fd4ae35-84c1-45c1-a49c-14f2a8ed1c82-workload-socket\") pod \"e2e-worker-w1-e516ff-openshift-default-6f4b957485-bjmn9\" (UID: \"3fd4ae35-84c1-45c1-a49c-14f2a8ed1c82\") " pod="openshift-ingress/e2e-worker-w1-e516ff-openshift-default-6f4b957485-bjmn9" Nov 04 02:44:30.140571 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:44:30.140444 4405 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3fd4ae35-84c1-45c1-a49c-14f2a8ed1c82-credential-socket\") pod \"e2e-worker-w1-e516ff-openshift-default-6f4b957485-bjmn9\" (UID: \"3fd4ae35-84c1-45c1-a49c-14f2a8ed1c82\") " pod="openshift-ingress/e2e-worker-w1-e516ff-openshift-default-6f4b957485-bjmn9" Nov 04 02:44:30.140571 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:44:30.140503 4405 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3fd4ae35-84c1-45c1-a49c-14f2a8ed1c82-workload-certs\") pod \"e2e-worker-w1-e516ff-openshift-default-6f4b957485-bjmn9\" (UID: \"3fd4ae35-84c1-45c1-a49c-14f2a8ed1c82\") " pod="openshift-ingress/e2e-worker-w1-e516ff-openshift-default-6f4b957485-bjmn9" Nov 04 02:44:30.140902 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:44:30.140650 4405 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3fd4ae35-84c1-45c1-a49c-14f2a8ed1c82-workload-socket\") pod \"e2e-worker-w1-e516ff-openshift-default-6f4b957485-bjmn9\" (UID: \"3fd4ae35-84c1-45c1-a49c-14f2a8ed1c82\") " pod="openshift-ingress/e2e-worker-w1-e516ff-openshift-default-6f4b957485-bjmn9" Nov 04 02:44:30.140902 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:44:30.140816 4405 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3fd4ae35-84c1-45c1-a49c-14f2a8ed1c82-istio-data\") pod \"e2e-worker-w1-e516ff-openshift-default-6f4b957485-bjmn9\" (UID: \"3fd4ae35-84c1-45c1-a49c-14f2a8ed1c82\") " pod="openshift-ingress/e2e-worker-w1-e516ff-openshift-default-6f4b957485-bjmn9" Nov 04 02:44:30.141124 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:44:30.141103 4405 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3fd4ae35-84c1-45c1-a49c-14f2a8ed1c82-istiod-ca-cert\") pod \"e2e-worker-w1-e516ff-openshift-default-6f4b957485-bjmn9\" (UID: \"3fd4ae35-84c1-45c1-a49c-14f2a8ed1c82\") " pod="openshift-ingress/e2e-worker-w1-e516ff-openshift-default-6f4b957485-bjmn9" Nov 04 02:44:30.142836 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:44:30.142812 4405 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3fd4ae35-84c1-45c1-a49c-14f2a8ed1c82-istio-envoy\") pod \"e2e-worker-w1-e516ff-openshift-default-6f4b957485-bjmn9\" (UID: \"3fd4ae35-84c1-45c1-a49c-14f2a8ed1c82\") " pod="openshift-ingress/e2e-worker-w1-e516ff-openshift-default-6f4b957485-bjmn9" Nov 04 02:44:30.143280 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:44:30.143263 4405 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3fd4ae35-84c1-45c1-a49c-14f2a8ed1c82-istio-podinfo\") pod \"e2e-worker-w1-e516ff-openshift-default-6f4b957485-bjmn9\" (UID: \"3fd4ae35-84c1-45c1-a49c-14f2a8ed1c82\") " pod="openshift-ingress/e2e-worker-w1-e516ff-openshift-default-6f4b957485-bjmn9" Nov 04 02:44:30.201438 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:44:30.201364 4405 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3fd4ae35-84c1-45c1-a49c-14f2a8ed1c82-istio-token\") pod \"e2e-worker-w1-e516ff-openshift-default-6f4b957485-bjmn9\" (UID: \"3fd4ae35-84c1-45c1-a49c-14f2a8ed1c82\") " pod="openshift-ingress/e2e-worker-w1-e516ff-openshift-default-6f4b957485-bjmn9" Nov 04 02:44:30.201644 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:44:30.201619 4405 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v5n9f\" (UniqueName: \"kubernetes.io/projected/3fd4ae35-84c1-45c1-a49c-14f2a8ed1c82-kube-api-access-v5n9f\") pod \"e2e-worker-w1-e516ff-openshift-default-6f4b957485-bjmn9\" (UID: \"3fd4ae35-84c1-45c1-a49c-14f2a8ed1c82\") " pod="openshift-ingress/e2e-worker-w1-e516ff-openshift-default-6f4b957485-bjmn9" Nov 04 02:44:30.255685 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:44:30.255639 4405 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-e516ff-openshift-default-6f4b957485-bjmn9" Nov 04 02:44:30.550569 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:44:30.550482 4405 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-cdcf67-openshift-default-6ff8f5cbc7-rwgvq" event={"ID":"dad304e5-de0a-44b2-a137-8139a2aa2f52","Type":"ContainerStarted","Data":"443defcf2b22d24c59168e7d581684bb84006228c2e217384d9131e127596c20"} Nov 04 02:44:30.550569 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:44:30.550546 4405 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-cdcf67-openshift-default-6ff8f5cbc7-rwgvq" event={"ID":"dad304e5-de0a-44b2-a137-8139a2aa2f52","Type":"ContainerStarted","Data":"8c7750471f5b1c23493d386a91a2700251a5b0c68dc090adb4b7c64b96cd81d5"} Nov 04 02:44:30.587329 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:44:30.587297 4405 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w5-3b7476-openshift-default-59975457bf-msl8m"] Nov 04 02:44:30.591926 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:44:30.591902 4405 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-3b7476-openshift-default-59975457bf-msl8m" Nov 04 02:44:30.596165 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:44:30.596146 4405 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-cdcf67-openshift-default-6ff8f5cbc7-rwgvq" Nov 04 02:44:30.646844 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:44:30.646810 4405 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-74xm5\" (UniqueName: \"kubernetes.io/projected/8a0008f3-4ab9-4ceb-8feb-0f536776610a-kube-api-access-74xm5\") pod \"e2e-worker-w5-3b7476-openshift-default-59975457bf-msl8m\" (UID: \"8a0008f3-4ab9-4ceb-8feb-0f536776610a\") " pod="openshift-ingress/e2e-worker-w5-3b7476-openshift-default-59975457bf-msl8m" Nov 04 02:44:30.647031 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:44:30.646868 4405 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8a0008f3-4ab9-4ceb-8feb-0f536776610a-istio-token\") pod \"e2e-worker-w5-3b7476-openshift-default-59975457bf-msl8m\" (UID: \"8a0008f3-4ab9-4ceb-8feb-0f536776610a\") " pod="openshift-ingress/e2e-worker-w5-3b7476-openshift-default-59975457bf-msl8m" Nov 04 02:44:30.647105 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:44:30.647085 4405 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8a0008f3-4ab9-4ceb-8feb-0f536776610a-workload-certs\") pod \"e2e-worker-w5-3b7476-openshift-default-59975457bf-msl8m\" (UID: \"8a0008f3-4ab9-4ceb-8feb-0f536776610a\") " pod="openshift-ingress/e2e-worker-w5-3b7476-openshift-default-59975457bf-msl8m" Nov 04 02:44:30.647163 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:44:30.647128 4405 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8a0008f3-4ab9-4ceb-8feb-0f536776610a-workload-socket\") pod \"e2e-worker-w5-3b7476-openshift-default-59975457bf-msl8m\" (UID: \"8a0008f3-4ab9-4ceb-8feb-0f536776610a\") " pod="openshift-ingress/e2e-worker-w5-3b7476-openshift-default-59975457bf-msl8m" Nov 04 02:44:30.647323 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:44:30.647294 4405 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8a0008f3-4ab9-4ceb-8feb-0f536776610a-credential-socket\") pod \"e2e-worker-w5-3b7476-openshift-default-59975457bf-msl8m\" (UID: \"8a0008f3-4ab9-4ceb-8feb-0f536776610a\") " pod="openshift-ingress/e2e-worker-w5-3b7476-openshift-default-59975457bf-msl8m" Nov 04 02:44:30.647467 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:44:30.647375 4405 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8a0008f3-4ab9-4ceb-8feb-0f536776610a-istio-envoy\") pod \"e2e-worker-w5-3b7476-openshift-default-59975457bf-msl8m\" (UID: \"8a0008f3-4ab9-4ceb-8feb-0f536776610a\") " pod="openshift-ingress/e2e-worker-w5-3b7476-openshift-default-59975457bf-msl8m" Nov 04 02:44:30.647467 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:44:30.647434 4405 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8a0008f3-4ab9-4ceb-8feb-0f536776610a-istio-data\") pod \"e2e-worker-w5-3b7476-openshift-default-59975457bf-msl8m\" (UID: \"8a0008f3-4ab9-4ceb-8feb-0f536776610a\") " pod="openshift-ingress/e2e-worker-w5-3b7476-openshift-default-59975457bf-msl8m" Nov 04 02:44:30.647467 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:44:30.647464 4405 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8a0008f3-4ab9-4ceb-8feb-0f536776610a-istiod-ca-cert\") pod \"e2e-worker-w5-3b7476-openshift-default-59975457bf-msl8m\" (UID: \"8a0008f3-4ab9-4ceb-8feb-0f536776610a\") " pod="openshift-ingress/e2e-worker-w5-3b7476-openshift-default-59975457bf-msl8m" Nov 04 02:44:30.647645 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:44:30.647493 4405 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8a0008f3-4ab9-4ceb-8feb-0f536776610a-istio-podinfo\") pod \"e2e-worker-w5-3b7476-openshift-default-59975457bf-msl8m\" (UID: \"8a0008f3-4ab9-4ceb-8feb-0f536776610a\") " pod="openshift-ingress/e2e-worker-w5-3b7476-openshift-default-59975457bf-msl8m" Nov 04 02:44:30.653122 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:44:30.653097 4405 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-3b7476-openshift-default-59975457bf-msl8m"] Nov 04 02:44:30.670976 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:44:30.670910 4405 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-cdcf67-openshift-default-6ff8f5cbc7-rwgvq" podStartSLOduration=1.670891511 podStartE2EDuration="1.670891511s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 02:44:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 02:44:30.669485766 +0000 UTC m=+717.100185885" watchObservedRunningTime="2026-11-04 02:44:30.670891511 +0000 UTC m=+717.101591632" Nov 04 02:44:30.699473 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:44:30.699446 4405 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w5-3b7476-openshift-default-dockercfg-t2888\"" Nov 04 02:44:30.708779 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:44:30.708756 4405 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-e516ff-openshift-default-6f4b957485-bjmn9"] Nov 04 02:44:30.711234 ip-10-0-134-125 kubenswrapper[4405]: W1104 02:44:30.711207 4405 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3fd4ae35_84c1_45c1_a49c_14f2a8ed1c82.slice/crio-925d73507f5b928b84a3b79b8b6216df3c1d6a231ab744b7f472a3b626cd6d25 WatchSource:0}: Error finding container 925d73507f5b928b84a3b79b8b6216df3c1d6a231ab744b7f472a3b626cd6d25: Status 404 returned error can't find the container with id 925d73507f5b928b84a3b79b8b6216df3c1d6a231ab744b7f472a3b626cd6d25 Nov 04 02:44:30.714136 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:44:30.714106 4405 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 04 02:44:30.714212 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:44:30.714168 4405 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 04 02:44:30.714212 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:44:30.714198 4405 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 04 02:44:30.747924 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:44:30.747896 4405 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8a0008f3-4ab9-4ceb-8feb-0f536776610a-istio-podinfo\") pod \"e2e-worker-w5-3b7476-openshift-default-59975457bf-msl8m\" (UID: \"8a0008f3-4ab9-4ceb-8feb-0f536776610a\") " pod="openshift-ingress/e2e-worker-w5-3b7476-openshift-default-59975457bf-msl8m" Nov 04 02:44:30.748055 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:44:30.747934 4405 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-74xm5\" (UniqueName: \"kubernetes.io/projected/8a0008f3-4ab9-4ceb-8feb-0f536776610a-kube-api-access-74xm5\") pod \"e2e-worker-w5-3b7476-openshift-default-59975457bf-msl8m\" (UID: \"8a0008f3-4ab9-4ceb-8feb-0f536776610a\") " pod="openshift-ingress/e2e-worker-w5-3b7476-openshift-default-59975457bf-msl8m" Nov 04 02:44:30.748055 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:44:30.747976 4405 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8a0008f3-4ab9-4ceb-8feb-0f536776610a-istio-token\") pod \"e2e-worker-w5-3b7476-openshift-default-59975457bf-msl8m\" (UID: \"8a0008f3-4ab9-4ceb-8feb-0f536776610a\") " pod="openshift-ingress/e2e-worker-w5-3b7476-openshift-default-59975457bf-msl8m" Nov 04 02:44:30.748055 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:44:30.748013 4405 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8a0008f3-4ab9-4ceb-8feb-0f536776610a-workload-certs\") pod \"e2e-worker-w5-3b7476-openshift-default-59975457bf-msl8m\" (UID: \"8a0008f3-4ab9-4ceb-8feb-0f536776610a\") " pod="openshift-ingress/e2e-worker-w5-3b7476-openshift-default-59975457bf-msl8m" Nov 04 02:44:30.748055 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:44:30.748042 4405 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8a0008f3-4ab9-4ceb-8feb-0f536776610a-workload-socket\") pod \"e2e-worker-w5-3b7476-openshift-default-59975457bf-msl8m\" (UID: \"8a0008f3-4ab9-4ceb-8feb-0f536776610a\") " pod="openshift-ingress/e2e-worker-w5-3b7476-openshift-default-59975457bf-msl8m" Nov 04 02:44:30.748267 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:44:30.748182 4405 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8a0008f3-4ab9-4ceb-8feb-0f536776610a-credential-socket\") pod \"e2e-worker-w5-3b7476-openshift-default-59975457bf-msl8m\" (UID: \"8a0008f3-4ab9-4ceb-8feb-0f536776610a\") " pod="openshift-ingress/e2e-worker-w5-3b7476-openshift-default-59975457bf-msl8m" Nov 04 02:44:30.748267 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:44:30.748242 4405 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8a0008f3-4ab9-4ceb-8feb-0f536776610a-istio-envoy\") pod \"e2e-worker-w5-3b7476-openshift-default-59975457bf-msl8m\" (UID: \"8a0008f3-4ab9-4ceb-8feb-0f536776610a\") " pod="openshift-ingress/e2e-worker-w5-3b7476-openshift-default-59975457bf-msl8m" Nov 04 02:44:30.748376 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:44:30.748285 4405 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8a0008f3-4ab9-4ceb-8feb-0f536776610a-istio-data\") pod \"e2e-worker-w5-3b7476-openshift-default-59975457bf-msl8m\" (UID: \"8a0008f3-4ab9-4ceb-8feb-0f536776610a\") " pod="openshift-ingress/e2e-worker-w5-3b7476-openshift-default-59975457bf-msl8m" Nov 04 02:44:30.748376 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:44:30.748316 4405 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8a0008f3-4ab9-4ceb-8feb-0f536776610a-istiod-ca-cert\") pod \"e2e-worker-w5-3b7476-openshift-default-59975457bf-msl8m\" (UID: \"8a0008f3-4ab9-4ceb-8feb-0f536776610a\") " pod="openshift-ingress/e2e-worker-w5-3b7476-openshift-default-59975457bf-msl8m" Nov 04 02:44:30.748475 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:44:30.748430 4405 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8a0008f3-4ab9-4ceb-8feb-0f536776610a-workload-certs\") pod \"e2e-worker-w5-3b7476-openshift-default-59975457bf-msl8m\" (UID: \"8a0008f3-4ab9-4ceb-8feb-0f536776610a\") " pod="openshift-ingress/e2e-worker-w5-3b7476-openshift-default-59975457bf-msl8m" Nov 04 02:44:30.748577 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:44:30.748540 4405 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8a0008f3-4ab9-4ceb-8feb-0f536776610a-credential-socket\") pod \"e2e-worker-w5-3b7476-openshift-default-59975457bf-msl8m\" (UID: \"8a0008f3-4ab9-4ceb-8feb-0f536776610a\") " pod="openshift-ingress/e2e-worker-w5-3b7476-openshift-default-59975457bf-msl8m" Nov 04 02:44:30.748703 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:44:30.748652 4405 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8a0008f3-4ab9-4ceb-8feb-0f536776610a-workload-socket\") pod \"e2e-worker-w5-3b7476-openshift-default-59975457bf-msl8m\" (UID: \"8a0008f3-4ab9-4ceb-8feb-0f536776610a\") " pod="openshift-ingress/e2e-worker-w5-3b7476-openshift-default-59975457bf-msl8m" Nov 04 02:44:30.748789 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:44:30.748706 4405 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8a0008f3-4ab9-4ceb-8feb-0f536776610a-istio-data\") pod \"e2e-worker-w5-3b7476-openshift-default-59975457bf-msl8m\" (UID: \"8a0008f3-4ab9-4ceb-8feb-0f536776610a\") " pod="openshift-ingress/e2e-worker-w5-3b7476-openshift-default-59975457bf-msl8m" Nov 04 02:44:30.749321 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:44:30.749299 4405 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8a0008f3-4ab9-4ceb-8feb-0f536776610a-istiod-ca-cert\") pod \"e2e-worker-w5-3b7476-openshift-default-59975457bf-msl8m\" (UID: \"8a0008f3-4ab9-4ceb-8feb-0f536776610a\") " pod="openshift-ingress/e2e-worker-w5-3b7476-openshift-default-59975457bf-msl8m" Nov 04 02:44:30.750454 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:44:30.750433 4405 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8a0008f3-4ab9-4ceb-8feb-0f536776610a-istio-envoy\") pod \"e2e-worker-w5-3b7476-openshift-default-59975457bf-msl8m\" (UID: \"8a0008f3-4ab9-4ceb-8feb-0f536776610a\") " pod="openshift-ingress/e2e-worker-w5-3b7476-openshift-default-59975457bf-msl8m" Nov 04 02:44:30.750727 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:44:30.750710 4405 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8a0008f3-4ab9-4ceb-8feb-0f536776610a-istio-podinfo\") pod \"e2e-worker-w5-3b7476-openshift-default-59975457bf-msl8m\" (UID: \"8a0008f3-4ab9-4ceb-8feb-0f536776610a\") " pod="openshift-ingress/e2e-worker-w5-3b7476-openshift-default-59975457bf-msl8m" Nov 04 02:44:30.810368 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:44:30.808963 4405 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8a0008f3-4ab9-4ceb-8feb-0f536776610a-istio-token\") pod \"e2e-worker-w5-3b7476-openshift-default-59975457bf-msl8m\" (UID: \"8a0008f3-4ab9-4ceb-8feb-0f536776610a\") " pod="openshift-ingress/e2e-worker-w5-3b7476-openshift-default-59975457bf-msl8m" Nov 04 02:44:30.810368 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:44:30.809161 4405 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-74xm5\" (UniqueName: \"kubernetes.io/projected/8a0008f3-4ab9-4ceb-8feb-0f536776610a-kube-api-access-74xm5\") pod \"e2e-worker-w5-3b7476-openshift-default-59975457bf-msl8m\" (UID: \"8a0008f3-4ab9-4ceb-8feb-0f536776610a\") " pod="openshift-ingress/e2e-worker-w5-3b7476-openshift-default-59975457bf-msl8m" Nov 04 02:44:30.904716 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:44:30.904659 4405 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-3b7476-openshift-default-59975457bf-msl8m" Nov 04 02:44:31.298424 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:44:31.298396 4405 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-3b7476-openshift-default-59975457bf-msl8m"] Nov 04 02:44:31.300139 ip-10-0-134-125 kubenswrapper[4405]: W1104 02:44:31.300108 4405 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8a0008f3_4ab9_4ceb_8feb_0f536776610a.slice/crio-d58b6d456cfb5364ee59eadf97dfb1e570168cc90911c0197bb520fb34e2cfbb WatchSource:0}: Error finding container d58b6d456cfb5364ee59eadf97dfb1e570168cc90911c0197bb520fb34e2cfbb: Status 404 returned error can't find the container with id d58b6d456cfb5364ee59eadf97dfb1e570168cc90911c0197bb520fb34e2cfbb Nov 04 02:44:31.302567 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:44:31.302536 4405 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 04 02:44:31.302687 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:44:31.302614 4405 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 04 02:44:31.302756 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:44:31.302659 4405 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 04 02:44:31.556691 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:44:31.556566 4405 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-3b7476-openshift-default-59975457bf-msl8m" event={"ID":"8a0008f3-4ab9-4ceb-8feb-0f536776610a","Type":"ContainerStarted","Data":"b6a2edf16fc64d8d3cfeceaa4932a3db1cdd4fcbfee7de8866224fa041a79b0b"} Nov 04 02:44:31.556691 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:44:31.556636 4405 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-3b7476-openshift-default-59975457bf-msl8m" event={"ID":"8a0008f3-4ab9-4ceb-8feb-0f536776610a","Type":"ContainerStarted","Data":"d58b6d456cfb5364ee59eadf97dfb1e570168cc90911c0197bb520fb34e2cfbb"} Nov 04 02:44:31.558231 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:44:31.558209 4405 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-e516ff-openshift-default-6f4b957485-bjmn9" event={"ID":"3fd4ae35-84c1-45c1-a49c-14f2a8ed1c82","Type":"ContainerStarted","Data":"dc4e5d146d6b9dd6d75a3cb505756dee5d5552f16791b4deb553fc92ab758cc8"} Nov 04 02:44:31.558373 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:44:31.558247 4405 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-e516ff-openshift-default-6f4b957485-bjmn9" event={"ID":"3fd4ae35-84c1-45c1-a49c-14f2a8ed1c82","Type":"ContainerStarted","Data":"925d73507f5b928b84a3b79b8b6216df3c1d6a231ab744b7f472a3b626cd6d25"} Nov 04 02:44:31.602760 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:44:31.602730 4405 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-cdcf67-openshift-default-6ff8f5cbc7-rwgvq" Nov 04 02:44:31.677158 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:44:31.677093 4405 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w5-3b7476-openshift-default-59975457bf-msl8m" podStartSLOduration=1.677076038 podStartE2EDuration="1.677076038s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 02:44:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 02:44:31.675681346 +0000 UTC m=+718.106381461" watchObservedRunningTime="2026-11-04 02:44:31.677076038 +0000 UTC m=+718.107776159" Nov 04 02:44:31.905555 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:44:31.905516 4405 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w5-3b7476-openshift-default-59975457bf-msl8m" Nov 04 02:44:31.924951 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:44:31.924812 4405 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-e516ff-openshift-default-6f4b957485-bjmn9" podStartSLOduration=2.924798623 podStartE2EDuration="2.924798623s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 02:44:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 02:44:31.923644994 +0000 UTC m=+718.354345117" watchObservedRunningTime="2026-11-04 02:44:31.924798623 +0000 UTC m=+718.355498743" Nov 04 02:44:32.256422 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:44:32.256321 4405 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-e516ff-openshift-default-6f4b957485-bjmn9" Nov 04 02:44:32.261265 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:44:32.261242 4405 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-e516ff-openshift-default-6f4b957485-bjmn9" Nov 04 02:44:32.564111 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:44:32.564022 4405 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-e516ff-openshift-default-6f4b957485-bjmn9" Nov 04 02:44:32.564111 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:44:32.564087 4405 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-cdcf67-openshift-default-6ff8f5cbc7-rwgvq" Nov 04 02:44:32.564887 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:44:32.564868 4405 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-e516ff-openshift-default-6f4b957485-bjmn9" Nov 04 02:44:32.564985 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:44:32.564917 4405 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-cdcf67-openshift-default-6ff8f5cbc7-rwgvq" Nov 04 02:44:32.910913 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:44:32.910885 4405 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w5-3b7476-openshift-default-59975457bf-msl8m" Nov 04 02:44:33.568023 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:44:33.567990 4405 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w5-3b7476-openshift-default-59975457bf-msl8m" Nov 04 02:44:33.569047 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:44:33.569028 4405 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w5-3b7476-openshift-default-59975457bf-msl8m" Nov 04 02:44:36.882815 ip-10-0-134-125 kubenswrapper[4405]: E1104 02:44:36.882776 4405 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5f2vb" containerName="extractor" Nov 04 02:44:37.867994 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:44:37.867960 4405 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w2-a0bcbc-69fdc5f5c9-d4d6x"] Nov 04 02:44:37.872588 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:44:37.872571 4405 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-a0bcbc-69fdc5f5c9-d4d6x" Nov 04 02:44:37.916590 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:44:37.916562 4405 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ed86675a-731f-4625-bca5-02ba4200e7ad-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w2-a0bcbc-69fdc5f5c9-d4d6x\" (UID: \"ed86675a-731f-4625-bca5-02ba4200e7ad\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-a0bcbc-69fdc5f5c9-d4d6x" Nov 04 02:44:37.916952 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:44:37.916623 4405 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5f4mh\" (UniqueName: \"kubernetes.io/projected/ed86675a-731f-4625-bca5-02ba4200e7ad-kube-api-access-5f4mh\") pod \"payload-pre-processing-e2e-worker-w2-a0bcbc-69fdc5f5c9-d4d6x\" (UID: \"ed86675a-731f-4625-bca5-02ba4200e7ad\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-a0bcbc-69fdc5f5c9-d4d6x" Nov 04 02:44:37.930188 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:44:37.930163 4405 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w2-a0bcbc\"" Nov 04 02:44:37.984896 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:44:37.984860 4405 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w2-a0bcbc-69fdc5f5c9-d4d6x"] Nov 04 02:44:37.985093 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:44:37.984997 4405 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w2-a0bcbc-dockercfg-4rk4g\"" Nov 04 02:44:38.018145 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:44:38.018103 4405 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5f4mh\" (UniqueName: \"kubernetes.io/projected/ed86675a-731f-4625-bca5-02ba4200e7ad-kube-api-access-5f4mh\") pod \"payload-pre-processing-e2e-worker-w2-a0bcbc-69fdc5f5c9-d4d6x\" (UID: \"ed86675a-731f-4625-bca5-02ba4200e7ad\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-a0bcbc-69fdc5f5c9-d4d6x" Nov 04 02:44:38.018317 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:44:38.018296 4405 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ed86675a-731f-4625-bca5-02ba4200e7ad-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w2-a0bcbc-69fdc5f5c9-d4d6x\" (UID: \"ed86675a-731f-4625-bca5-02ba4200e7ad\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-a0bcbc-69fdc5f5c9-d4d6x" Nov 04 02:44:38.019240 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:44:38.019215 4405 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ed86675a-731f-4625-bca5-02ba4200e7ad-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w2-a0bcbc-69fdc5f5c9-d4d6x\" (UID: \"ed86675a-731f-4625-bca5-02ba4200e7ad\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-a0bcbc-69fdc5f5c9-d4d6x" Nov 04 02:44:38.079591 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:44:38.079561 4405 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5f4mh\" (UniqueName: \"kubernetes.io/projected/ed86675a-731f-4625-bca5-02ba4200e7ad-kube-api-access-5f4mh\") pod \"payload-pre-processing-e2e-worker-w2-a0bcbc-69fdc5f5c9-d4d6x\" (UID: \"ed86675a-731f-4625-bca5-02ba4200e7ad\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-a0bcbc-69fdc5f5c9-d4d6x" Nov 04 02:44:38.184662 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:44:38.184571 4405 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-a0bcbc-69fdc5f5c9-d4d6x" Nov 04 02:44:38.558827 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:44:38.558757 4405 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w2-a0bcbc-69fdc5f5c9-d4d6x"] Nov 04 02:44:38.561603 ip-10-0-134-125 kubenswrapper[4405]: W1104 02:44:38.561573 4405 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poded86675a_731f_4625_bca5_02ba4200e7ad.slice/crio-24e4046d6a8c209ba6f6a2597803f1667bc4d006f21add2f7b11f9d3c4f3131b WatchSource:0}: Error finding container 24e4046d6a8c209ba6f6a2597803f1667bc4d006f21add2f7b11f9d3c4f3131b: Status 404 returned error can't find the container with id 24e4046d6a8c209ba6f6a2597803f1667bc4d006f21add2f7b11f9d3c4f3131b Nov 04 02:44:38.593342 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:44:38.593308 4405 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-a0bcbc-69fdc5f5c9-d4d6x" event={"ID":"ed86675a-731f-4625-bca5-02ba4200e7ad","Type":"ContainerStarted","Data":"24e4046d6a8c209ba6f6a2597803f1667bc4d006f21add2f7b11f9d3c4f3131b"} Nov 04 02:44:39.737472 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:44:39.737437 4405 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w3-cdcf67-556f77498f-fltxv"] Nov 04 02:44:39.742087 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:44:39.742069 4405 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-cdcf67-556f77498f-fltxv" Nov 04 02:44:39.835765 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:44:39.835734 4405 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b1d26a63-824f-40dd-9379-603c6f9d77e1-metrics-certs\") pod \"maas-api-e2e-worker-w3-cdcf67-556f77498f-fltxv\" (UID: \"b1d26a63-824f-40dd-9379-603c6f9d77e1\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-cdcf67-556f77498f-fltxv" Nov 04 02:44:39.835951 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:44:39.835773 4405 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b1d26a63-824f-40dd-9379-603c6f9d77e1-openshift-service-ca\") pod \"maas-api-e2e-worker-w3-cdcf67-556f77498f-fltxv\" (UID: \"b1d26a63-824f-40dd-9379-603c6f9d77e1\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-cdcf67-556f77498f-fltxv" Nov 04 02:44:39.835951 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:44:39.835790 4405 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r5l28\" (UniqueName: \"kubernetes.io/projected/b1d26a63-824f-40dd-9379-603c6f9d77e1-kube-api-access-r5l28\") pod \"maas-api-e2e-worker-w3-cdcf67-556f77498f-fltxv\" (UID: \"b1d26a63-824f-40dd-9379-603c6f9d77e1\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-cdcf67-556f77498f-fltxv" Nov 04 02:44:39.835951 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:44:39.835935 4405 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b1d26a63-824f-40dd-9379-603c6f9d77e1-maas-api-tls\") pod \"maas-api-e2e-worker-w3-cdcf67-556f77498f-fltxv\" (UID: \"b1d26a63-824f-40dd-9379-603c6f9d77e1\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-cdcf67-556f77498f-fltxv" Nov 04 02:44:39.847758 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:44:39.847622 4405 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 04 02:44:39.847758 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:44:39.847713 4405 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-w3-cdcf67-serving-cert\"" Nov 04 02:44:39.847758 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:44:39.847713 4405 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 04 02:44:39.849948 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:44:39.849926 4405 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w3-cdcf67-556f77498f-fltxv"] Nov 04 02:44:39.894182 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:44:39.894152 4405 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-qtjr2\"" Nov 04 02:44:39.894182 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:44:39.894152 4405 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\"" Nov 04 02:44:39.937198 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:44:39.937163 4405 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b1d26a63-824f-40dd-9379-603c6f9d77e1-metrics-certs\") pod \"maas-api-e2e-worker-w3-cdcf67-556f77498f-fltxv\" (UID: \"b1d26a63-824f-40dd-9379-603c6f9d77e1\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-cdcf67-556f77498f-fltxv" Nov 04 02:44:39.937198 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:44:39.937200 4405 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b1d26a63-824f-40dd-9379-603c6f9d77e1-openshift-service-ca\") pod \"maas-api-e2e-worker-w3-cdcf67-556f77498f-fltxv\" (UID: \"b1d26a63-824f-40dd-9379-603c6f9d77e1\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-cdcf67-556f77498f-fltxv" Nov 04 02:44:39.937388 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:44:39.937215 4405 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r5l28\" (UniqueName: \"kubernetes.io/projected/b1d26a63-824f-40dd-9379-603c6f9d77e1-kube-api-access-r5l28\") pod \"maas-api-e2e-worker-w3-cdcf67-556f77498f-fltxv\" (UID: \"b1d26a63-824f-40dd-9379-603c6f9d77e1\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-cdcf67-556f77498f-fltxv" Nov 04 02:44:39.937388 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:44:39.937263 4405 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b1d26a63-824f-40dd-9379-603c6f9d77e1-maas-api-tls\") pod \"maas-api-e2e-worker-w3-cdcf67-556f77498f-fltxv\" (UID: \"b1d26a63-824f-40dd-9379-603c6f9d77e1\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-cdcf67-556f77498f-fltxv" Nov 04 02:44:39.937478 ip-10-0-134-125 kubenswrapper[4405]: E1104 02:44:39.937389 4405 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-worker-w3-cdcf67-serving-cert: secret "maas-api-e2e-worker-w3-cdcf67-serving-cert" not found Nov 04 02:44:39.937514 ip-10-0-134-125 kubenswrapper[4405]: E1104 02:44:39.937500 4405 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b1d26a63-824f-40dd-9379-603c6f9d77e1-maas-api-tls podName:b1d26a63-824f-40dd-9379-603c6f9d77e1 nodeName:}" failed. No retries permitted until 2026-11-04 02:44:40.437469178 +0000 UTC m=+726.868169290 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/b1d26a63-824f-40dd-9379-603c6f9d77e1-maas-api-tls") pod "maas-api-e2e-worker-w3-cdcf67-556f77498f-fltxv" (UID: "b1d26a63-824f-40dd-9379-603c6f9d77e1") : secret "maas-api-e2e-worker-w3-cdcf67-serving-cert" not found Nov 04 02:44:39.937929 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:44:39.937910 4405 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b1d26a63-824f-40dd-9379-603c6f9d77e1-openshift-service-ca\") pod \"maas-api-e2e-worker-w3-cdcf67-556f77498f-fltxv\" (UID: \"b1d26a63-824f-40dd-9379-603c6f9d77e1\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-cdcf67-556f77498f-fltxv" Nov 04 02:44:39.939923 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:44:39.939901 4405 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b1d26a63-824f-40dd-9379-603c6f9d77e1-metrics-certs\") pod \"maas-api-e2e-worker-w3-cdcf67-556f77498f-fltxv\" (UID: \"b1d26a63-824f-40dd-9379-603c6f9d77e1\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-cdcf67-556f77498f-fltxv" Nov 04 02:44:40.003642 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:44:40.003563 4405 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r5l28\" (UniqueName: \"kubernetes.io/projected/b1d26a63-824f-40dd-9379-603c6f9d77e1-kube-api-access-r5l28\") pod \"maas-api-e2e-worker-w3-cdcf67-556f77498f-fltxv\" (UID: \"b1d26a63-824f-40dd-9379-603c6f9d77e1\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-cdcf67-556f77498f-fltxv" Nov 04 02:44:40.443906 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:44:40.443862 4405 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b1d26a63-824f-40dd-9379-603c6f9d77e1-maas-api-tls\") pod \"maas-api-e2e-worker-w3-cdcf67-556f77498f-fltxv\" (UID: \"b1d26a63-824f-40dd-9379-603c6f9d77e1\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-cdcf67-556f77498f-fltxv" Nov 04 02:44:40.446606 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:44:40.446573 4405 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b1d26a63-824f-40dd-9379-603c6f9d77e1-maas-api-tls\") pod \"maas-api-e2e-worker-w3-cdcf67-556f77498f-fltxv\" (UID: \"b1d26a63-824f-40dd-9379-603c6f9d77e1\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-cdcf67-556f77498f-fltxv" Nov 04 02:44:40.655512 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:44:40.655473 4405 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-cdcf67-556f77498f-fltxv" Nov 04 02:44:41.035703 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:44:41.035657 4405 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w3-cdcf67-556f77498f-fltxv"] Nov 04 02:44:41.037727 ip-10-0-134-125 kubenswrapper[4405]: W1104 02:44:41.037697 4405 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb1d26a63_824f_40dd_9379_603c6f9d77e1.slice/crio-d4096c1bde5c345cc89fe0a0d446fe60297b01e095afcd15bdd2f311a78cec7e WatchSource:0}: Error finding container d4096c1bde5c345cc89fe0a0d446fe60297b01e095afcd15bdd2f311a78cec7e: Status 404 returned error can't find the container with id d4096c1bde5c345cc89fe0a0d446fe60297b01e095afcd15bdd2f311a78cec7e Nov 04 02:44:41.609338 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:44:41.609297 4405 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-cdcf67-556f77498f-fltxv" event={"ID":"b1d26a63-824f-40dd-9379-603c6f9d77e1","Type":"ContainerStarted","Data":"d4096c1bde5c345cc89fe0a0d446fe60297b01e095afcd15bdd2f311a78cec7e"} Nov 04 02:44:42.617613 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:44:42.617507 4405 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-cdcf67-556f77498f-fltxv" event={"ID":"b1d26a63-824f-40dd-9379-603c6f9d77e1","Type":"ContainerStarted","Data":"2b5873d4155af77cf1362163aebf56cbcd8fc195fbeeb459466d0d40e482ac11"} Nov 04 02:44:42.617613 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:44:42.617586 4405 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-cdcf67-556f77498f-fltxv" Nov 04 02:44:42.619874 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:44:42.619845 4405 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-a0bcbc-69fdc5f5c9-d4d6x" event={"ID":"ed86675a-731f-4625-bca5-02ba4200e7ad","Type":"ContainerStarted","Data":"8da84aeee90456df8fa2e6415da3074463772dbec91eedcf12da9984dc29d885"} Nov 04 02:44:42.620147 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:44:42.620127 4405 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-a0bcbc-69fdc5f5c9-d4d6x" Nov 04 02:44:42.735089 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:44:42.735033 4405 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-cdcf67-556f77498f-fltxv" podStartSLOduration=2.677748892 podStartE2EDuration="3.73501893s" totalImagesPullingTime="1.057270038s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 02:44:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 02:44:42.734025635 +0000 UTC m=+729.164725758" watchObservedRunningTime="2026-11-04 02:44:42.73501893 +0000 UTC m=+729.165719050" Nov 04 02:44:42.852113 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:44:42.852056 4405 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-a0bcbc-69fdc5f5c9-d4d6x" podStartSLOduration=2.116404481 podStartE2EDuration="5.852041381s" totalImagesPullingTime="3.7356369s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 02:44:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 02:44:42.850433652 +0000 UTC m=+729.281133773" watchObservedRunningTime="2026-11-04 02:44:42.852041381 +0000 UTC m=+729.282741544" Nov 04 02:44:43.043449 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:44:43.043371 4405 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w3-cdcf67-76579b7df4-vqbkj"] Nov 04 02:44:43.047022 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:44:43.047006 4405 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w3-cdcf67-76579b7df4-vqbkj" Nov 04 02:44:43.099757 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:44:43.099726 4405 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w3-cdcf67\"" Nov 04 02:44:43.104806 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:44:43.104777 4405 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w3-cdcf67-76579b7df4-vqbkj"] Nov 04 02:44:43.155384 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:44:43.155353 4405 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w3-cdcf67-dockercfg-kklkb\"" Nov 04 02:44:43.175017 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:44:43.174985 4405 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d280cbd2-035b-422b-85c2-79a39d4a06cc-plugins-config-volume\") pod \"payload-processing-e2e-worker-w3-cdcf67-76579b7df4-vqbkj\" (UID: \"d280cbd2-035b-422b-85c2-79a39d4a06cc\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-cdcf67-76579b7df4-vqbkj" Nov 04 02:44:43.175179 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:44:43.175036 4405 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7lk5m\" (UniqueName: \"kubernetes.io/projected/d280cbd2-035b-422b-85c2-79a39d4a06cc-kube-api-access-7lk5m\") pod \"payload-processing-e2e-worker-w3-cdcf67-76579b7df4-vqbkj\" (UID: \"d280cbd2-035b-422b-85c2-79a39d4a06cc\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-cdcf67-76579b7df4-vqbkj" Nov 04 02:44:43.276441 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:44:43.276404 4405 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d280cbd2-035b-422b-85c2-79a39d4a06cc-plugins-config-volume\") pod \"payload-processing-e2e-worker-w3-cdcf67-76579b7df4-vqbkj\" (UID: \"d280cbd2-035b-422b-85c2-79a39d4a06cc\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-cdcf67-76579b7df4-vqbkj" Nov 04 02:44:43.276618 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:44:43.276458 4405 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7lk5m\" (UniqueName: \"kubernetes.io/projected/d280cbd2-035b-422b-85c2-79a39d4a06cc-kube-api-access-7lk5m\") pod \"payload-processing-e2e-worker-w3-cdcf67-76579b7df4-vqbkj\" (UID: \"d280cbd2-035b-422b-85c2-79a39d4a06cc\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-cdcf67-76579b7df4-vqbkj" Nov 04 02:44:43.277203 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:44:43.277184 4405 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d280cbd2-035b-422b-85c2-79a39d4a06cc-plugins-config-volume\") pod \"payload-processing-e2e-worker-w3-cdcf67-76579b7df4-vqbkj\" (UID: \"d280cbd2-035b-422b-85c2-79a39d4a06cc\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-cdcf67-76579b7df4-vqbkj" Nov 04 02:44:43.337593 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:44:43.337555 4405 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7lk5m\" (UniqueName: \"kubernetes.io/projected/d280cbd2-035b-422b-85c2-79a39d4a06cc-kube-api-access-7lk5m\") pod \"payload-processing-e2e-worker-w3-cdcf67-76579b7df4-vqbkj\" (UID: \"d280cbd2-035b-422b-85c2-79a39d4a06cc\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-cdcf67-76579b7df4-vqbkj" Nov 04 02:44:43.363633 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:44:43.363597 4405 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w3-cdcf67-76579b7df4-vqbkj" Nov 04 02:44:43.736032 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:44:43.736010 4405 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w3-cdcf67-76579b7df4-vqbkj"] Nov 04 02:44:43.737970 ip-10-0-134-125 kubenswrapper[4405]: W1104 02:44:43.737944 4405 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd280cbd2_035b_422b_85c2_79a39d4a06cc.slice/crio-552946d45d4c42ea9b8f51f4e9cfc6b2167d24be77b19e89ccab86330e0b2b16 WatchSource:0}: Error finding container 552946d45d4c42ea9b8f51f4e9cfc6b2167d24be77b19e89ccab86330e0b2b16: Status 404 returned error can't find the container with id 552946d45d4c42ea9b8f51f4e9cfc6b2167d24be77b19e89ccab86330e0b2b16 Nov 04 02:44:44.630171 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:44:44.630135 4405 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w3-cdcf67-76579b7df4-vqbkj" event={"ID":"d280cbd2-035b-422b-85c2-79a39d4a06cc","Type":"ContainerStarted","Data":"ba2c291bdac8dd46b4dbde415acaa0def7a9b987b27f52dcaff5dca346b06fc8"} Nov 04 02:44:44.630384 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:44:44.630185 4405 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w3-cdcf67-76579b7df4-vqbkj" event={"ID":"d280cbd2-035b-422b-85c2-79a39d4a06cc","Type":"ContainerStarted","Data":"552946d45d4c42ea9b8f51f4e9cfc6b2167d24be77b19e89ccab86330e0b2b16"} Nov 04 02:44:44.630384 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:44:44.630271 4405 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w3-cdcf67-76579b7df4-vqbkj" Nov 04 02:44:44.745950 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:44:44.745897 4405 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w3-cdcf67-76579b7df4-vqbkj" podStartSLOduration=1.745876169 podStartE2EDuration="1.745876169s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 02:44:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 02:44:44.744494661 +0000 UTC m=+731.175194793" watchObservedRunningTime="2026-11-04 02:44:44.745876169 +0000 UTC m=+731.176576312" Nov 04 02:44:47.927625 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:44:47.927590 4405 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-fe4e0c-7dcb6cb4dd-vdb4p"] Nov 04 02:44:47.931370 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:44:47.931353 4405 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w0-fe4e0c-7dcb6cb4dd-vdb4p" Nov 04 02:44:47.995640 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:44:47.995607 4405 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-fe4e0c-7dcb6cb4dd-vdb4p"] Nov 04 02:44:48.034572 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:44:48.027430 4405 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7d5dc4d7-ea24-4845-8087-1e999e739f22-plugins-config-volume\") pod \"payload-processing-e2e-worker-w0-fe4e0c-7dcb6cb4dd-vdb4p\" (UID: \"7d5dc4d7-ea24-4845-8087-1e999e739f22\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-fe4e0c-7dcb6cb4dd-vdb4p" Nov 04 02:44:48.034572 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:44:48.027637 4405 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z7hvd\" (UniqueName: \"kubernetes.io/projected/7d5dc4d7-ea24-4845-8087-1e999e739f22-kube-api-access-z7hvd\") pod \"payload-processing-e2e-worker-w0-fe4e0c-7dcb6cb4dd-vdb4p\" (UID: \"7d5dc4d7-ea24-4845-8087-1e999e739f22\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-fe4e0c-7dcb6cb4dd-vdb4p" Nov 04 02:44:48.034572 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:44:48.033346 4405 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w0-fe4e0c\"" Nov 04 02:44:48.036530 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:44:48.036492 4405 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w0-fe4e0c-dockercfg-rjf74\"" Nov 04 02:44:48.128716 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:44:48.128657 4405 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z7hvd\" (UniqueName: \"kubernetes.io/projected/7d5dc4d7-ea24-4845-8087-1e999e739f22-kube-api-access-z7hvd\") pod \"payload-processing-e2e-worker-w0-fe4e0c-7dcb6cb4dd-vdb4p\" (UID: \"7d5dc4d7-ea24-4845-8087-1e999e739f22\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-fe4e0c-7dcb6cb4dd-vdb4p" Nov 04 02:44:48.128924 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:44:48.128901 4405 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7d5dc4d7-ea24-4845-8087-1e999e739f22-plugins-config-volume\") pod \"payload-processing-e2e-worker-w0-fe4e0c-7dcb6cb4dd-vdb4p\" (UID: \"7d5dc4d7-ea24-4845-8087-1e999e739f22\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-fe4e0c-7dcb6cb4dd-vdb4p" Nov 04 02:44:48.130011 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:44:48.129982 4405 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7d5dc4d7-ea24-4845-8087-1e999e739f22-plugins-config-volume\") pod \"payload-processing-e2e-worker-w0-fe4e0c-7dcb6cb4dd-vdb4p\" (UID: \"7d5dc4d7-ea24-4845-8087-1e999e739f22\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-fe4e0c-7dcb6cb4dd-vdb4p" Nov 04 02:44:48.191308 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:44:48.191235 4405 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z7hvd\" (UniqueName: \"kubernetes.io/projected/7d5dc4d7-ea24-4845-8087-1e999e739f22-kube-api-access-z7hvd\") pod \"payload-processing-e2e-worker-w0-fe4e0c-7dcb6cb4dd-vdb4p\" (UID: \"7d5dc4d7-ea24-4845-8087-1e999e739f22\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-fe4e0c-7dcb6cb4dd-vdb4p" Nov 04 02:44:48.225497 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:44:48.225452 4405 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-bl5mg"] Nov 04 02:44:48.244849 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:44:48.244811 4405 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w0-fe4e0c-7dcb6cb4dd-vdb4p" Nov 04 02:44:48.617877 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:44:48.617850 4405 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-fe4e0c-7dcb6cb4dd-vdb4p"] Nov 04 02:44:48.619985 ip-10-0-134-125 kubenswrapper[4405]: W1104 02:44:48.619956 4405 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7d5dc4d7_ea24_4845_8087_1e999e739f22.slice/crio-54978bcace432a3a1b05f4957dd709d76e5e460732528339ac7eb53fb75a2c54 WatchSource:0}: Error finding container 54978bcace432a3a1b05f4957dd709d76e5e460732528339ac7eb53fb75a2c54: Status 404 returned error can't find the container with id 54978bcace432a3a1b05f4957dd709d76e5e460732528339ac7eb53fb75a2c54 Nov 04 02:44:48.629473 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:44:48.629453 4405 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-cdcf67-556f77498f-fltxv" Nov 04 02:44:48.649583 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:44:48.649557 4405 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w0-fe4e0c-7dcb6cb4dd-vdb4p" event={"ID":"7d5dc4d7-ea24-4845-8087-1e999e739f22","Type":"ContainerStarted","Data":"54978bcace432a3a1b05f4957dd709d76e5e460732528339ac7eb53fb75a2c54"} Nov 04 02:44:49.656518 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:44:49.656480 4405 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w0-fe4e0c-7dcb6cb4dd-vdb4p" event={"ID":"7d5dc4d7-ea24-4845-8087-1e999e739f22","Type":"ContainerStarted","Data":"3af3e50749525b71bf5ef60f63d83cb4e87edec65450942627b6648b148a7c6e"} Nov 04 02:44:49.657065 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:44:49.656607 4405 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w0-fe4e0c-7dcb6cb4dd-vdb4p" Nov 04 02:44:49.787358 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:44:49.787307 4405 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w0-fe4e0c-7dcb6cb4dd-vdb4p" podStartSLOduration=2.78729286 podStartE2EDuration="2.78729286s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 02:44:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 02:44:49.785797619 +0000 UTC m=+736.216497740" watchObservedRunningTime="2026-11-04 02:44:49.78729286 +0000 UTC m=+736.217992982" Nov 04 02:44:53.627005 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:44:53.626973 4405 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-a0bcbc-69fdc5f5c9-d4d6x" Nov 04 02:44:55.638100 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:44:55.638057 4405 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w3-cdcf67-76579b7df4-vqbkj" Nov 04 02:44:55.741913 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:44:55.741880 4405 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-bl5mg"] Nov 04 02:44:55.744140 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:44:55.744110 4405 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w2-a0bcbc-5b45966b8d-dgchr"] Nov 04 02:44:55.747834 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:44:55.747806 4405 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-a0bcbc-5b45966b8d-dgchr" Nov 04 02:44:55.792703 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:44:55.792629 4405 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w2-a0bcbc-5b45966b8d-dgchr"] Nov 04 02:44:55.799985 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:44:55.799936 4405 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-w2-a0bcbc-serving-cert\"" Nov 04 02:44:55.833525 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:44:55.833489 4405 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/44902e0e-53e2-443d-8e78-881319ea6fef-maas-api-tls\") pod \"maas-api-e2e-worker-w2-a0bcbc-5b45966b8d-dgchr\" (UID: \"44902e0e-53e2-443d-8e78-881319ea6fef\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-a0bcbc-5b45966b8d-dgchr" Nov 04 02:44:55.833750 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:44:55.833687 4405 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/44902e0e-53e2-443d-8e78-881319ea6fef-metrics-certs\") pod \"maas-api-e2e-worker-w2-a0bcbc-5b45966b8d-dgchr\" (UID: \"44902e0e-53e2-443d-8e78-881319ea6fef\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-a0bcbc-5b45966b8d-dgchr" Nov 04 02:44:55.833854 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:44:55.833834 4405 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tvhcv\" (UniqueName: \"kubernetes.io/projected/44902e0e-53e2-443d-8e78-881319ea6fef-kube-api-access-tvhcv\") pod \"maas-api-e2e-worker-w2-a0bcbc-5b45966b8d-dgchr\" (UID: \"44902e0e-53e2-443d-8e78-881319ea6fef\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-a0bcbc-5b45966b8d-dgchr" Nov 04 02:44:55.833912 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:44:55.833881 4405 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/44902e0e-53e2-443d-8e78-881319ea6fef-openshift-service-ca\") pod \"maas-api-e2e-worker-w2-a0bcbc-5b45966b8d-dgchr\" (UID: \"44902e0e-53e2-443d-8e78-881319ea6fef\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-a0bcbc-5b45966b8d-dgchr" Nov 04 02:44:55.935603 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:44:55.935503 4405 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/44902e0e-53e2-443d-8e78-881319ea6fef-maas-api-tls\") pod \"maas-api-e2e-worker-w2-a0bcbc-5b45966b8d-dgchr\" (UID: \"44902e0e-53e2-443d-8e78-881319ea6fef\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-a0bcbc-5b45966b8d-dgchr" Nov 04 02:44:55.935603 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:44:55.935602 4405 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/44902e0e-53e2-443d-8e78-881319ea6fef-metrics-certs\") pod \"maas-api-e2e-worker-w2-a0bcbc-5b45966b8d-dgchr\" (UID: \"44902e0e-53e2-443d-8e78-881319ea6fef\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-a0bcbc-5b45966b8d-dgchr" Nov 04 02:44:55.935850 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:44:55.935697 4405 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tvhcv\" (UniqueName: \"kubernetes.io/projected/44902e0e-53e2-443d-8e78-881319ea6fef-kube-api-access-tvhcv\") pod \"maas-api-e2e-worker-w2-a0bcbc-5b45966b8d-dgchr\" (UID: \"44902e0e-53e2-443d-8e78-881319ea6fef\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-a0bcbc-5b45966b8d-dgchr" Nov 04 02:44:55.935850 ip-10-0-134-125 kubenswrapper[4405]: E1104 02:44:55.935724 4405 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-worker-w2-a0bcbc-serving-cert: secret "maas-api-e2e-worker-w2-a0bcbc-serving-cert" not found Nov 04 02:44:55.935850 ip-10-0-134-125 kubenswrapper[4405]: E1104 02:44:55.935809 4405 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/44902e0e-53e2-443d-8e78-881319ea6fef-maas-api-tls podName:44902e0e-53e2-443d-8e78-881319ea6fef nodeName:}" failed. No retries permitted until 2026-11-04 02:44:56.43578771 +0000 UTC m=+742.866487822 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/44902e0e-53e2-443d-8e78-881319ea6fef-maas-api-tls") pod "maas-api-e2e-worker-w2-a0bcbc-5b45966b8d-dgchr" (UID: "44902e0e-53e2-443d-8e78-881319ea6fef") : secret "maas-api-e2e-worker-w2-a0bcbc-serving-cert" not found Nov 04 02:44:55.935850 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:44:55.935733 4405 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/44902e0e-53e2-443d-8e78-881319ea6fef-openshift-service-ca\") pod \"maas-api-e2e-worker-w2-a0bcbc-5b45966b8d-dgchr\" (UID: \"44902e0e-53e2-443d-8e78-881319ea6fef\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-a0bcbc-5b45966b8d-dgchr" Nov 04 02:44:55.936414 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:44:55.936396 4405 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/44902e0e-53e2-443d-8e78-881319ea6fef-openshift-service-ca\") pod \"maas-api-e2e-worker-w2-a0bcbc-5b45966b8d-dgchr\" (UID: \"44902e0e-53e2-443d-8e78-881319ea6fef\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-a0bcbc-5b45966b8d-dgchr" Nov 04 02:44:55.939093 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:44:55.939068 4405 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/44902e0e-53e2-443d-8e78-881319ea6fef-metrics-certs\") pod \"maas-api-e2e-worker-w2-a0bcbc-5b45966b8d-dgchr\" (UID: \"44902e0e-53e2-443d-8e78-881319ea6fef\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-a0bcbc-5b45966b8d-dgchr" Nov 04 02:44:55.997018 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:44:55.996984 4405 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tvhcv\" (UniqueName: \"kubernetes.io/projected/44902e0e-53e2-443d-8e78-881319ea6fef-kube-api-access-tvhcv\") pod \"maas-api-e2e-worker-w2-a0bcbc-5b45966b8d-dgchr\" (UID: \"44902e0e-53e2-443d-8e78-881319ea6fef\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-a0bcbc-5b45966b8d-dgchr" Nov 04 02:44:56.440881 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:44:56.440841 4405 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/44902e0e-53e2-443d-8e78-881319ea6fef-maas-api-tls\") pod \"maas-api-e2e-worker-w2-a0bcbc-5b45966b8d-dgchr\" (UID: \"44902e0e-53e2-443d-8e78-881319ea6fef\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-a0bcbc-5b45966b8d-dgchr" Nov 04 02:44:56.443532 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:44:56.443499 4405 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/44902e0e-53e2-443d-8e78-881319ea6fef-maas-api-tls\") pod \"maas-api-e2e-worker-w2-a0bcbc-5b45966b8d-dgchr\" (UID: \"44902e0e-53e2-443d-8e78-881319ea6fef\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-a0bcbc-5b45966b8d-dgchr" Nov 04 02:44:56.663922 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:44:56.663886 4405 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-a0bcbc-5b45966b8d-dgchr" Nov 04 02:44:57.038273 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:44:57.038248 4405 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w2-a0bcbc-5b45966b8d-dgchr"] Nov 04 02:44:57.041246 ip-10-0-134-125 kubenswrapper[4405]: W1104 02:44:57.041216 4405 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod44902e0e_53e2_443d_8e78_881319ea6fef.slice/crio-6526e97b42cb7f9a4031dfc3f520afd50445b2f1ae411a27d7175712c2b321e3 WatchSource:0}: Error finding container 6526e97b42cb7f9a4031dfc3f520afd50445b2f1ae411a27d7175712c2b321e3: Status 404 returned error can't find the container with id 6526e97b42cb7f9a4031dfc3f520afd50445b2f1ae411a27d7175712c2b321e3 Nov 04 02:44:57.700749 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:44:57.700713 4405 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-a0bcbc-5b45966b8d-dgchr" event={"ID":"44902e0e-53e2-443d-8e78-881319ea6fef","Type":"ContainerStarted","Data":"6526e97b42cb7f9a4031dfc3f520afd50445b2f1ae411a27d7175712c2b321e3"} Nov 04 02:44:58.707209 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:44:58.707175 4405 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-a0bcbc-5b45966b8d-dgchr" event={"ID":"44902e0e-53e2-443d-8e78-881319ea6fef","Type":"ContainerStarted","Data":"d85309cb93ec8c62aeab9ca9c8b11a86d8098192bc207876944852bcd7572204"} Nov 04 02:44:58.707749 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:44:58.707301 4405 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-a0bcbc-5b45966b8d-dgchr" Nov 04 02:44:58.708971 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:44:58.708949 4405 generic.go:356] "Generic (PLEG): container finished" podID="dad304e5-de0a-44b2-a137-8139a2aa2f52" containerID="443defcf2b22d24c59168e7d581684bb84006228c2e217384d9131e127596c20" exitCode=0 Nov 04 02:44:58.709089 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:44:58.709039 4405 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-cdcf67-openshift-default-6ff8f5cbc7-rwgvq" event={"ID":"dad304e5-de0a-44b2-a137-8139a2aa2f52","Type":"ContainerDied","Data":"443defcf2b22d24c59168e7d581684bb84006228c2e217384d9131e127596c20"} Nov 04 02:44:58.709391 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:44:58.709373 4405 scope.go:122] "RemoveContainer" containerID="443defcf2b22d24c59168e7d581684bb84006228c2e217384d9131e127596c20" Nov 04 02:44:58.710570 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:44:58.710546 4405 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 04 02:44:58.710631 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:44:58.710612 4405 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 04 02:44:58.710680 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:44:58.710642 4405 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 04 02:44:58.825128 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:44:58.825065 4405 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-a0bcbc-5b45966b8d-dgchr" podStartSLOduration=2.925007215 podStartE2EDuration="3.825045841s" totalImagesPullingTime="900.038626ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 02:44:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 02:44:58.824105135 +0000 UTC m=+745.254805257" watchObservedRunningTime="2026-11-04 02:44:58.825045841 +0000 UTC m=+745.255745963" Nov 04 02:44:59.723845 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:44:59.723809 4405 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-cdcf67-openshift-default-6ff8f5cbc7-rwgvq" event={"ID":"dad304e5-de0a-44b2-a137-8139a2aa2f52","Type":"ContainerStarted","Data":"6a196d872afe8c00e1ad270f8064013e8a2638c5c7797c9e5c2a69524f602fff"} Nov 04 02:45:00.179056 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:45:00.179009 4405 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29896005-9vnjz"] Nov 04 02:45:00.183948 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:45:00.183919 4405 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29896005-9vnjz" Nov 04 02:45:00.237017 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:45:00.236975 4405 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-slbq8\"" Nov 04 02:45:00.276496 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:45:00.276465 4405 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29896005-9vnjz"] Nov 04 02:45:00.285490 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:45:00.282606 4405 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r97sh\" (UniqueName: \"kubernetes.io/projected/62629b7a-09c4-4fd8-bbab-ca11ecd6c696-kube-api-access-r97sh\") pod \"maas-api-key-cleanup-29896005-9vnjz\" (UID: \"62629b7a-09c4-4fd8-bbab-ca11ecd6c696\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29896005-9vnjz" Nov 04 02:45:00.285490 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:45:00.282717 4405 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/62629b7a-09c4-4fd8-bbab-ca11ecd6c696-maas-api-service-ca\") pod \"maas-api-key-cleanup-29896005-9vnjz\" (UID: \"62629b7a-09c4-4fd8-bbab-ca11ecd6c696\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29896005-9vnjz" Nov 04 02:45:00.383527 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:45:00.383490 4405 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r97sh\" (UniqueName: \"kubernetes.io/projected/62629b7a-09c4-4fd8-bbab-ca11ecd6c696-kube-api-access-r97sh\") pod \"maas-api-key-cleanup-29896005-9vnjz\" (UID: \"62629b7a-09c4-4fd8-bbab-ca11ecd6c696\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29896005-9vnjz" Nov 04 02:45:00.383731 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:45:00.383573 4405 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/62629b7a-09c4-4fd8-bbab-ca11ecd6c696-maas-api-service-ca\") pod \"maas-api-key-cleanup-29896005-9vnjz\" (UID: \"62629b7a-09c4-4fd8-bbab-ca11ecd6c696\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29896005-9vnjz" Nov 04 02:45:00.384454 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:45:00.384425 4405 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/62629b7a-09c4-4fd8-bbab-ca11ecd6c696-maas-api-service-ca\") pod \"maas-api-key-cleanup-29896005-9vnjz\" (UID: \"62629b7a-09c4-4fd8-bbab-ca11ecd6c696\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29896005-9vnjz" Nov 04 02:45:00.446490 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:45:00.446072 4405 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w6-ebb7fd-b8fd547bb-7jkvh"] Nov 04 02:45:00.452479 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:45:00.451010 4405 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w6-ebb7fd-b8fd547bb-7jkvh" Nov 04 02:45:00.457756 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:45:00.456381 4405 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r97sh\" (UniqueName: \"kubernetes.io/projected/62629b7a-09c4-4fd8-bbab-ca11ecd6c696-kube-api-access-r97sh\") pod \"maas-api-key-cleanup-29896005-9vnjz\" (UID: \"62629b7a-09c4-4fd8-bbab-ca11ecd6c696\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29896005-9vnjz" Nov 04 02:45:00.510701 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:45:00.507011 4405 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w6-ebb7fd\"" Nov 04 02:45:00.510701 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:45:00.510410 4405 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w6-ebb7fd-b8fd547bb-7jkvh"] Nov 04 02:45:00.520192 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:45:00.520164 4405 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w6-ebb7fd-dockercfg-6pcjs\"" Nov 04 02:45:00.535450 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:45:00.534613 4405 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29896005-9vnjz" Nov 04 02:45:00.585691 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:45:00.585387 4405 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dcgv6\" (UniqueName: \"kubernetes.io/projected/af244918-878e-430e-a893-aac69d2498ef-kube-api-access-dcgv6\") pod \"payload-processing-e2e-worker-w6-ebb7fd-b8fd547bb-7jkvh\" (UID: \"af244918-878e-430e-a893-aac69d2498ef\") " pod="openshift-ingress/payload-processing-e2e-worker-w6-ebb7fd-b8fd547bb-7jkvh" Nov 04 02:45:00.585691 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:45:00.585444 4405 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/af244918-878e-430e-a893-aac69d2498ef-plugins-config-volume\") pod \"payload-processing-e2e-worker-w6-ebb7fd-b8fd547bb-7jkvh\" (UID: \"af244918-878e-430e-a893-aac69d2498ef\") " pod="openshift-ingress/payload-processing-e2e-worker-w6-ebb7fd-b8fd547bb-7jkvh" Nov 04 02:45:00.613694 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:45:00.611956 4405 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-cdcf67-openshift-default-6ff8f5cbc7-rwgvq" Nov 04 02:45:00.613694 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:45:00.612084 4405 patch_prober.go:29] interesting pod/e2e-worker-w3-cdcf67-openshift-default-6ff8f5cbc7-rwgvq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.34:15021/healthz/ready\": dial tcp 10.134.0.34:15021: connect: connection refused" start-of-body= Nov 04 02:45:00.613694 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:45:00.612126 4405 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-cdcf67-openshift-default-6ff8f5cbc7-rwgvq" podUID="dad304e5-de0a-44b2-a137-8139a2aa2f52" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.34:15021/healthz/ready\": dial tcp 10.134.0.34:15021: connect: connection refused" Nov 04 02:45:00.665225 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:45:00.665193 4405 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w0-fe4e0c-7dcb6cb4dd-vdb4p" Nov 04 02:45:00.686645 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:45:00.686615 4405 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dcgv6\" (UniqueName: \"kubernetes.io/projected/af244918-878e-430e-a893-aac69d2498ef-kube-api-access-dcgv6\") pod \"payload-processing-e2e-worker-w6-ebb7fd-b8fd547bb-7jkvh\" (UID: \"af244918-878e-430e-a893-aac69d2498ef\") " pod="openshift-ingress/payload-processing-e2e-worker-w6-ebb7fd-b8fd547bb-7jkvh" Nov 04 02:45:00.686824 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:45:00.686683 4405 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/af244918-878e-430e-a893-aac69d2498ef-plugins-config-volume\") pod \"payload-processing-e2e-worker-w6-ebb7fd-b8fd547bb-7jkvh\" (UID: \"af244918-878e-430e-a893-aac69d2498ef\") " pod="openshift-ingress/payload-processing-e2e-worker-w6-ebb7fd-b8fd547bb-7jkvh" Nov 04 02:45:00.687414 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:45:00.687396 4405 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/af244918-878e-430e-a893-aac69d2498ef-plugins-config-volume\") pod \"payload-processing-e2e-worker-w6-ebb7fd-b8fd547bb-7jkvh\" (UID: \"af244918-878e-430e-a893-aac69d2498ef\") " pod="openshift-ingress/payload-processing-e2e-worker-w6-ebb7fd-b8fd547bb-7jkvh" Nov 04 02:45:00.748345 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:45:00.748270 4405 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dcgv6\" (UniqueName: \"kubernetes.io/projected/af244918-878e-430e-a893-aac69d2498ef-kube-api-access-dcgv6\") pod \"payload-processing-e2e-worker-w6-ebb7fd-b8fd547bb-7jkvh\" (UID: \"af244918-878e-430e-a893-aac69d2498ef\") " pod="openshift-ingress/payload-processing-e2e-worker-w6-ebb7fd-b8fd547bb-7jkvh" Nov 04 02:45:00.785551 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:45:00.785521 4405 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w6-ebb7fd-b8fd547bb-7jkvh" Nov 04 02:45:00.951003 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:45:00.950827 4405 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29896005-9vnjz"] Nov 04 02:45:00.953605 ip-10-0-134-125 kubenswrapper[4405]: W1104 02:45:00.953575 4405 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod62629b7a_09c4_4fd8_bbab_ca11ecd6c696.slice/crio-fe3410900d98147211f3f565c7e9652a2049d83c23efb2f2ad2aa05e195ea8ee WatchSource:0}: Error finding container fe3410900d98147211f3f565c7e9652a2049d83c23efb2f2ad2aa05e195ea8ee: Status 404 returned error can't find the container with id fe3410900d98147211f3f565c7e9652a2049d83c23efb2f2ad2aa05e195ea8ee Nov 04 02:45:01.222368 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:45:01.222327 4405 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w6-ebb7fd-b8fd547bb-7jkvh"] Nov 04 02:45:01.607461 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:45:01.607420 4405 patch_prober.go:29] interesting pod/e2e-worker-w3-cdcf67-openshift-default-6ff8f5cbc7-rwgvq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.34:15021/healthz/ready\": dial tcp 10.134.0.34:15021: connect: connection refused" start-of-body= Nov 04 02:45:01.607784 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:45:01.607753 4405 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-cdcf67-openshift-default-6ff8f5cbc7-rwgvq" podUID="dad304e5-de0a-44b2-a137-8139a2aa2f52" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.34:15021/healthz/ready\": dial tcp 10.134.0.34:15021: connect: connection refused" Nov 04 02:45:01.738019 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:45:01.737984 4405 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w6-ebb7fd-b8fd547bb-7jkvh" event={"ID":"af244918-878e-430e-a893-aac69d2498ef","Type":"ContainerStarted","Data":"ac88d64eb459108f07130e51fce895960a73ba7d8556f333dd1abf4501429d23"} Nov 04 02:45:01.738261 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:45:01.738046 4405 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w6-ebb7fd-b8fd547bb-7jkvh" event={"ID":"af244918-878e-430e-a893-aac69d2498ef","Type":"ContainerStarted","Data":"dc26c9356ad761332dcdcfefe4001f0ac58707c719d4b5187b8e00b88dfcc214"} Nov 04 02:45:01.738261 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:45:01.738104 4405 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w6-ebb7fd-b8fd547bb-7jkvh" Nov 04 02:45:01.739279 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:45:01.739256 4405 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29896005-9vnjz" event={"ID":"62629b7a-09c4-4fd8-bbab-ca11ecd6c696","Type":"ContainerStarted","Data":"fe3410900d98147211f3f565c7e9652a2049d83c23efb2f2ad2aa05e195ea8ee"} Nov 04 02:45:01.854990 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:45:01.854921 4405 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w6-ebb7fd-b8fd547bb-7jkvh" podStartSLOduration=1.8549063289999999 podStartE2EDuration="1.854906329s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 02:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 02:45:01.852833859 +0000 UTC m=+748.283533992" watchObservedRunningTime="2026-11-04 02:45:01.854906329 +0000 UTC m=+748.285606449" Nov 04 02:45:02.564298 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:45:02.564259 4405 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-cdcf67-openshift-default-6ff8f5cbc7-rwgvq" Nov 04 02:45:02.597173 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:45:02.597133 4405 patch_prober.go:29] interesting pod/e2e-worker-w3-cdcf67-openshift-default-6ff8f5cbc7-rwgvq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.34:15021/healthz/ready\": dial tcp 10.134.0.34:15021: connect: connection refused" start-of-body= Nov 04 02:45:02.597481 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:45:02.597449 4405 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-cdcf67-openshift-default-6ff8f5cbc7-rwgvq" podUID="dad304e5-de0a-44b2-a137-8139a2aa2f52" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.34:15021/healthz/ready\": dial tcp 10.134.0.34:15021: connect: connection refused" Nov 04 02:45:03.601146 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:45:03.601105 4405 patch_prober.go:29] interesting pod/e2e-worker-w3-cdcf67-openshift-default-6ff8f5cbc7-rwgvq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.34:15021/healthz/ready\": dial tcp 10.134.0.34:15021: connect: connection refused" start-of-body= Nov 04 02:45:03.601796 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:45:03.601745 4405 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-cdcf67-openshift-default-6ff8f5cbc7-rwgvq" podUID="dad304e5-de0a-44b2-a137-8139a2aa2f52" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.34:15021/healthz/ready\": dial tcp 10.134.0.34:15021: connect: connection refused" Nov 04 02:45:04.602628 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:45:04.602133 4405 patch_prober.go:29] interesting pod/e2e-worker-w3-cdcf67-openshift-default-6ff8f5cbc7-rwgvq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.34:15021/healthz/ready\": dial tcp 10.134.0.34:15021: connect: connection refused" start-of-body= Nov 04 02:45:04.602628 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:45:04.602195 4405 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-cdcf67-openshift-default-6ff8f5cbc7-rwgvq" podUID="dad304e5-de0a-44b2-a137-8139a2aa2f52" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.34:15021/healthz/ready\": dial tcp 10.134.0.34:15021: connect: connection refused" Nov 04 02:45:04.732347 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:45:04.732317 4405 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-a0bcbc-5b45966b8d-dgchr" Nov 04 02:45:05.600775 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:45:05.600745 4405 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-cdcf67-openshift-default-6ff8f5cbc7-rwgvq" Nov 04 02:45:05.601768 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:45:05.601749 4405 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-cdcf67-openshift-default-6ff8f5cbc7-rwgvq" Nov 04 02:45:06.771705 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:45:06.771594 4405 generic.go:356] "Generic (PLEG): container finished" podID="62629b7a-09c4-4fd8-bbab-ca11ecd6c696" containerID="21ac11ca838e434856247d531d3f32c5020aae71451724e78c71a2bd1cd364b3" exitCode=60 Nov 04 02:45:06.771705 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:45:06.771637 4405 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29896005-9vnjz" event={"ID":"62629b7a-09c4-4fd8-bbab-ca11ecd6c696","Type":"ContainerDied","Data":"21ac11ca838e434856247d531d3f32c5020aae71451724e78c71a2bd1cd364b3"} Nov 04 02:45:06.772243 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:45:06.771971 4405 scope.go:122] "RemoveContainer" containerID="21ac11ca838e434856247d531d3f32c5020aae71451724e78c71a2bd1cd364b3" Nov 04 02:45:06.883425 ip-10-0-134-125 kubenswrapper[4405]: E1104 02:45:06.883392 4405 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5f2vb" containerName="extractor" Nov 04 02:45:07.778760 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:45:07.778722 4405 generic.go:356] "Generic (PLEG): container finished" podID="62629b7a-09c4-4fd8-bbab-ca11ecd6c696" containerID="f08be2b8ca935960672c2e9fb4f1c506a0e5feba3e1fa0c57fbed0554b235ace" exitCode=60 Nov 04 02:45:07.779221 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:45:07.778804 4405 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29896005-9vnjz" event={"ID":"62629b7a-09c4-4fd8-bbab-ca11ecd6c696","Type":"ContainerDied","Data":"f08be2b8ca935960672c2e9fb4f1c506a0e5feba3e1fa0c57fbed0554b235ace"} Nov 04 02:45:07.779221 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:45:07.778859 4405 scope.go:122] "RemoveContainer" containerID="21ac11ca838e434856247d531d3f32c5020aae71451724e78c71a2bd1cd364b3" Nov 04 02:45:07.779221 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:45:07.779122 4405 scope.go:122] "RemoveContainer" containerID="f08be2b8ca935960672c2e9fb4f1c506a0e5feba3e1fa0c57fbed0554b235ace" Nov 04 02:45:07.779442 ip-10-0-134-125 kubenswrapper[4405]: E1104 02:45:07.779420 4405 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29896005-9vnjz_odh-ai-gateway-infra(62629b7a-09c4-4fd8-bbab-ca11ecd6c696)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29896005-9vnjz" podUID="62629b7a-09c4-4fd8-bbab-ca11ecd6c696" Nov 04 02:45:08.785194 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:45:08.785162 4405 scope.go:122] "RemoveContainer" containerID="f08be2b8ca935960672c2e9fb4f1c506a0e5feba3e1fa0c57fbed0554b235ace" Nov 04 02:45:08.785687 ip-10-0-134-125 kubenswrapper[4405]: E1104 02:45:08.785473 4405 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29896005-9vnjz_odh-ai-gateway-infra(62629b7a-09c4-4fd8-bbab-ca11ecd6c696)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29896005-9vnjz" podUID="62629b7a-09c4-4fd8-bbab-ca11ecd6c696" Nov 04 02:45:11.335663 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:45:11.335629 4405 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-bl5mg"] Nov 04 02:45:12.748844 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:45:12.748815 4405 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w6-ebb7fd-b8fd547bb-7jkvh" Nov 04 02:45:19.612704 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:45:19.611806 4405 patch_prober.go:29] interesting pod/e2e-worker-w5-3b7476-openshift-default-59975457bf-msl8m container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.36:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 04 02:45:19.612704 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:45:19.611891 4405 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w5-3b7476-openshift-default-59975457bf-msl8m" podUID="8a0008f3-4ab9-4ceb-8feb-0f536776610a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.36:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 04 02:45:19.694430 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:45:19.694399 4405 scope.go:122] "RemoveContainer" containerID="f08be2b8ca935960672c2e9fb4f1c506a0e5feba3e1fa0c57fbed0554b235ace" Nov 04 02:45:20.860381 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:45:20.860346 4405 generic.go:356] "Generic (PLEG): container finished" podID="8a0008f3-4ab9-4ceb-8feb-0f536776610a" containerID="b6a2edf16fc64d8d3cfeceaa4932a3db1cdd4fcbfee7de8866224fa041a79b0b" exitCode=0 Nov 04 02:45:20.860848 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:45:20.860429 4405 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-3b7476-openshift-default-59975457bf-msl8m" event={"ID":"8a0008f3-4ab9-4ceb-8feb-0f536776610a","Type":"ContainerDied","Data":"b6a2edf16fc64d8d3cfeceaa4932a3db1cdd4fcbfee7de8866224fa041a79b0b"} Nov 04 02:45:20.860927 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:45:20.860909 4405 scope.go:122] "RemoveContainer" containerID="b6a2edf16fc64d8d3cfeceaa4932a3db1cdd4fcbfee7de8866224fa041a79b0b" Nov 04 02:45:20.862305 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:45:20.862278 4405 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 04 02:45:20.862409 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:45:20.862345 4405 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 04 02:45:20.862409 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:45:20.862382 4405 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 04 02:45:20.862487 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:45:20.862452 4405 generic.go:356] "Generic (PLEG): container finished" podID="62629b7a-09c4-4fd8-bbab-ca11ecd6c696" containerID="9379545f2370b805be89e6aca980816e2db043c900e3f352d1266044880b035a" exitCode=60 Nov 04 02:45:20.862532 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:45:20.862518 4405 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29896005-9vnjz" event={"ID":"62629b7a-09c4-4fd8-bbab-ca11ecd6c696","Type":"ContainerDied","Data":"9379545f2370b805be89e6aca980816e2db043c900e3f352d1266044880b035a"} Nov 04 02:45:20.862588 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:45:20.862564 4405 scope.go:122] "RemoveContainer" containerID="f08be2b8ca935960672c2e9fb4f1c506a0e5feba3e1fa0c57fbed0554b235ace" Nov 04 02:45:20.862904 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:45:20.862888 4405 scope.go:122] "RemoveContainer" containerID="9379545f2370b805be89e6aca980816e2db043c900e3f352d1266044880b035a" Nov 04 02:45:20.863130 ip-10-0-134-125 kubenswrapper[4405]: E1104 02:45:20.863104 4405 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-29896005-9vnjz_odh-ai-gateway-infra(62629b7a-09c4-4fd8-bbab-ca11ecd6c696)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29896005-9vnjz" podUID="62629b7a-09c4-4fd8-bbab-ca11ecd6c696" Nov 04 02:45:21.870588 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:45:21.870550 4405 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-3b7476-openshift-default-59975457bf-msl8m" event={"ID":"8a0008f3-4ab9-4ceb-8feb-0f536776610a","Type":"ContainerStarted","Data":"a0089c3a7587fb582b2f368dfc4276ef22829f9419fca354c13b5d981a281fb0"} Nov 04 02:45:21.912695 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:45:21.905115 4405 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w5-3b7476-openshift-default-59975457bf-msl8m" Nov 04 02:45:21.912695 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:45:21.905245 4405 patch_prober.go:29] interesting pod/e2e-worker-w5-3b7476-openshift-default-59975457bf-msl8m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.36:15021/healthz/ready\": dial tcp 10.134.0.36:15021: connect: connection refused" start-of-body= Nov 04 02:45:21.912695 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:45:21.905302 4405 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-3b7476-openshift-default-59975457bf-msl8m" podUID="8a0008f3-4ab9-4ceb-8feb-0f536776610a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.36:15021/healthz/ready\": dial tcp 10.134.0.36:15021: connect: connection refused" Nov 04 02:45:22.188148 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:45:22.188064 4405 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29896005-9vnjz"] Nov 04 02:45:22.242695 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:45:22.237286 4405 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-203c35-854575bcff-vvzfb"] Nov 04 02:45:22.261875 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:45:22.261840 4405 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-203c35-854575bcff-vvzfb" Nov 04 02:45:22.266564 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:45:22.266538 4405 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/abcee031-4335-4e23-8ae7-7cd526a633d0-metrics-certs\") pod \"maas-api-e2e-worker-w4-203c35-854575bcff-vvzfb\" (UID: \"abcee031-4335-4e23-8ae7-7cd526a633d0\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-203c35-854575bcff-vvzfb" Nov 04 02:45:22.266709 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:45:22.266585 4405 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9ppdv\" (UniqueName: \"kubernetes.io/projected/abcee031-4335-4e23-8ae7-7cd526a633d0-kube-api-access-9ppdv\") pod \"maas-api-e2e-worker-w4-203c35-854575bcff-vvzfb\" (UID: \"abcee031-4335-4e23-8ae7-7cd526a633d0\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-203c35-854575bcff-vvzfb" Nov 04 02:45:22.266709 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:45:22.266614 4405 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/abcee031-4335-4e23-8ae7-7cd526a633d0-maas-api-tls\") pod \"maas-api-e2e-worker-w4-203c35-854575bcff-vvzfb\" (UID: \"abcee031-4335-4e23-8ae7-7cd526a633d0\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-203c35-854575bcff-vvzfb" Nov 04 02:45:22.266709 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:45:22.266659 4405 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/abcee031-4335-4e23-8ae7-7cd526a633d0-openshift-service-ca\") pod \"maas-api-e2e-worker-w4-203c35-854575bcff-vvzfb\" (UID: \"abcee031-4335-4e23-8ae7-7cd526a633d0\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-203c35-854575bcff-vvzfb" Nov 04 02:45:22.308927 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:45:22.306454 4405 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-203c35-854575bcff-vvzfb"] Nov 04 02:45:22.315713 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:45:22.315414 4405 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-w4-203c35-serving-cert\"" Nov 04 02:45:22.372023 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:45:22.369872 4405 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/abcee031-4335-4e23-8ae7-7cd526a633d0-metrics-certs\") pod \"maas-api-e2e-worker-w4-203c35-854575bcff-vvzfb\" (UID: \"abcee031-4335-4e23-8ae7-7cd526a633d0\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-203c35-854575bcff-vvzfb" Nov 04 02:45:22.372023 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:45:22.369980 4405 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9ppdv\" (UniqueName: \"kubernetes.io/projected/abcee031-4335-4e23-8ae7-7cd526a633d0-kube-api-access-9ppdv\") pod \"maas-api-e2e-worker-w4-203c35-854575bcff-vvzfb\" (UID: \"abcee031-4335-4e23-8ae7-7cd526a633d0\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-203c35-854575bcff-vvzfb" Nov 04 02:45:22.372023 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:45:22.370031 4405 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/abcee031-4335-4e23-8ae7-7cd526a633d0-maas-api-tls\") pod \"maas-api-e2e-worker-w4-203c35-854575bcff-vvzfb\" (UID: \"abcee031-4335-4e23-8ae7-7cd526a633d0\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-203c35-854575bcff-vvzfb" Nov 04 02:45:22.372023 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:45:22.370131 4405 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/abcee031-4335-4e23-8ae7-7cd526a633d0-openshift-service-ca\") pod \"maas-api-e2e-worker-w4-203c35-854575bcff-vvzfb\" (UID: \"abcee031-4335-4e23-8ae7-7cd526a633d0\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-203c35-854575bcff-vvzfb" Nov 04 02:45:22.372023 ip-10-0-134-125 kubenswrapper[4405]: E1104 02:45:22.370843 4405 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-worker-w4-203c35-serving-cert: secret "maas-api-e2e-worker-w4-203c35-serving-cert" not found Nov 04 02:45:22.372023 ip-10-0-134-125 kubenswrapper[4405]: E1104 02:45:22.370907 4405 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/abcee031-4335-4e23-8ae7-7cd526a633d0-maas-api-tls podName:abcee031-4335-4e23-8ae7-7cd526a633d0 nodeName:}" failed. No retries permitted until 2026-11-04 02:45:22.870887818 +0000 UTC m=+769.301587930 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/abcee031-4335-4e23-8ae7-7cd526a633d0-maas-api-tls") pod "maas-api-e2e-worker-w4-203c35-854575bcff-vvzfb" (UID: "abcee031-4335-4e23-8ae7-7cd526a633d0") : secret "maas-api-e2e-worker-w4-203c35-serving-cert" not found Nov 04 02:45:22.372023 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:45:22.371194 4405 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/abcee031-4335-4e23-8ae7-7cd526a633d0-openshift-service-ca\") pod \"maas-api-e2e-worker-w4-203c35-854575bcff-vvzfb\" (UID: \"abcee031-4335-4e23-8ae7-7cd526a633d0\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-203c35-854575bcff-vvzfb" Nov 04 02:45:22.374999 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:45:22.374399 4405 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/abcee031-4335-4e23-8ae7-7cd526a633d0-metrics-certs\") pod \"maas-api-e2e-worker-w4-203c35-854575bcff-vvzfb\" (UID: \"abcee031-4335-4e23-8ae7-7cd526a633d0\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-203c35-854575bcff-vvzfb" Nov 04 02:45:22.460528 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:45:22.460399 4405 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9ppdv\" (UniqueName: \"kubernetes.io/projected/abcee031-4335-4e23-8ae7-7cd526a633d0-kube-api-access-9ppdv\") pod \"maas-api-e2e-worker-w4-203c35-854575bcff-vvzfb\" (UID: \"abcee031-4335-4e23-8ae7-7cd526a633d0\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-203c35-854575bcff-vvzfb" Nov 04 02:45:22.662952 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:45:22.661204 4405 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29896005-9vnjz" Nov 04 02:45:22.673943 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:45:22.673906 4405 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/62629b7a-09c4-4fd8-bbab-ca11ecd6c696-kube-api-access-r97sh\" (UniqueName: \"kubernetes.io/projected/62629b7a-09c4-4fd8-bbab-ca11ecd6c696-kube-api-access-r97sh\") pod \"62629b7a-09c4-4fd8-bbab-ca11ecd6c696\" (UID: \"62629b7a-09c4-4fd8-bbab-ca11ecd6c696\") " Nov 04 02:45:22.674646 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:45:22.674622 4405 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/62629b7a-09c4-4fd8-bbab-ca11ecd6c696-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/62629b7a-09c4-4fd8-bbab-ca11ecd6c696-maas-api-service-ca\") pod \"62629b7a-09c4-4fd8-bbab-ca11ecd6c696\" (UID: \"62629b7a-09c4-4fd8-bbab-ca11ecd6c696\") " Nov 04 02:45:22.675522 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:45:22.675498 4405 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/62629b7a-09c4-4fd8-bbab-ca11ecd6c696-maas-api-service-ca" pod "62629b7a-09c4-4fd8-bbab-ca11ecd6c696" (UID: "62629b7a-09c4-4fd8-bbab-ca11ecd6c696"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 02:45:22.676427 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:45:22.676401 4405 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/62629b7a-09c4-4fd8-bbab-ca11ecd6c696-kube-api-access-r97sh" pod "62629b7a-09c4-4fd8-bbab-ca11ecd6c696" (UID: "62629b7a-09c4-4fd8-bbab-ca11ecd6c696"). InnerVolumeSpecName "kube-api-access-r97sh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 02:45:22.776183 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:45:22.776092 4405 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/62629b7a-09c4-4fd8-bbab-ca11ecd6c696-maas-api-service-ca\") on node \"ip-10-0-134-125.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 02:45:22.776183 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:45:22.776128 4405 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r97sh\" (UniqueName: \"kubernetes.io/projected/62629b7a-09c4-4fd8-bbab-ca11ecd6c696-kube-api-access-r97sh\") on node \"ip-10-0-134-125.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 02:45:22.877526 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:45:22.877104 4405 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/abcee031-4335-4e23-8ae7-7cd526a633d0-maas-api-tls\") pod \"maas-api-e2e-worker-w4-203c35-854575bcff-vvzfb\" (UID: \"abcee031-4335-4e23-8ae7-7cd526a633d0\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-203c35-854575bcff-vvzfb" Nov 04 02:45:22.880289 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:45:22.880260 4405 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/abcee031-4335-4e23-8ae7-7cd526a633d0-maas-api-tls\") pod \"maas-api-e2e-worker-w4-203c35-854575bcff-vvzfb\" (UID: \"abcee031-4335-4e23-8ae7-7cd526a633d0\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-203c35-854575bcff-vvzfb" Nov 04 02:45:22.880418 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:45:22.880336 4405 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29896005-9vnjz" Nov 04 02:45:22.880539 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:45:22.880513 4405 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29896005-9vnjz" event={"ID":"62629b7a-09c4-4fd8-bbab-ca11ecd6c696","Type":"ContainerDied","Data":"fe3410900d98147211f3f565c7e9652a2049d83c23efb2f2ad2aa05e195ea8ee"} Nov 04 02:45:22.880699 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:45:22.880653 4405 scope.go:122] "RemoveContainer" containerID="9379545f2370b805be89e6aca980816e2db043c900e3f352d1266044880b035a" Nov 04 02:45:22.920732 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:45:22.915097 4405 patch_prober.go:29] interesting pod/e2e-worker-w5-3b7476-openshift-default-59975457bf-msl8m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.36:15021/healthz/ready\": dial tcp 10.134.0.36:15021: connect: connection refused" start-of-body= Nov 04 02:45:22.920732 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:45:22.915158 4405 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-3b7476-openshift-default-59975457bf-msl8m" podUID="8a0008f3-4ab9-4ceb-8feb-0f536776610a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.36:15021/healthz/ready\": dial tcp 10.134.0.36:15021: connect: connection refused" Nov 04 02:45:23.051325 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:45:23.051151 4405 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29896005-9vnjz"] Nov 04 02:45:23.055887 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:45:23.055854 4405 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29896005-9vnjz"] Nov 04 02:45:23.175811 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:45:23.175783 4405 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-203c35-854575bcff-vvzfb" Nov 04 02:45:23.615694 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:45:23.608548 4405 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-203c35-854575bcff-vvzfb"] Nov 04 02:45:23.650901 ip-10-0-134-125 kubenswrapper[4405]: W1104 02:45:23.650868 4405 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podabcee031_4335_4e23_8ae7_7cd526a633d0.slice/crio-15270c53ac73208220376ae3dbd9ad878be87624bddbe4eb607f0d83dae51df6 WatchSource:0}: Error finding container 15270c53ac73208220376ae3dbd9ad878be87624bddbe4eb607f0d83dae51df6: Status 404 returned error can't find the container with id 15270c53ac73208220376ae3dbd9ad878be87624bddbe4eb607f0d83dae51df6 Nov 04 02:45:23.748319 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:45:23.748288 4405 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="62629b7a-09c4-4fd8-bbab-ca11ecd6c696" path="/var/lib/kubelet/pods/62629b7a-09c4-4fd8-bbab-ca11ecd6c696/volumes" Nov 04 02:45:23.886405 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:45:23.886315 4405 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-203c35-854575bcff-vvzfb" event={"ID":"abcee031-4335-4e23-8ae7-7cd526a633d0","Type":"ContainerStarted","Data":"15270c53ac73208220376ae3dbd9ad878be87624bddbe4eb607f0d83dae51df6"} Nov 04 02:45:23.905527 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:45:23.905056 4405 patch_prober.go:29] interesting pod/e2e-worker-w5-3b7476-openshift-default-59975457bf-msl8m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.36:15021/healthz/ready\": dial tcp 10.134.0.36:15021: connect: connection refused" start-of-body= Nov 04 02:45:23.905527 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:45:23.905121 4405 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-3b7476-openshift-default-59975457bf-msl8m" podUID="8a0008f3-4ab9-4ceb-8feb-0f536776610a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.36:15021/healthz/ready\": dial tcp 10.134.0.36:15021: connect: connection refused" Nov 04 02:45:24.932752 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:45:24.931107 4405 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-203c35-854575bcff-vvzfb" event={"ID":"abcee031-4335-4e23-8ae7-7cd526a633d0","Type":"ContainerStarted","Data":"5dae8546b8234161cd91bab1f9efb2fb6f241050981e8cd284615e06cee89a04"} Nov 04 02:45:24.932752 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:45:24.932105 4405 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-203c35-854575bcff-vvzfb" Nov 04 02:45:24.949703 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:45:24.949535 4405 patch_prober.go:29] interesting pod/e2e-worker-w5-3b7476-openshift-default-59975457bf-msl8m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.36:15021/healthz/ready\": dial tcp 10.134.0.36:15021: connect: connection refused" start-of-body= Nov 04 02:45:24.949703 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:45:24.949612 4405 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-3b7476-openshift-default-59975457bf-msl8m" podUID="8a0008f3-4ab9-4ceb-8feb-0f536776610a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.36:15021/healthz/ready\": dial tcp 10.134.0.36:15021: connect: connection refused" Nov 04 02:45:25.050104 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:45:25.050043 4405 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-203c35-854575bcff-vvzfb" podStartSLOduration=2.036583896 podStartE2EDuration="3.050025425s" totalImagesPullingTime="1.013441529s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 02:45:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 02:45:25.049381816 +0000 UTC m=+771.480081938" watchObservedRunningTime="2026-11-04 02:45:25.050025425 +0000 UTC m=+771.480725546" Nov 04 02:45:25.905126 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:45:25.905085 4405 patch_prober.go:29] interesting pod/e2e-worker-w5-3b7476-openshift-default-59975457bf-msl8m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.36:15021/healthz/ready\": dial tcp 10.134.0.36:15021: connect: connection refused" start-of-body= Nov 04 02:45:25.905447 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:45:25.905419 4405 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-3b7476-openshift-default-59975457bf-msl8m" podUID="8a0008f3-4ab9-4ceb-8feb-0f536776610a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.36:15021/healthz/ready\": dial tcp 10.134.0.36:15021: connect: connection refused" Nov 04 02:45:26.912695 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:45:26.907040 4405 patch_prober.go:29] interesting pod/e2e-worker-w5-3b7476-openshift-default-59975457bf-msl8m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.36:15021/healthz/ready\": dial tcp 10.134.0.36:15021: connect: connection refused" start-of-body= Nov 04 02:45:26.912695 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:45:26.907101 4405 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-3b7476-openshift-default-59975457bf-msl8m" podUID="8a0008f3-4ab9-4ceb-8feb-0f536776610a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.36:15021/healthz/ready\": dial tcp 10.134.0.36:15021: connect: connection refused" Nov 04 02:45:27.477870 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:45:27.477828 4405 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w6-ebb7fd-656b55b679-888kl"] Nov 04 02:45:27.478289 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:45:27.478275 4405 cpu_manager.go:414] "containerMap: removing container" podUID="62629b7a-09c4-4fd8-bbab-ca11ecd6c696" containerName="cleanup" Nov 04 02:45:27.478333 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:45:27.478291 4405 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="62629b7a-09c4-4fd8-bbab-ca11ecd6c696" containerName="cleanup" Nov 04 02:45:27.478333 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:45:27.478302 4405 cpu_manager.go:414] "containerMap: removing container" podUID="62629b7a-09c4-4fd8-bbab-ca11ecd6c696" containerName="cleanup" Nov 04 02:45:27.478333 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:45:27.478307 4405 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="62629b7a-09c4-4fd8-bbab-ca11ecd6c696" containerName="cleanup" Nov 04 02:45:27.478333 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:45:27.478320 4405 cpu_manager.go:414] "containerMap: removing container" podUID="62629b7a-09c4-4fd8-bbab-ca11ecd6c696" containerName="cleanup" Nov 04 02:45:27.478333 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:45:27.478326 4405 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="62629b7a-09c4-4fd8-bbab-ca11ecd6c696" containerName="cleanup" Nov 04 02:45:27.478498 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:45:27.478409 4405 memory_manager.go:365] "RemoveStaleState removing state" podUID="62629b7a-09c4-4fd8-bbab-ca11ecd6c696" containerName="cleanup" Nov 04 02:45:27.478498 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:45:27.478470 4405 memory_manager.go:365] "RemoveStaleState removing state" podUID="62629b7a-09c4-4fd8-bbab-ca11ecd6c696" containerName="cleanup" Nov 04 02:45:27.478498 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:45:27.478480 4405 memory_manager.go:365] "RemoveStaleState removing state" podUID="62629b7a-09c4-4fd8-bbab-ca11ecd6c696" containerName="cleanup" Nov 04 02:45:27.481941 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:45:27.481918 4405 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-ebb7fd-656b55b679-888kl" Nov 04 02:45:27.534334 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:45:27.534298 4405 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-w6-ebb7fd-serving-cert\"" Nov 04 02:45:27.584206 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:45:27.584170 4405 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/52a0927a-e7c2-4c09-aa18-fab7751ca679-metrics-certs\") pod \"maas-api-e2e-worker-w6-ebb7fd-656b55b679-888kl\" (UID: \"52a0927a-e7c2-4c09-aa18-fab7751ca679\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-ebb7fd-656b55b679-888kl" Nov 04 02:45:27.584414 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:45:27.584301 4405 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/52a0927a-e7c2-4c09-aa18-fab7751ca679-maas-api-tls\") pod \"maas-api-e2e-worker-w6-ebb7fd-656b55b679-888kl\" (UID: \"52a0927a-e7c2-4c09-aa18-fab7751ca679\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-ebb7fd-656b55b679-888kl" Nov 04 02:45:27.584414 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:45:27.584371 4405 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/52a0927a-e7c2-4c09-aa18-fab7751ca679-openshift-service-ca\") pod \"maas-api-e2e-worker-w6-ebb7fd-656b55b679-888kl\" (UID: \"52a0927a-e7c2-4c09-aa18-fab7751ca679\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-ebb7fd-656b55b679-888kl" Nov 04 02:45:27.584531 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:45:27.584426 4405 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hntrh\" (UniqueName: \"kubernetes.io/projected/52a0927a-e7c2-4c09-aa18-fab7751ca679-kube-api-access-hntrh\") pod \"maas-api-e2e-worker-w6-ebb7fd-656b55b679-888kl\" (UID: \"52a0927a-e7c2-4c09-aa18-fab7751ca679\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-ebb7fd-656b55b679-888kl" Nov 04 02:45:27.586426 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:45:27.586396 4405 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w6-ebb7fd-656b55b679-888kl"] Nov 04 02:45:27.686094 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:45:27.686056 4405 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/52a0927a-e7c2-4c09-aa18-fab7751ca679-maas-api-tls\") pod \"maas-api-e2e-worker-w6-ebb7fd-656b55b679-888kl\" (UID: \"52a0927a-e7c2-4c09-aa18-fab7751ca679\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-ebb7fd-656b55b679-888kl" Nov 04 02:45:27.686298 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:45:27.686126 4405 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/52a0927a-e7c2-4c09-aa18-fab7751ca679-openshift-service-ca\") pod \"maas-api-e2e-worker-w6-ebb7fd-656b55b679-888kl\" (UID: \"52a0927a-e7c2-4c09-aa18-fab7751ca679\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-ebb7fd-656b55b679-888kl" Nov 04 02:45:27.686298 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:45:27.686189 4405 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hntrh\" (UniqueName: \"kubernetes.io/projected/52a0927a-e7c2-4c09-aa18-fab7751ca679-kube-api-access-hntrh\") pod \"maas-api-e2e-worker-w6-ebb7fd-656b55b679-888kl\" (UID: \"52a0927a-e7c2-4c09-aa18-fab7751ca679\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-ebb7fd-656b55b679-888kl" Nov 04 02:45:27.686298 ip-10-0-134-125 kubenswrapper[4405]: E1104 02:45:27.686233 4405 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-worker-w6-ebb7fd-serving-cert: secret "maas-api-e2e-worker-w6-ebb7fd-serving-cert" not found Nov 04 02:45:27.686298 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:45:27.686287 4405 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/52a0927a-e7c2-4c09-aa18-fab7751ca679-metrics-certs\") pod \"maas-api-e2e-worker-w6-ebb7fd-656b55b679-888kl\" (UID: \"52a0927a-e7c2-4c09-aa18-fab7751ca679\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-ebb7fd-656b55b679-888kl" Nov 04 02:45:27.686552 ip-10-0-134-125 kubenswrapper[4405]: E1104 02:45:27.686314 4405 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/52a0927a-e7c2-4c09-aa18-fab7751ca679-maas-api-tls podName:52a0927a-e7c2-4c09-aa18-fab7751ca679 nodeName:}" failed. No retries permitted until 2026-11-04 02:45:28.186293097 +0000 UTC m=+774.616993195 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/52a0927a-e7c2-4c09-aa18-fab7751ca679-maas-api-tls") pod "maas-api-e2e-worker-w6-ebb7fd-656b55b679-888kl" (UID: "52a0927a-e7c2-4c09-aa18-fab7751ca679") : secret "maas-api-e2e-worker-w6-ebb7fd-serving-cert" not found Nov 04 02:45:27.686874 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:45:27.686856 4405 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/52a0927a-e7c2-4c09-aa18-fab7751ca679-openshift-service-ca\") pod \"maas-api-e2e-worker-w6-ebb7fd-656b55b679-888kl\" (UID: \"52a0927a-e7c2-4c09-aa18-fab7751ca679\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-ebb7fd-656b55b679-888kl" Nov 04 02:45:27.689112 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:45:27.689081 4405 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/52a0927a-e7c2-4c09-aa18-fab7751ca679-metrics-certs\") pod \"maas-api-e2e-worker-w6-ebb7fd-656b55b679-888kl\" (UID: \"52a0927a-e7c2-4c09-aa18-fab7751ca679\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-ebb7fd-656b55b679-888kl" Nov 04 02:45:27.749825 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:45:27.749752 4405 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hntrh\" (UniqueName: \"kubernetes.io/projected/52a0927a-e7c2-4c09-aa18-fab7751ca679-kube-api-access-hntrh\") pod \"maas-api-e2e-worker-w6-ebb7fd-656b55b679-888kl\" (UID: \"52a0927a-e7c2-4c09-aa18-fab7751ca679\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-ebb7fd-656b55b679-888kl" Nov 04 02:45:27.924391 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:45:27.924209 4405 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w5-3b7476-openshift-default-59975457bf-msl8m" Nov 04 02:45:27.925647 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:45:27.925618 4405 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w5-3b7476-openshift-default-59975457bf-msl8m" Nov 04 02:45:27.927389 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:45:27.927364 4405 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w5-3b7476-openshift-default-59975457bf-msl8m" Nov 04 02:45:28.191916 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:45:28.191874 4405 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/52a0927a-e7c2-4c09-aa18-fab7751ca679-maas-api-tls\") pod \"maas-api-e2e-worker-w6-ebb7fd-656b55b679-888kl\" (UID: \"52a0927a-e7c2-4c09-aa18-fab7751ca679\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-ebb7fd-656b55b679-888kl" Nov 04 02:45:28.195602 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:45:28.195373 4405 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/52a0927a-e7c2-4c09-aa18-fab7751ca679-maas-api-tls\") pod \"maas-api-e2e-worker-w6-ebb7fd-656b55b679-888kl\" (UID: \"52a0927a-e7c2-4c09-aa18-fab7751ca679\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-ebb7fd-656b55b679-888kl" Nov 04 02:45:28.251066 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:45:28.251033 4405 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-bl5mg"] Nov 04 02:45:28.395580 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:45:28.395502 4405 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-ebb7fd-656b55b679-888kl" Nov 04 02:45:28.806969 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:45:28.806645 4405 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w6-ebb7fd-656b55b679-888kl"] Nov 04 02:45:28.811006 ip-10-0-134-125 kubenswrapper[4405]: W1104 02:45:28.810971 4405 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod52a0927a_e7c2_4c09_aa18_fab7751ca679.slice/crio-0f4dfa51d507242d38c3871eea04b847edbce30c884afde5483e479e745a689d WatchSource:0}: Error finding container 0f4dfa51d507242d38c3871eea04b847edbce30c884afde5483e479e745a689d: Status 404 returned error can't find the container with id 0f4dfa51d507242d38c3871eea04b847edbce30c884afde5483e479e745a689d Nov 04 02:45:28.975217 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:45:28.975179 4405 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-ebb7fd-656b55b679-888kl" event={"ID":"52a0927a-e7c2-4c09-aa18-fab7751ca679","Type":"ContainerStarted","Data":"0f4dfa51d507242d38c3871eea04b847edbce30c884afde5483e479e745a689d"} Nov 04 02:45:29.982429 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:45:29.982393 4405 generic.go:356] "Generic (PLEG): container finished" podID="3fd4ae35-84c1-45c1-a49c-14f2a8ed1c82" containerID="dc4e5d146d6b9dd6d75a3cb505756dee5d5552f16791b4deb553fc92ab758cc8" exitCode=0 Nov 04 02:45:29.982840 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:45:29.982459 4405 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-e516ff-openshift-default-6f4b957485-bjmn9" event={"ID":"3fd4ae35-84c1-45c1-a49c-14f2a8ed1c82","Type":"ContainerDied","Data":"dc4e5d146d6b9dd6d75a3cb505756dee5d5552f16791b4deb553fc92ab758cc8"} Nov 04 02:45:29.982905 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:45:29.982892 4405 scope.go:122] "RemoveContainer" containerID="dc4e5d146d6b9dd6d75a3cb505756dee5d5552f16791b4deb553fc92ab758cc8" Nov 04 02:45:29.984203 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:45:29.984173 4405 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 04 02:45:29.984310 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:45:29.984246 4405 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 04 02:45:29.984310 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:45:29.984279 4405 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 04 02:45:30.991020 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:45:30.990980 4405 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-e516ff-openshift-default-6f4b957485-bjmn9" event={"ID":"3fd4ae35-84c1-45c1-a49c-14f2a8ed1c82","Type":"ContainerStarted","Data":"1c64e020a17f41c8ad7c11a821b583d8becae3a4e813baa156a5e6f237a17ade"} Nov 04 02:45:30.993036 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:45:30.993004 4405 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-ebb7fd-656b55b679-888kl" event={"ID":"52a0927a-e7c2-4c09-aa18-fab7751ca679","Type":"ContainerStarted","Data":"e22f477f23d5d43f7cca76976e417db7e18b13645a59ade7ef689a90b05e2782"} Nov 04 02:45:30.993955 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:45:30.993938 4405 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-ebb7fd-656b55b679-888kl" Nov 04 02:45:31.228234 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:45:31.228179 4405 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-ebb7fd-656b55b679-888kl" podStartSLOduration=3.094048446 podStartE2EDuration="4.228163746s" totalImagesPullingTime="1.1341153s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 02:45:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 02:45:31.226791117 +0000 UTC m=+777.657491239" watchObservedRunningTime="2026-11-04 02:45:31.228163746 +0000 UTC m=+777.658863866" Nov 04 02:45:31.256661 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:45:31.256575 4405 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-e516ff-openshift-default-6f4b957485-bjmn9" Nov 04 02:45:31.257944 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:45:31.257920 4405 patch_prober.go:29] interesting pod/e2e-worker-w1-e516ff-openshift-default-6f4b957485-bjmn9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.35:15021/healthz/ready\": dial tcp 10.134.0.35:15021: connect: connection refused" start-of-body= Nov 04 02:45:31.258000 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:45:31.257974 4405 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-e516ff-openshift-default-6f4b957485-bjmn9" podUID="3fd4ae35-84c1-45c1-a49c-14f2a8ed1c82" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.35:15021/healthz/ready\": dial tcp 10.134.0.35:15021: connect: connection refused" Nov 04 02:45:31.968994 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:45:31.968966 4405 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-203c35-854575bcff-vvzfb" Nov 04 02:45:32.256244 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:45:32.256156 4405 patch_prober.go:29] interesting pod/e2e-worker-w1-e516ff-openshift-default-6f4b957485-bjmn9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.35:15021/healthz/ready\": dial tcp 10.134.0.35:15021: connect: connection refused" start-of-body= Nov 04 02:45:32.256244 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:45:32.256219 4405 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-e516ff-openshift-default-6f4b957485-bjmn9" podUID="3fd4ae35-84c1-45c1-a49c-14f2a8ed1c82" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.35:15021/healthz/ready\": dial tcp 10.134.0.35:15021: connect: connection refused" Nov 04 02:45:32.564245 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:45:32.564163 4405 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-e516ff-openshift-default-6f4b957485-bjmn9" Nov 04 02:45:33.256814 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:45:33.256777 4405 patch_prober.go:29] interesting pod/e2e-worker-w1-e516ff-openshift-default-6f4b957485-bjmn9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.35:15021/healthz/ready\": dial tcp 10.134.0.35:15021: connect: connection refused" start-of-body= Nov 04 02:45:33.257353 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:45:33.256853 4405 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-e516ff-openshift-default-6f4b957485-bjmn9" podUID="3fd4ae35-84c1-45c1-a49c-14f2a8ed1c82" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.35:15021/healthz/ready\": dial tcp 10.134.0.35:15021: connect: connection refused" Nov 04 02:45:34.256749 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:45:34.256709 4405 patch_prober.go:29] interesting pod/e2e-worker-w1-e516ff-openshift-default-6f4b957485-bjmn9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.35:15021/healthz/ready\": dial tcp 10.134.0.35:15021: connect: connection refused" start-of-body= Nov 04 02:45:34.256941 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:45:34.256776 4405 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-e516ff-openshift-default-6f4b957485-bjmn9" podUID="3fd4ae35-84c1-45c1-a49c-14f2a8ed1c82" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.35:15021/healthz/ready\": dial tcp 10.134.0.35:15021: connect: connection refused" Nov 04 02:45:35.256928 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:45:35.256892 4405 patch_prober.go:29] interesting pod/e2e-worker-w1-e516ff-openshift-default-6f4b957485-bjmn9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.35:15021/healthz/ready\": dial tcp 10.134.0.35:15021: connect: connection refused" start-of-body= Nov 04 02:45:35.257351 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:45:35.256951 4405 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-e516ff-openshift-default-6f4b957485-bjmn9" podUID="3fd4ae35-84c1-45c1-a49c-14f2a8ed1c82" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.35:15021/healthz/ready\": dial tcp 10.134.0.35:15021: connect: connection refused" Nov 04 02:45:36.261237 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:45:36.261205 4405 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-e516ff-openshift-default-6f4b957485-bjmn9" Nov 04 02:45:36.262505 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:45:36.262484 4405 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-e516ff-openshift-default-6f4b957485-bjmn9" Nov 04 02:45:36.883423 ip-10-0-134-125 kubenswrapper[4405]: E1104 02:45:36.883388 4405 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5f2vb" containerName="extractor" Nov 04 02:45:38.034817 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:45:38.034790 4405 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-ebb7fd-656b55b679-888kl" Nov 04 02:45:47.750481 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:45:47.750450 4405 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-bl5mg"] Nov 04 02:46:02.371399 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:46:02.371361 4405 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-bl5mg"] Nov 04 02:46:06.884187 ip-10-0-134-125 kubenswrapper[4405]: E1104 02:46:06.884147 4405 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5f2vb" containerName="extractor" Nov 04 02:46:20.190898 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:46:20.190864 4405 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-bl5mg"] Nov 04 02:46:36.884121 ip-10-0-134-125 kubenswrapper[4405]: E1104 02:46:36.884076 4405 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5f2vb" containerName="extractor" Nov 04 02:46:39.198764 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:46:39.198730 4405 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-bl5mg"] Nov 04 02:46:44.497891 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:46:44.497862 4405 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w4-203c35/simulator-w4-203c35-kserve-f9c4bd695-999ns"] Nov 04 02:46:44.501676 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:46:44.501647 4405 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-203c35/simulator-w4-203c35-kserve-f9c4bd695-999ns" Nov 04 02:46:44.567776 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:46:44.567714 4405 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kqsr7\" (UniqueName: \"kubernetes.io/projected/eb0d8bdd-6808-4fd0-b9b5-567dff3e7ca5-kube-api-access-kqsr7\") pod \"simulator-w4-203c35-kserve-f9c4bd695-999ns\" (UID: \"eb0d8bdd-6808-4fd0-b9b5-567dff3e7ca5\") " pod="e2e-models-e2e-worker-w4-203c35/simulator-w4-203c35-kserve-f9c4bd695-999ns" Nov 04 02:46:44.567961 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:46:44.567830 4405 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/eb0d8bdd-6808-4fd0-b9b5-567dff3e7ca5-tmp-dir\") pod \"simulator-w4-203c35-kserve-f9c4bd695-999ns\" (UID: \"eb0d8bdd-6808-4fd0-b9b5-567dff3e7ca5\") " pod="e2e-models-e2e-worker-w4-203c35/simulator-w4-203c35-kserve-f9c4bd695-999ns" Nov 04 02:46:44.567961 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:46:44.567887 4405 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/eb0d8bdd-6808-4fd0-b9b5-567dff3e7ca5-home\") pod \"simulator-w4-203c35-kserve-f9c4bd695-999ns\" (UID: \"eb0d8bdd-6808-4fd0-b9b5-567dff3e7ca5\") " pod="e2e-models-e2e-worker-w4-203c35/simulator-w4-203c35-kserve-f9c4bd695-999ns" Nov 04 02:46:44.567961 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:46:44.567957 4405 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/eb0d8bdd-6808-4fd0-b9b5-567dff3e7ca5-tls-certs\") pod \"simulator-w4-203c35-kserve-f9c4bd695-999ns\" (UID: \"eb0d8bdd-6808-4fd0-b9b5-567dff3e7ca5\") " pod="e2e-models-e2e-worker-w4-203c35/simulator-w4-203c35-kserve-f9c4bd695-999ns" Nov 04 02:46:44.568147 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:46:44.568071 4405 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/eb0d8bdd-6808-4fd0-b9b5-567dff3e7ca5-dshm\") pod \"simulator-w4-203c35-kserve-f9c4bd695-999ns\" (UID: \"eb0d8bdd-6808-4fd0-b9b5-567dff3e7ca5\") " pod="e2e-models-e2e-worker-w4-203c35/simulator-w4-203c35-kserve-f9c4bd695-999ns" Nov 04 02:46:44.568267 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:46:44.568244 4405 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/eb0d8bdd-6808-4fd0-b9b5-567dff3e7ca5-model-cache\") pod \"simulator-w4-203c35-kserve-f9c4bd695-999ns\" (UID: \"eb0d8bdd-6808-4fd0-b9b5-567dff3e7ca5\") " pod="e2e-models-e2e-worker-w4-203c35/simulator-w4-203c35-kserve-f9c4bd695-999ns" Nov 04 02:46:44.606717 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:46:44.606678 4405 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-203c35\"/\"default-dockercfg-tkts4\"" Nov 04 02:46:44.606889 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:46:44.606776 4405 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-203c35\"/\"simulator-w4-203c35-kserve-self-signed-certs\"" Nov 04 02:46:44.606889 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:46:44.606826 4405 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w4-203c35\"/\"kube-root-ca.crt\"" Nov 04 02:46:44.606889 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:46:44.606863 4405 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w4-203c35\"/\"openshift-service-ca.crt\"" Nov 04 02:46:44.608586 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:46:44.608559 4405 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-203c35/simulator-w4-203c35-kserve-f9c4bd695-999ns"] Nov 04 02:46:44.669240 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:46:44.669206 4405 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/eb0d8bdd-6808-4fd0-b9b5-567dff3e7ca5-tls-certs\") pod \"simulator-w4-203c35-kserve-f9c4bd695-999ns\" (UID: \"eb0d8bdd-6808-4fd0-b9b5-567dff3e7ca5\") " pod="e2e-models-e2e-worker-w4-203c35/simulator-w4-203c35-kserve-f9c4bd695-999ns" Nov 04 02:46:44.669433 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:46:44.669249 4405 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/eb0d8bdd-6808-4fd0-b9b5-567dff3e7ca5-dshm\") pod \"simulator-w4-203c35-kserve-f9c4bd695-999ns\" (UID: \"eb0d8bdd-6808-4fd0-b9b5-567dff3e7ca5\") " pod="e2e-models-e2e-worker-w4-203c35/simulator-w4-203c35-kserve-f9c4bd695-999ns" Nov 04 02:46:44.669433 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:46:44.669282 4405 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/eb0d8bdd-6808-4fd0-b9b5-567dff3e7ca5-model-cache\") pod \"simulator-w4-203c35-kserve-f9c4bd695-999ns\" (UID: \"eb0d8bdd-6808-4fd0-b9b5-567dff3e7ca5\") " pod="e2e-models-e2e-worker-w4-203c35/simulator-w4-203c35-kserve-f9c4bd695-999ns" Nov 04 02:46:44.669433 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:46:44.669333 4405 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kqsr7\" (UniqueName: \"kubernetes.io/projected/eb0d8bdd-6808-4fd0-b9b5-567dff3e7ca5-kube-api-access-kqsr7\") pod \"simulator-w4-203c35-kserve-f9c4bd695-999ns\" (UID: \"eb0d8bdd-6808-4fd0-b9b5-567dff3e7ca5\") " pod="e2e-models-e2e-worker-w4-203c35/simulator-w4-203c35-kserve-f9c4bd695-999ns" Nov 04 02:46:44.669433 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:46:44.669397 4405 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/eb0d8bdd-6808-4fd0-b9b5-567dff3e7ca5-tmp-dir\") pod \"simulator-w4-203c35-kserve-f9c4bd695-999ns\" (UID: \"eb0d8bdd-6808-4fd0-b9b5-567dff3e7ca5\") " pod="e2e-models-e2e-worker-w4-203c35/simulator-w4-203c35-kserve-f9c4bd695-999ns" Nov 04 02:46:44.669624 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:46:44.669440 4405 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/eb0d8bdd-6808-4fd0-b9b5-567dff3e7ca5-home\") pod \"simulator-w4-203c35-kserve-f9c4bd695-999ns\" (UID: \"eb0d8bdd-6808-4fd0-b9b5-567dff3e7ca5\") " pod="e2e-models-e2e-worker-w4-203c35/simulator-w4-203c35-kserve-f9c4bd695-999ns" Nov 04 02:46:44.669899 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:46:44.669873 4405 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/eb0d8bdd-6808-4fd0-b9b5-567dff3e7ca5-model-cache\") pod \"simulator-w4-203c35-kserve-f9c4bd695-999ns\" (UID: \"eb0d8bdd-6808-4fd0-b9b5-567dff3e7ca5\") " pod="e2e-models-e2e-worker-w4-203c35/simulator-w4-203c35-kserve-f9c4bd695-999ns" Nov 04 02:46:44.670049 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:46:44.669875 4405 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/eb0d8bdd-6808-4fd0-b9b5-567dff3e7ca5-home\") pod \"simulator-w4-203c35-kserve-f9c4bd695-999ns\" (UID: \"eb0d8bdd-6808-4fd0-b9b5-567dff3e7ca5\") " pod="e2e-models-e2e-worker-w4-203c35/simulator-w4-203c35-kserve-f9c4bd695-999ns" Nov 04 02:46:44.670126 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:46:44.670082 4405 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/eb0d8bdd-6808-4fd0-b9b5-567dff3e7ca5-tmp-dir\") pod \"simulator-w4-203c35-kserve-f9c4bd695-999ns\" (UID: \"eb0d8bdd-6808-4fd0-b9b5-567dff3e7ca5\") " pod="e2e-models-e2e-worker-w4-203c35/simulator-w4-203c35-kserve-f9c4bd695-999ns" Nov 04 02:46:44.671803 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:46:44.671784 4405 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/eb0d8bdd-6808-4fd0-b9b5-567dff3e7ca5-dshm\") pod \"simulator-w4-203c35-kserve-f9c4bd695-999ns\" (UID: \"eb0d8bdd-6808-4fd0-b9b5-567dff3e7ca5\") " pod="e2e-models-e2e-worker-w4-203c35/simulator-w4-203c35-kserve-f9c4bd695-999ns" Nov 04 02:46:44.671979 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:46:44.671962 4405 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/eb0d8bdd-6808-4fd0-b9b5-567dff3e7ca5-tls-certs\") pod \"simulator-w4-203c35-kserve-f9c4bd695-999ns\" (UID: \"eb0d8bdd-6808-4fd0-b9b5-567dff3e7ca5\") " pod="e2e-models-e2e-worker-w4-203c35/simulator-w4-203c35-kserve-f9c4bd695-999ns" Nov 04 02:46:44.730333 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:46:44.730304 4405 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kqsr7\" (UniqueName: \"kubernetes.io/projected/eb0d8bdd-6808-4fd0-b9b5-567dff3e7ca5-kube-api-access-kqsr7\") pod \"simulator-w4-203c35-kserve-f9c4bd695-999ns\" (UID: \"eb0d8bdd-6808-4fd0-b9b5-567dff3e7ca5\") " pod="e2e-models-e2e-worker-w4-203c35/simulator-w4-203c35-kserve-f9c4bd695-999ns" Nov 04 02:46:44.846463 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:46:44.846426 4405 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-203c35/simulator-w4-203c35-kserve-f9c4bd695-999ns" Nov 04 02:46:45.221100 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:46:45.221076 4405 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-203c35/simulator-w4-203c35-kserve-f9c4bd695-999ns"] Nov 04 02:46:45.223454 ip-10-0-134-125 kubenswrapper[4405]: W1104 02:46:45.223428 4405 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podeb0d8bdd_6808_4fd0_b9b5_567dff3e7ca5.slice/crio-62a9f44f8d8642c7d9d62f72a0791919c0889270b4cef102a86eb5dcf211039b WatchSource:0}: Error finding container 62a9f44f8d8642c7d9d62f72a0791919c0889270b4cef102a86eb5dcf211039b: Status 404 returned error can't find the container with id 62a9f44f8d8642c7d9d62f72a0791919c0889270b4cef102a86eb5dcf211039b Nov 04 02:46:45.411925 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:46:45.411892 4405 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-203c35/simulator-w4-203c35-kserve-f9c4bd695-999ns" event={"ID":"eb0d8bdd-6808-4fd0-b9b5-567dff3e7ca5","Type":"ContainerStarted","Data":"829ab80bf344e6089721bbc516ad81919e0c2be8bbe7a1596cd5c5db213f1adb"} Nov 04 02:46:45.412127 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:46:45.411934 4405 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-203c35/simulator-w4-203c35-kserve-f9c4bd695-999ns" event={"ID":"eb0d8bdd-6808-4fd0-b9b5-567dff3e7ca5","Type":"ContainerStarted","Data":"62a9f44f8d8642c7d9d62f72a0791919c0889270b4cef102a86eb5dcf211039b"} Nov 04 02:46:45.532913 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:46:45.532787 4405 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w4-203c35/simulator-w4-203c35-kserve-f9c4bd695-999ns" podStartSLOduration=1.532765068 podStartE2EDuration="1.532765068s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 02:46:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 02:46:45.530979231 +0000 UTC m=+851.961679355" watchObservedRunningTime="2026-11-04 02:46:45.532765068 +0000 UTC m=+851.963465192" Nov 04 02:46:45.847000 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:46:45.846967 4405 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-203c35/simulator-w4-203c35-kserve-f9c4bd695-999ns" Nov 04 02:46:54.846749 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:46:54.846705 4405 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w4-203c35/simulator-w4-203c35-kserve-f9c4bd695-999ns" Nov 04 02:46:54.859855 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:46:54.859821 4405 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w4-203c35/simulator-w4-203c35-kserve-f9c4bd695-999ns" Nov 04 02:46:55.476311 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:46:55.476281 4405 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-203c35/simulator-w4-203c35-kserve-f9c4bd695-999ns" Nov 04 02:47:05.203606 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:47:05.203573 4405 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-bl5mg"] Nov 04 02:47:06.883272 ip-10-0-134-125 kubenswrapper[4405]: E1104 02:47:06.883234 4405 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5f2vb" containerName="extractor" Nov 04 02:47:18.363470 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:47:18.363437 4405 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w2-a0bcbc/premium-w2-a0bcbc-kserve-78dfd6d78c-lwth2"] Nov 04 02:47:18.367264 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:47:18.367244 4405 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-a0bcbc/premium-w2-a0bcbc-kserve-78dfd6d78c-lwth2" Nov 04 02:47:18.419617 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:47:18.419586 4405 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w2-a0bcbc\"/\"kube-root-ca.crt\"" Nov 04 02:47:18.430940 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:47:18.430907 4405 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/33b620c1-389c-48b4-b277-90252018c4b6-tmp-dir\") pod \"premium-w2-a0bcbc-kserve-78dfd6d78c-lwth2\" (UID: \"33b620c1-389c-48b4-b277-90252018c4b6\") " pod="e2e-models-e2e-worker-w2-a0bcbc/premium-w2-a0bcbc-kserve-78dfd6d78c-lwth2" Nov 04 02:47:18.431092 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:47:18.431010 4405 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/33b620c1-389c-48b4-b277-90252018c4b6-home\") pod \"premium-w2-a0bcbc-kserve-78dfd6d78c-lwth2\" (UID: \"33b620c1-389c-48b4-b277-90252018c4b6\") " pod="e2e-models-e2e-worker-w2-a0bcbc/premium-w2-a0bcbc-kserve-78dfd6d78c-lwth2" Nov 04 02:47:18.431092 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:47:18.431041 4405 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/33b620c1-389c-48b4-b277-90252018c4b6-model-cache\") pod \"premium-w2-a0bcbc-kserve-78dfd6d78c-lwth2\" (UID: \"33b620c1-389c-48b4-b277-90252018c4b6\") " pod="e2e-models-e2e-worker-w2-a0bcbc/premium-w2-a0bcbc-kserve-78dfd6d78c-lwth2" Nov 04 02:47:18.431185 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:47:18.431101 4405 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-45xsf\" (UniqueName: \"kubernetes.io/projected/33b620c1-389c-48b4-b277-90252018c4b6-kube-api-access-45xsf\") pod \"premium-w2-a0bcbc-kserve-78dfd6d78c-lwth2\" (UID: \"33b620c1-389c-48b4-b277-90252018c4b6\") " pod="e2e-models-e2e-worker-w2-a0bcbc/premium-w2-a0bcbc-kserve-78dfd6d78c-lwth2" Nov 04 02:47:18.431185 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:47:18.431124 4405 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/33b620c1-389c-48b4-b277-90252018c4b6-dshm\") pod \"premium-w2-a0bcbc-kserve-78dfd6d78c-lwth2\" (UID: \"33b620c1-389c-48b4-b277-90252018c4b6\") " pod="e2e-models-e2e-worker-w2-a0bcbc/premium-w2-a0bcbc-kserve-78dfd6d78c-lwth2" Nov 04 02:47:18.431185 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:47:18.431165 4405 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/33b620c1-389c-48b4-b277-90252018c4b6-tls-certs\") pod \"premium-w2-a0bcbc-kserve-78dfd6d78c-lwth2\" (UID: \"33b620c1-389c-48b4-b277-90252018c4b6\") " pod="e2e-models-e2e-worker-w2-a0bcbc/premium-w2-a0bcbc-kserve-78dfd6d78c-lwth2" Nov 04 02:47:18.471519 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:47:18.471486 4405 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-a0bcbc/premium-w2-a0bcbc-kserve-78dfd6d78c-lwth2"] Nov 04 02:47:18.471951 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:47:18.471929 4405 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-a0bcbc\"/\"premium-w2-a0bcbc-kserve-self-signed-certs\"" Nov 04 02:47:18.472050 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:47:18.472021 4405 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-a0bcbc\"/\"default-dockercfg-fr85m\"" Nov 04 02:47:18.519615 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:47:18.519583 4405 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w2-a0bcbc\"/\"openshift-service-ca.crt\"" Nov 04 02:47:18.531873 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:47:18.531844 4405 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/33b620c1-389c-48b4-b277-90252018c4b6-tls-certs\") pod \"premium-w2-a0bcbc-kserve-78dfd6d78c-lwth2\" (UID: \"33b620c1-389c-48b4-b277-90252018c4b6\") " pod="e2e-models-e2e-worker-w2-a0bcbc/premium-w2-a0bcbc-kserve-78dfd6d78c-lwth2" Nov 04 02:47:18.532058 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:47:18.531927 4405 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/33b620c1-389c-48b4-b277-90252018c4b6-tmp-dir\") pod \"premium-w2-a0bcbc-kserve-78dfd6d78c-lwth2\" (UID: \"33b620c1-389c-48b4-b277-90252018c4b6\") " pod="e2e-models-e2e-worker-w2-a0bcbc/premium-w2-a0bcbc-kserve-78dfd6d78c-lwth2" Nov 04 02:47:18.532058 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:47:18.531972 4405 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/33b620c1-389c-48b4-b277-90252018c4b6-home\") pod \"premium-w2-a0bcbc-kserve-78dfd6d78c-lwth2\" (UID: \"33b620c1-389c-48b4-b277-90252018c4b6\") " pod="e2e-models-e2e-worker-w2-a0bcbc/premium-w2-a0bcbc-kserve-78dfd6d78c-lwth2" Nov 04 02:47:18.532058 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:47:18.531990 4405 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/33b620c1-389c-48b4-b277-90252018c4b6-model-cache\") pod \"premium-w2-a0bcbc-kserve-78dfd6d78c-lwth2\" (UID: \"33b620c1-389c-48b4-b277-90252018c4b6\") " pod="e2e-models-e2e-worker-w2-a0bcbc/premium-w2-a0bcbc-kserve-78dfd6d78c-lwth2" Nov 04 02:47:18.532058 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:47:18.532043 4405 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-45xsf\" (UniqueName: \"kubernetes.io/projected/33b620c1-389c-48b4-b277-90252018c4b6-kube-api-access-45xsf\") pod \"premium-w2-a0bcbc-kserve-78dfd6d78c-lwth2\" (UID: \"33b620c1-389c-48b4-b277-90252018c4b6\") " pod="e2e-models-e2e-worker-w2-a0bcbc/premium-w2-a0bcbc-kserve-78dfd6d78c-lwth2" Nov 04 02:47:18.532275 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:47:18.532076 4405 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/33b620c1-389c-48b4-b277-90252018c4b6-dshm\") pod \"premium-w2-a0bcbc-kserve-78dfd6d78c-lwth2\" (UID: \"33b620c1-389c-48b4-b277-90252018c4b6\") " pod="e2e-models-e2e-worker-w2-a0bcbc/premium-w2-a0bcbc-kserve-78dfd6d78c-lwth2" Nov 04 02:47:18.532548 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:47:18.532524 4405 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/33b620c1-389c-48b4-b277-90252018c4b6-tmp-dir\") pod \"premium-w2-a0bcbc-kserve-78dfd6d78c-lwth2\" (UID: \"33b620c1-389c-48b4-b277-90252018c4b6\") " pod="e2e-models-e2e-worker-w2-a0bcbc/premium-w2-a0bcbc-kserve-78dfd6d78c-lwth2" Nov 04 02:47:18.532860 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:47:18.532570 4405 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/33b620c1-389c-48b4-b277-90252018c4b6-model-cache\") pod \"premium-w2-a0bcbc-kserve-78dfd6d78c-lwth2\" (UID: \"33b620c1-389c-48b4-b277-90252018c4b6\") " pod="e2e-models-e2e-worker-w2-a0bcbc/premium-w2-a0bcbc-kserve-78dfd6d78c-lwth2" Nov 04 02:47:18.532860 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:47:18.532662 4405 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/33b620c1-389c-48b4-b277-90252018c4b6-home\") pod \"premium-w2-a0bcbc-kserve-78dfd6d78c-lwth2\" (UID: \"33b620c1-389c-48b4-b277-90252018c4b6\") " pod="e2e-models-e2e-worker-w2-a0bcbc/premium-w2-a0bcbc-kserve-78dfd6d78c-lwth2" Nov 04 02:47:18.534685 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:47:18.534648 4405 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/33b620c1-389c-48b4-b277-90252018c4b6-tls-certs\") pod \"premium-w2-a0bcbc-kserve-78dfd6d78c-lwth2\" (UID: \"33b620c1-389c-48b4-b277-90252018c4b6\") " pod="e2e-models-e2e-worker-w2-a0bcbc/premium-w2-a0bcbc-kserve-78dfd6d78c-lwth2" Nov 04 02:47:18.534785 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:47:18.534742 4405 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/33b620c1-389c-48b4-b277-90252018c4b6-dshm\") pod \"premium-w2-a0bcbc-kserve-78dfd6d78c-lwth2\" (UID: \"33b620c1-389c-48b4-b277-90252018c4b6\") " pod="e2e-models-e2e-worker-w2-a0bcbc/premium-w2-a0bcbc-kserve-78dfd6d78c-lwth2" Nov 04 02:47:18.594096 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:47:18.594038 4405 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-45xsf\" (UniqueName: \"kubernetes.io/projected/33b620c1-389c-48b4-b277-90252018c4b6-kube-api-access-45xsf\") pod \"premium-w2-a0bcbc-kserve-78dfd6d78c-lwth2\" (UID: \"33b620c1-389c-48b4-b277-90252018c4b6\") " pod="e2e-models-e2e-worker-w2-a0bcbc/premium-w2-a0bcbc-kserve-78dfd6d78c-lwth2" Nov 04 02:47:18.679979 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:47:18.679883 4405 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-a0bcbc/premium-w2-a0bcbc-kserve-78dfd6d78c-lwth2" Nov 04 02:47:19.071460 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:47:19.071432 4405 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-a0bcbc/premium-w2-a0bcbc-kserve-78dfd6d78c-lwth2"] Nov 04 02:47:19.073519 ip-10-0-134-125 kubenswrapper[4405]: W1104 02:47:19.073493 4405 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod33b620c1_389c_48b4_b277_90252018c4b6.slice/crio-ef6737ae624d86a300660bbba32e94d789f873fd430f2a839933742afa1ddc0b WatchSource:0}: Error finding container ef6737ae624d86a300660bbba32e94d789f873fd430f2a839933742afa1ddc0b: Status 404 returned error can't find the container with id ef6737ae624d86a300660bbba32e94d789f873fd430f2a839933742afa1ddc0b Nov 04 02:47:19.604282 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:47:19.604249 4405 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-a0bcbc/premium-w2-a0bcbc-kserve-78dfd6d78c-lwth2" event={"ID":"33b620c1-389c-48b4-b277-90252018c4b6","Type":"ContainerStarted","Data":"1249ce7eac538936bf2968a97afdd41b3b45bf0686f95dc6312213addef6436f"} Nov 04 02:47:19.604282 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:47:19.604288 4405 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-a0bcbc/premium-w2-a0bcbc-kserve-78dfd6d78c-lwth2" event={"ID":"33b620c1-389c-48b4-b277-90252018c4b6","Type":"ContainerStarted","Data":"ef6737ae624d86a300660bbba32e94d789f873fd430f2a839933742afa1ddc0b"} Nov 04 02:47:19.680218 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:47:19.680184 4405 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-a0bcbc/premium-w2-a0bcbc-kserve-78dfd6d78c-lwth2" Nov 04 02:47:19.721543 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:47:19.721486 4405 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w2-a0bcbc/premium-w2-a0bcbc-kserve-78dfd6d78c-lwth2" podStartSLOduration=1.72147099 podStartE2EDuration="1.72147099s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 02:47:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 02:47:19.719060189 +0000 UTC m=+886.149760308" watchObservedRunningTime="2026-11-04 02:47:19.72147099 +0000 UTC m=+886.152171110" Nov 04 02:47:28.680809 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:47:28.680770 4405 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w2-a0bcbc/premium-w2-a0bcbc-kserve-78dfd6d78c-lwth2" Nov 04 02:47:28.694262 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:47:28.694234 4405 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w2-a0bcbc/premium-w2-a0bcbc-kserve-78dfd6d78c-lwth2" Nov 04 02:47:29.669140 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:47:29.669109 4405 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-a0bcbc/premium-w2-a0bcbc-kserve-78dfd6d78c-lwth2" Nov 04 02:47:33.770396 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:47:33.770366 4405 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-vwcx8_ffbb2d83-5a93-4a5b-bce3-e40afb8f58ea/ovn-acl-logging/0.log" Nov 04 02:47:33.772938 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:47:33.772916 4405 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-vwcx8_ffbb2d83-5a93-4a5b-bce3-e40afb8f58ea/ovn-acl-logging/0.log" Nov 04 02:47:36.882253 ip-10-0-134-125 kubenswrapper[4405]: E1104 02:47:36.882218 4405 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5f2vb" containerName="extractor" Nov 04 02:47:41.149789 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:47:41.149751 4405 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-bl5mg"] Nov 04 02:47:46.202977 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:47:46.202932 4405 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-fe4e0c-openshift-default-c7f4fc84c-6hhrb"] Nov 04 02:47:46.207113 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:47:46.207089 4405 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-fe4e0c-openshift-default-c7f4fc84c-6hhrb" Nov 04 02:47:46.259223 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:47:46.259194 4405 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w0-fe4e0c-openshift-default-dockercfg-gx9rt\"" Nov 04 02:47:46.312135 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:47:46.312102 4405 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-fe4e0c-openshift-default-c7f4fc84c-6hhrb"] Nov 04 02:47:46.322043 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:47:46.322013 4405 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/40a233af-2ad7-4c33-85b2-a2e251d7ea3e-workload-certs\") pod \"e2e-worker-w0-fe4e0c-openshift-default-c7f4fc84c-6hhrb\" (UID: \"40a233af-2ad7-4c33-85b2-a2e251d7ea3e\") " pod="openshift-ingress/e2e-worker-w0-fe4e0c-openshift-default-c7f4fc84c-6hhrb" Nov 04 02:47:46.322225 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:47:46.322056 4405 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/40a233af-2ad7-4c33-85b2-a2e251d7ea3e-istio-data\") pod \"e2e-worker-w0-fe4e0c-openshift-default-c7f4fc84c-6hhrb\" (UID: \"40a233af-2ad7-4c33-85b2-a2e251d7ea3e\") " pod="openshift-ingress/e2e-worker-w0-fe4e0c-openshift-default-c7f4fc84c-6hhrb" Nov 04 02:47:46.322225 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:47:46.322083 4405 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/40a233af-2ad7-4c33-85b2-a2e251d7ea3e-credential-socket\") pod \"e2e-worker-w0-fe4e0c-openshift-default-c7f4fc84c-6hhrb\" (UID: \"40a233af-2ad7-4c33-85b2-a2e251d7ea3e\") " pod="openshift-ingress/e2e-worker-w0-fe4e0c-openshift-default-c7f4fc84c-6hhrb" Nov 04 02:47:46.322346 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:47:46.322234 4405 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/40a233af-2ad7-4c33-85b2-a2e251d7ea3e-istio-podinfo\") pod \"e2e-worker-w0-fe4e0c-openshift-default-c7f4fc84c-6hhrb\" (UID: \"40a233af-2ad7-4c33-85b2-a2e251d7ea3e\") " pod="openshift-ingress/e2e-worker-w0-fe4e0c-openshift-default-c7f4fc84c-6hhrb" Nov 04 02:47:46.322346 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:47:46.322275 4405 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rbn5g\" (UniqueName: \"kubernetes.io/projected/40a233af-2ad7-4c33-85b2-a2e251d7ea3e-kube-api-access-rbn5g\") pod \"e2e-worker-w0-fe4e0c-openshift-default-c7f4fc84c-6hhrb\" (UID: \"40a233af-2ad7-4c33-85b2-a2e251d7ea3e\") " pod="openshift-ingress/e2e-worker-w0-fe4e0c-openshift-default-c7f4fc84c-6hhrb" Nov 04 02:47:46.322463 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:47:46.322368 4405 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/40a233af-2ad7-4c33-85b2-a2e251d7ea3e-workload-socket\") pod \"e2e-worker-w0-fe4e0c-openshift-default-c7f4fc84c-6hhrb\" (UID: \"40a233af-2ad7-4c33-85b2-a2e251d7ea3e\") " pod="openshift-ingress/e2e-worker-w0-fe4e0c-openshift-default-c7f4fc84c-6hhrb" Nov 04 02:47:46.322463 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:47:46.322406 4405 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/40a233af-2ad7-4c33-85b2-a2e251d7ea3e-istio-envoy\") pod \"e2e-worker-w0-fe4e0c-openshift-default-c7f4fc84c-6hhrb\" (UID: \"40a233af-2ad7-4c33-85b2-a2e251d7ea3e\") " pod="openshift-ingress/e2e-worker-w0-fe4e0c-openshift-default-c7f4fc84c-6hhrb" Nov 04 02:47:46.322595 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:47:46.322574 4405 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/40a233af-2ad7-4c33-85b2-a2e251d7ea3e-istio-token\") pod \"e2e-worker-w0-fe4e0c-openshift-default-c7f4fc84c-6hhrb\" (UID: \"40a233af-2ad7-4c33-85b2-a2e251d7ea3e\") " pod="openshift-ingress/e2e-worker-w0-fe4e0c-openshift-default-c7f4fc84c-6hhrb" Nov 04 02:47:46.322654 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:47:46.322607 4405 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/40a233af-2ad7-4c33-85b2-a2e251d7ea3e-istiod-ca-cert\") pod \"e2e-worker-w0-fe4e0c-openshift-default-c7f4fc84c-6hhrb\" (UID: \"40a233af-2ad7-4c33-85b2-a2e251d7ea3e\") " pod="openshift-ingress/e2e-worker-w0-fe4e0c-openshift-default-c7f4fc84c-6hhrb" Nov 04 02:47:46.423334 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:47:46.423295 4405 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/40a233af-2ad7-4c33-85b2-a2e251d7ea3e-istio-token\") pod \"e2e-worker-w0-fe4e0c-openshift-default-c7f4fc84c-6hhrb\" (UID: \"40a233af-2ad7-4c33-85b2-a2e251d7ea3e\") " pod="openshift-ingress/e2e-worker-w0-fe4e0c-openshift-default-c7f4fc84c-6hhrb" Nov 04 02:47:46.423334 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:47:46.423337 4405 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/40a233af-2ad7-4c33-85b2-a2e251d7ea3e-istiod-ca-cert\") pod \"e2e-worker-w0-fe4e0c-openshift-default-c7f4fc84c-6hhrb\" (UID: \"40a233af-2ad7-4c33-85b2-a2e251d7ea3e\") " pod="openshift-ingress/e2e-worker-w0-fe4e0c-openshift-default-c7f4fc84c-6hhrb" Nov 04 02:47:46.423592 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:47:46.423374 4405 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/40a233af-2ad7-4c33-85b2-a2e251d7ea3e-workload-certs\") pod \"e2e-worker-w0-fe4e0c-openshift-default-c7f4fc84c-6hhrb\" (UID: \"40a233af-2ad7-4c33-85b2-a2e251d7ea3e\") " pod="openshift-ingress/e2e-worker-w0-fe4e0c-openshift-default-c7f4fc84c-6hhrb" Nov 04 02:47:46.423592 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:47:46.423409 4405 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/40a233af-2ad7-4c33-85b2-a2e251d7ea3e-istio-data\") pod \"e2e-worker-w0-fe4e0c-openshift-default-c7f4fc84c-6hhrb\" (UID: \"40a233af-2ad7-4c33-85b2-a2e251d7ea3e\") " pod="openshift-ingress/e2e-worker-w0-fe4e0c-openshift-default-c7f4fc84c-6hhrb" Nov 04 02:47:46.423592 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:47:46.423445 4405 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/40a233af-2ad7-4c33-85b2-a2e251d7ea3e-credential-socket\") pod \"e2e-worker-w0-fe4e0c-openshift-default-c7f4fc84c-6hhrb\" (UID: \"40a233af-2ad7-4c33-85b2-a2e251d7ea3e\") " pod="openshift-ingress/e2e-worker-w0-fe4e0c-openshift-default-c7f4fc84c-6hhrb" Nov 04 02:47:46.423592 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:47:46.423492 4405 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/40a233af-2ad7-4c33-85b2-a2e251d7ea3e-istio-podinfo\") pod \"e2e-worker-w0-fe4e0c-openshift-default-c7f4fc84c-6hhrb\" (UID: \"40a233af-2ad7-4c33-85b2-a2e251d7ea3e\") " pod="openshift-ingress/e2e-worker-w0-fe4e0c-openshift-default-c7f4fc84c-6hhrb" Nov 04 02:47:46.423592 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:47:46.423515 4405 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rbn5g\" (UniqueName: \"kubernetes.io/projected/40a233af-2ad7-4c33-85b2-a2e251d7ea3e-kube-api-access-rbn5g\") pod \"e2e-worker-w0-fe4e0c-openshift-default-c7f4fc84c-6hhrb\" (UID: \"40a233af-2ad7-4c33-85b2-a2e251d7ea3e\") " pod="openshift-ingress/e2e-worker-w0-fe4e0c-openshift-default-c7f4fc84c-6hhrb" Nov 04 02:47:46.423592 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:47:46.423575 4405 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/40a233af-2ad7-4c33-85b2-a2e251d7ea3e-workload-socket\") pod \"e2e-worker-w0-fe4e0c-openshift-default-c7f4fc84c-6hhrb\" (UID: \"40a233af-2ad7-4c33-85b2-a2e251d7ea3e\") " pod="openshift-ingress/e2e-worker-w0-fe4e0c-openshift-default-c7f4fc84c-6hhrb" Nov 04 02:47:46.423984 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:47:46.423622 4405 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/40a233af-2ad7-4c33-85b2-a2e251d7ea3e-istio-envoy\") pod \"e2e-worker-w0-fe4e0c-openshift-default-c7f4fc84c-6hhrb\" (UID: \"40a233af-2ad7-4c33-85b2-a2e251d7ea3e\") " pod="openshift-ingress/e2e-worker-w0-fe4e0c-openshift-default-c7f4fc84c-6hhrb" Nov 04 02:47:46.423984 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:47:46.423928 4405 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/40a233af-2ad7-4c33-85b2-a2e251d7ea3e-credential-socket\") pod \"e2e-worker-w0-fe4e0c-openshift-default-c7f4fc84c-6hhrb\" (UID: \"40a233af-2ad7-4c33-85b2-a2e251d7ea3e\") " pod="openshift-ingress/e2e-worker-w0-fe4e0c-openshift-default-c7f4fc84c-6hhrb" Nov 04 02:47:46.423984 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:47:46.423954 4405 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/40a233af-2ad7-4c33-85b2-a2e251d7ea3e-workload-certs\") pod \"e2e-worker-w0-fe4e0c-openshift-default-c7f4fc84c-6hhrb\" (UID: \"40a233af-2ad7-4c33-85b2-a2e251d7ea3e\") " pod="openshift-ingress/e2e-worker-w0-fe4e0c-openshift-default-c7f4fc84c-6hhrb" Nov 04 02:47:46.424149 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:47:46.424044 4405 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/40a233af-2ad7-4c33-85b2-a2e251d7ea3e-istio-data\") pod \"e2e-worker-w0-fe4e0c-openshift-default-c7f4fc84c-6hhrb\" (UID: \"40a233af-2ad7-4c33-85b2-a2e251d7ea3e\") " pod="openshift-ingress/e2e-worker-w0-fe4e0c-openshift-default-c7f4fc84c-6hhrb" Nov 04 02:47:46.424149 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:47:46.424096 4405 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/40a233af-2ad7-4c33-85b2-a2e251d7ea3e-workload-socket\") pod \"e2e-worker-w0-fe4e0c-openshift-default-c7f4fc84c-6hhrb\" (UID: \"40a233af-2ad7-4c33-85b2-a2e251d7ea3e\") " pod="openshift-ingress/e2e-worker-w0-fe4e0c-openshift-default-c7f4fc84c-6hhrb" Nov 04 02:47:46.424239 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:47:46.424203 4405 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/40a233af-2ad7-4c33-85b2-a2e251d7ea3e-istiod-ca-cert\") pod \"e2e-worker-w0-fe4e0c-openshift-default-c7f4fc84c-6hhrb\" (UID: \"40a233af-2ad7-4c33-85b2-a2e251d7ea3e\") " pod="openshift-ingress/e2e-worker-w0-fe4e0c-openshift-default-c7f4fc84c-6hhrb" Nov 04 02:47:46.426219 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:47:46.426196 4405 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/40a233af-2ad7-4c33-85b2-a2e251d7ea3e-istio-envoy\") pod \"e2e-worker-w0-fe4e0c-openshift-default-c7f4fc84c-6hhrb\" (UID: \"40a233af-2ad7-4c33-85b2-a2e251d7ea3e\") " pod="openshift-ingress/e2e-worker-w0-fe4e0c-openshift-default-c7f4fc84c-6hhrb" Nov 04 02:47:46.426638 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:47:46.426619 4405 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/40a233af-2ad7-4c33-85b2-a2e251d7ea3e-istio-podinfo\") pod \"e2e-worker-w0-fe4e0c-openshift-default-c7f4fc84c-6hhrb\" (UID: \"40a233af-2ad7-4c33-85b2-a2e251d7ea3e\") " pod="openshift-ingress/e2e-worker-w0-fe4e0c-openshift-default-c7f4fc84c-6hhrb" Nov 04 02:47:46.484153 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:47:46.484082 4405 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/40a233af-2ad7-4c33-85b2-a2e251d7ea3e-istio-token\") pod \"e2e-worker-w0-fe4e0c-openshift-default-c7f4fc84c-6hhrb\" (UID: \"40a233af-2ad7-4c33-85b2-a2e251d7ea3e\") " pod="openshift-ingress/e2e-worker-w0-fe4e0c-openshift-default-c7f4fc84c-6hhrb" Nov 04 02:47:46.484280 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:47:46.484198 4405 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rbn5g\" (UniqueName: \"kubernetes.io/projected/40a233af-2ad7-4c33-85b2-a2e251d7ea3e-kube-api-access-rbn5g\") pod \"e2e-worker-w0-fe4e0c-openshift-default-c7f4fc84c-6hhrb\" (UID: \"40a233af-2ad7-4c33-85b2-a2e251d7ea3e\") " pod="openshift-ingress/e2e-worker-w0-fe4e0c-openshift-default-c7f4fc84c-6hhrb" Nov 04 02:47:46.520940 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:47:46.520906 4405 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-fe4e0c-openshift-default-c7f4fc84c-6hhrb" Nov 04 02:47:46.950253 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:47:46.950220 4405 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-fe4e0c-openshift-default-c7f4fc84c-6hhrb"] Nov 04 02:47:46.951884 ip-10-0-134-125 kubenswrapper[4405]: W1104 02:47:46.951847 4405 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod40a233af_2ad7_4c33_85b2_a2e251d7ea3e.slice/crio-c7ee253eb0b150b935c215cd852522e6f9156e8a6c0d0ee547d8640a3f901b86 WatchSource:0}: Error finding container c7ee253eb0b150b935c215cd852522e6f9156e8a6c0d0ee547d8640a3f901b86: Status 404 returned error can't find the container with id c7ee253eb0b150b935c215cd852522e6f9156e8a6c0d0ee547d8640a3f901b86 Nov 04 02:47:46.954343 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:47:46.954312 4405 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 04 02:47:46.954440 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:47:46.954373 4405 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 04 02:47:46.954440 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:47:46.954400 4405 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 04 02:47:47.757833 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:47:47.757794 4405 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-fe4e0c-openshift-default-c7f4fc84c-6hhrb" event={"ID":"40a233af-2ad7-4c33-85b2-a2e251d7ea3e","Type":"ContainerStarted","Data":"31d61c1873e1df2d29db741027dcf73fcb3dd3c14329a064a7f68b47cd9b0557"} Nov 04 02:47:47.758349 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:47:47.757855 4405 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-fe4e0c-openshift-default-c7f4fc84c-6hhrb" event={"ID":"40a233af-2ad7-4c33-85b2-a2e251d7ea3e","Type":"ContainerStarted","Data":"c7ee253eb0b150b935c215cd852522e6f9156e8a6c0d0ee547d8640a3f901b86"} Nov 04 02:47:47.878107 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:47:47.878038 4405 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-fe4e0c-openshift-default-c7f4fc84c-6hhrb" podStartSLOduration=1.878018653 podStartE2EDuration="1.878018653s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 02:47:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 02:47:47.877018984 +0000 UTC m=+914.307719103" watchObservedRunningTime="2026-11-04 02:47:47.878018653 +0000 UTC m=+914.308718777" Nov 04 02:47:48.521705 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:47:48.521619 4405 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-fe4e0c-openshift-default-c7f4fc84c-6hhrb" Nov 04 02:47:48.523421 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:47:48.523393 4405 patch_prober.go:29] interesting pod/e2e-worker-w0-fe4e0c-openshift-default-c7f4fc84c-6hhrb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.48:15021/healthz/ready\": dial tcp 10.134.0.48:15021: connect: connection refused" start-of-body= Nov 04 02:47:48.523529 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:47:48.523444 4405 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-fe4e0c-openshift-default-c7f4fc84c-6hhrb" podUID="40a233af-2ad7-4c33-85b2-a2e251d7ea3e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.48:15021/healthz/ready\": dial tcp 10.134.0.48:15021: connect: connection refused" Nov 04 02:47:49.521746 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:47:49.521711 4405 patch_prober.go:29] interesting pod/e2e-worker-w0-fe4e0c-openshift-default-c7f4fc84c-6hhrb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.48:15021/healthz/ready\": dial tcp 10.134.0.48:15021: connect: connection refused" start-of-body= Nov 04 02:47:49.522176 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:47:49.521765 4405 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-fe4e0c-openshift-default-c7f4fc84c-6hhrb" podUID="40a233af-2ad7-4c33-85b2-a2e251d7ea3e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.48:15021/healthz/ready\": dial tcp 10.134.0.48:15021: connect: connection refused" Nov 04 02:47:50.521646 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:47:50.521612 4405 patch_prober.go:29] interesting pod/e2e-worker-w0-fe4e0c-openshift-default-c7f4fc84c-6hhrb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.48:15021/healthz/ready\": dial tcp 10.134.0.48:15021: connect: connection refused" start-of-body= Nov 04 02:47:50.521852 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:47:50.521701 4405 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-fe4e0c-openshift-default-c7f4fc84c-6hhrb" podUID="40a233af-2ad7-4c33-85b2-a2e251d7ea3e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.48:15021/healthz/ready\": dial tcp 10.134.0.48:15021: connect: connection refused" Nov 04 02:47:51.521762 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:47:51.521722 4405 patch_prober.go:29] interesting pod/e2e-worker-w0-fe4e0c-openshift-default-c7f4fc84c-6hhrb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.48:15021/healthz/ready\": dial tcp 10.134.0.48:15021: connect: connection refused" start-of-body= Nov 04 02:47:51.521971 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:47:51.521780 4405 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-fe4e0c-openshift-default-c7f4fc84c-6hhrb" podUID="40a233af-2ad7-4c33-85b2-a2e251d7ea3e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.48:15021/healthz/ready\": dial tcp 10.134.0.48:15021: connect: connection refused" Nov 04 02:47:52.521416 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:47:52.521377 4405 patch_prober.go:29] interesting pod/e2e-worker-w0-fe4e0c-openshift-default-c7f4fc84c-6hhrb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.48:15021/healthz/ready\": dial tcp 10.134.0.48:15021: connect: connection refused" start-of-body= Nov 04 02:47:52.521600 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:47:52.521444 4405 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-fe4e0c-openshift-default-c7f4fc84c-6hhrb" podUID="40a233af-2ad7-4c33-85b2-a2e251d7ea3e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.48:15021/healthz/ready\": dial tcp 10.134.0.48:15021: connect: connection refused" Nov 04 02:47:53.522068 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:47:53.522027 4405 patch_prober.go:29] interesting pod/e2e-worker-w0-fe4e0c-openshift-default-c7f4fc84c-6hhrb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.48:15021/healthz/ready\": dial tcp 10.134.0.48:15021: connect: connection refused" start-of-body= Nov 04 02:47:53.522588 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:47:53.522089 4405 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-fe4e0c-openshift-default-c7f4fc84c-6hhrb" podUID="40a233af-2ad7-4c33-85b2-a2e251d7ea3e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.48:15021/healthz/ready\": dial tcp 10.134.0.48:15021: connect: connection refused" Nov 04 02:47:54.526049 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:47:54.526009 4405 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-fe4e0c-openshift-default-c7f4fc84c-6hhrb" Nov 04 02:47:54.526503 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:47:54.526320 4405 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-fe4e0c-openshift-default-c7f4fc84c-6hhrb" Nov 04 02:47:54.527019 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:47:54.527000 4405 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-fe4e0c-openshift-default-c7f4fc84c-6hhrb" Nov 04 02:48:00.767391 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:48:00.767359 4405 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-a0bcbc-openshift-default-65dc9d9b6c-m97sr"] Nov 04 02:48:00.771407 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:48:00.771383 4405 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-a0bcbc-openshift-default-65dc9d9b6c-m97sr" Nov 04 02:48:00.827993 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:48:00.827966 4405 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w2-a0bcbc-openshift-default-dockercfg-z8ns2\"" Nov 04 02:48:00.866046 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:48:00.866008 4405 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9a748832-c6ef-484f-8635-8c9148a38149-istio-envoy\") pod \"e2e-worker-w2-a0bcbc-openshift-default-65dc9d9b6c-m97sr\" (UID: \"9a748832-c6ef-484f-8635-8c9148a38149\") " pod="openshift-ingress/e2e-worker-w2-a0bcbc-openshift-default-65dc9d9b6c-m97sr" Nov 04 02:48:00.866263 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:48:00.866067 4405 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9a748832-c6ef-484f-8635-8c9148a38149-istio-podinfo\") pod \"e2e-worker-w2-a0bcbc-openshift-default-65dc9d9b6c-m97sr\" (UID: \"9a748832-c6ef-484f-8635-8c9148a38149\") " pod="openshift-ingress/e2e-worker-w2-a0bcbc-openshift-default-65dc9d9b6c-m97sr" Nov 04 02:48:00.866263 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:48:00.866108 4405 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9a748832-c6ef-484f-8635-8c9148a38149-istio-token\") pod \"e2e-worker-w2-a0bcbc-openshift-default-65dc9d9b6c-m97sr\" (UID: \"9a748832-c6ef-484f-8635-8c9148a38149\") " pod="openshift-ingress/e2e-worker-w2-a0bcbc-openshift-default-65dc9d9b6c-m97sr" Nov 04 02:48:00.866263 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:48:00.866134 4405 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9a748832-c6ef-484f-8635-8c9148a38149-istiod-ca-cert\") pod \"e2e-worker-w2-a0bcbc-openshift-default-65dc9d9b6c-m97sr\" (UID: \"9a748832-c6ef-484f-8635-8c9148a38149\") " pod="openshift-ingress/e2e-worker-w2-a0bcbc-openshift-default-65dc9d9b6c-m97sr" Nov 04 02:48:00.866263 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:48:00.866166 4405 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9a748832-c6ef-484f-8635-8c9148a38149-workload-socket\") pod \"e2e-worker-w2-a0bcbc-openshift-default-65dc9d9b6c-m97sr\" (UID: \"9a748832-c6ef-484f-8635-8c9148a38149\") " pod="openshift-ingress/e2e-worker-w2-a0bcbc-openshift-default-65dc9d9b6c-m97sr" Nov 04 02:48:00.866263 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:48:00.866206 4405 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9a748832-c6ef-484f-8635-8c9148a38149-workload-certs\") pod \"e2e-worker-w2-a0bcbc-openshift-default-65dc9d9b6c-m97sr\" (UID: \"9a748832-c6ef-484f-8635-8c9148a38149\") " pod="openshift-ingress/e2e-worker-w2-a0bcbc-openshift-default-65dc9d9b6c-m97sr" Nov 04 02:48:00.866496 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:48:00.866300 4405 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9a748832-c6ef-484f-8635-8c9148a38149-istio-data\") pod \"e2e-worker-w2-a0bcbc-openshift-default-65dc9d9b6c-m97sr\" (UID: \"9a748832-c6ef-484f-8635-8c9148a38149\") " pod="openshift-ingress/e2e-worker-w2-a0bcbc-openshift-default-65dc9d9b6c-m97sr" Nov 04 02:48:00.866496 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:48:00.866335 4405 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9a748832-c6ef-484f-8635-8c9148a38149-credential-socket\") pod \"e2e-worker-w2-a0bcbc-openshift-default-65dc9d9b6c-m97sr\" (UID: \"9a748832-c6ef-484f-8635-8c9148a38149\") " pod="openshift-ingress/e2e-worker-w2-a0bcbc-openshift-default-65dc9d9b6c-m97sr" Nov 04 02:48:00.866496 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:48:00.866401 4405 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8t7g7\" (UniqueName: \"kubernetes.io/projected/9a748832-c6ef-484f-8635-8c9148a38149-kube-api-access-8t7g7\") pod \"e2e-worker-w2-a0bcbc-openshift-default-65dc9d9b6c-m97sr\" (UID: \"9a748832-c6ef-484f-8635-8c9148a38149\") " pod="openshift-ingress/e2e-worker-w2-a0bcbc-openshift-default-65dc9d9b6c-m97sr" Nov 04 02:48:00.879731 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:48:00.879699 4405 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-a0bcbc-openshift-default-65dc9d9b6c-m97sr"] Nov 04 02:48:00.967599 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:48:00.967562 4405 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9a748832-c6ef-484f-8635-8c9148a38149-workload-certs\") pod \"e2e-worker-w2-a0bcbc-openshift-default-65dc9d9b6c-m97sr\" (UID: \"9a748832-c6ef-484f-8635-8c9148a38149\") " pod="openshift-ingress/e2e-worker-w2-a0bcbc-openshift-default-65dc9d9b6c-m97sr" Nov 04 02:48:00.967811 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:48:00.967643 4405 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9a748832-c6ef-484f-8635-8c9148a38149-istio-data\") pod \"e2e-worker-w2-a0bcbc-openshift-default-65dc9d9b6c-m97sr\" (UID: \"9a748832-c6ef-484f-8635-8c9148a38149\") " pod="openshift-ingress/e2e-worker-w2-a0bcbc-openshift-default-65dc9d9b6c-m97sr" Nov 04 02:48:00.967811 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:48:00.967678 4405 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9a748832-c6ef-484f-8635-8c9148a38149-credential-socket\") pod \"e2e-worker-w2-a0bcbc-openshift-default-65dc9d9b6c-m97sr\" (UID: \"9a748832-c6ef-484f-8635-8c9148a38149\") " pod="openshift-ingress/e2e-worker-w2-a0bcbc-openshift-default-65dc9d9b6c-m97sr" Nov 04 02:48:00.967811 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:48:00.967719 4405 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8t7g7\" (UniqueName: \"kubernetes.io/projected/9a748832-c6ef-484f-8635-8c9148a38149-kube-api-access-8t7g7\") pod \"e2e-worker-w2-a0bcbc-openshift-default-65dc9d9b6c-m97sr\" (UID: \"9a748832-c6ef-484f-8635-8c9148a38149\") " pod="openshift-ingress/e2e-worker-w2-a0bcbc-openshift-default-65dc9d9b6c-m97sr" Nov 04 02:48:00.967811 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:48:00.967783 4405 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9a748832-c6ef-484f-8635-8c9148a38149-istio-envoy\") pod \"e2e-worker-w2-a0bcbc-openshift-default-65dc9d9b6c-m97sr\" (UID: \"9a748832-c6ef-484f-8635-8c9148a38149\") " pod="openshift-ingress/e2e-worker-w2-a0bcbc-openshift-default-65dc9d9b6c-m97sr" Nov 04 02:48:00.968030 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:48:00.967814 4405 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9a748832-c6ef-484f-8635-8c9148a38149-istio-podinfo\") pod \"e2e-worker-w2-a0bcbc-openshift-default-65dc9d9b6c-m97sr\" (UID: \"9a748832-c6ef-484f-8635-8c9148a38149\") " pod="openshift-ingress/e2e-worker-w2-a0bcbc-openshift-default-65dc9d9b6c-m97sr" Nov 04 02:48:00.968030 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:48:00.967855 4405 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9a748832-c6ef-484f-8635-8c9148a38149-istio-token\") pod \"e2e-worker-w2-a0bcbc-openshift-default-65dc9d9b6c-m97sr\" (UID: \"9a748832-c6ef-484f-8635-8c9148a38149\") " pod="openshift-ingress/e2e-worker-w2-a0bcbc-openshift-default-65dc9d9b6c-m97sr" Nov 04 02:48:00.968030 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:48:00.967880 4405 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9a748832-c6ef-484f-8635-8c9148a38149-istiod-ca-cert\") pod \"e2e-worker-w2-a0bcbc-openshift-default-65dc9d9b6c-m97sr\" (UID: \"9a748832-c6ef-484f-8635-8c9148a38149\") " pod="openshift-ingress/e2e-worker-w2-a0bcbc-openshift-default-65dc9d9b6c-m97sr" Nov 04 02:48:00.968030 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:48:00.967914 4405 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9a748832-c6ef-484f-8635-8c9148a38149-workload-socket\") pod \"e2e-worker-w2-a0bcbc-openshift-default-65dc9d9b6c-m97sr\" (UID: \"9a748832-c6ef-484f-8635-8c9148a38149\") " pod="openshift-ingress/e2e-worker-w2-a0bcbc-openshift-default-65dc9d9b6c-m97sr" Nov 04 02:48:00.968241 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:48:00.968094 4405 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9a748832-c6ef-484f-8635-8c9148a38149-istio-data\") pod \"e2e-worker-w2-a0bcbc-openshift-default-65dc9d9b6c-m97sr\" (UID: \"9a748832-c6ef-484f-8635-8c9148a38149\") " pod="openshift-ingress/e2e-worker-w2-a0bcbc-openshift-default-65dc9d9b6c-m97sr" Nov 04 02:48:00.968420 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:48:00.968387 4405 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9a748832-c6ef-484f-8635-8c9148a38149-workload-socket\") pod \"e2e-worker-w2-a0bcbc-openshift-default-65dc9d9b6c-m97sr\" (UID: \"9a748832-c6ef-484f-8635-8c9148a38149\") " pod="openshift-ingress/e2e-worker-w2-a0bcbc-openshift-default-65dc9d9b6c-m97sr" Nov 04 02:48:00.968534 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:48:00.968437 4405 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9a748832-c6ef-484f-8635-8c9148a38149-workload-certs\") pod \"e2e-worker-w2-a0bcbc-openshift-default-65dc9d9b6c-m97sr\" (UID: \"9a748832-c6ef-484f-8635-8c9148a38149\") " pod="openshift-ingress/e2e-worker-w2-a0bcbc-openshift-default-65dc9d9b6c-m97sr" Nov 04 02:48:00.968534 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:48:00.968403 4405 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9a748832-c6ef-484f-8635-8c9148a38149-credential-socket\") pod \"e2e-worker-w2-a0bcbc-openshift-default-65dc9d9b6c-m97sr\" (UID: \"9a748832-c6ef-484f-8635-8c9148a38149\") " pod="openshift-ingress/e2e-worker-w2-a0bcbc-openshift-default-65dc9d9b6c-m97sr" Nov 04 02:48:00.969139 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:48:00.969110 4405 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9a748832-c6ef-484f-8635-8c9148a38149-istiod-ca-cert\") pod \"e2e-worker-w2-a0bcbc-openshift-default-65dc9d9b6c-m97sr\" (UID: \"9a748832-c6ef-484f-8635-8c9148a38149\") " pod="openshift-ingress/e2e-worker-w2-a0bcbc-openshift-default-65dc9d9b6c-m97sr" Nov 04 02:48:00.970621 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:48:00.970597 4405 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9a748832-c6ef-484f-8635-8c9148a38149-istio-envoy\") pod \"e2e-worker-w2-a0bcbc-openshift-default-65dc9d9b6c-m97sr\" (UID: \"9a748832-c6ef-484f-8635-8c9148a38149\") " pod="openshift-ingress/e2e-worker-w2-a0bcbc-openshift-default-65dc9d9b6c-m97sr" Nov 04 02:48:00.970774 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:48:00.970754 4405 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9a748832-c6ef-484f-8635-8c9148a38149-istio-podinfo\") pod \"e2e-worker-w2-a0bcbc-openshift-default-65dc9d9b6c-m97sr\" (UID: \"9a748832-c6ef-484f-8635-8c9148a38149\") " pod="openshift-ingress/e2e-worker-w2-a0bcbc-openshift-default-65dc9d9b6c-m97sr" Nov 04 02:48:01.029129 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:48:01.029047 4405 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9a748832-c6ef-484f-8635-8c9148a38149-istio-token\") pod \"e2e-worker-w2-a0bcbc-openshift-default-65dc9d9b6c-m97sr\" (UID: \"9a748832-c6ef-484f-8635-8c9148a38149\") " pod="openshift-ingress/e2e-worker-w2-a0bcbc-openshift-default-65dc9d9b6c-m97sr" Nov 04 02:48:01.029290 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:48:01.029223 4405 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8t7g7\" (UniqueName: \"kubernetes.io/projected/9a748832-c6ef-484f-8635-8c9148a38149-kube-api-access-8t7g7\") pod \"e2e-worker-w2-a0bcbc-openshift-default-65dc9d9b6c-m97sr\" (UID: \"9a748832-c6ef-484f-8635-8c9148a38149\") " pod="openshift-ingress/e2e-worker-w2-a0bcbc-openshift-default-65dc9d9b6c-m97sr" Nov 04 02:48:01.085941 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:48:01.085902 4405 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-a0bcbc-openshift-default-65dc9d9b6c-m97sr" Nov 04 02:48:01.526124 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:48:01.526091 4405 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-a0bcbc-openshift-default-65dc9d9b6c-m97sr"] Nov 04 02:48:01.528265 ip-10-0-134-125 kubenswrapper[4405]: W1104 02:48:01.528233 4405 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9a748832_c6ef_484f_8635_8c9148a38149.slice/crio-d11d2dd3276a99552ac98a263095e432b9c548215c60471ca0588f74928e4a05 WatchSource:0}: Error finding container d11d2dd3276a99552ac98a263095e432b9c548215c60471ca0588f74928e4a05: Status 404 returned error can't find the container with id d11d2dd3276a99552ac98a263095e432b9c548215c60471ca0588f74928e4a05 Nov 04 02:48:01.530614 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:48:01.530584 4405 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 04 02:48:01.530730 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:48:01.530651 4405 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 04 02:48:01.530780 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:48:01.530727 4405 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 04 02:48:01.871535 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:48:01.871496 4405 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-a0bcbc-openshift-default-65dc9d9b6c-m97sr" event={"ID":"9a748832-c6ef-484f-8635-8c9148a38149","Type":"ContainerStarted","Data":"a1661cfbc2299c62f5a5a787738fe1d5401ae1e8caf90694fa945ac361182bce"} Nov 04 02:48:01.872101 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:48:01.871566 4405 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-a0bcbc-openshift-default-65dc9d9b6c-m97sr" event={"ID":"9a748832-c6ef-484f-8635-8c9148a38149","Type":"ContainerStarted","Data":"d11d2dd3276a99552ac98a263095e432b9c548215c60471ca0588f74928e4a05"} Nov 04 02:48:01.993559 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:48:01.993511 4405 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-a0bcbc-openshift-default-65dc9d9b6c-m97sr" podStartSLOduration=1.993497762 podStartE2EDuration="1.993497762s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 02:48:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 02:48:01.993178461 +0000 UTC m=+928.423878581" watchObservedRunningTime="2026-11-04 02:48:01.993497762 +0000 UTC m=+928.424197882" Nov 04 02:48:02.086138 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:48:02.086100 4405 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-a0bcbc-openshift-default-65dc9d9b6c-m97sr" Nov 04 02:48:02.087942 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:48:02.087868 4405 patch_prober.go:29] interesting pod/e2e-worker-w2-a0bcbc-openshift-default-65dc9d9b6c-m97sr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.49:15021/healthz/ready\": dial tcp 10.134.0.49:15021: connect: connection refused" start-of-body= Nov 04 02:48:02.088106 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:48:02.087966 4405 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-a0bcbc-openshift-default-65dc9d9b6c-m97sr" podUID="9a748832-c6ef-484f-8635-8c9148a38149" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.49:15021/healthz/ready\": dial tcp 10.134.0.49:15021: connect: connection refused" Nov 04 02:48:03.086249 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:48:03.086215 4405 patch_prober.go:29] interesting pod/e2e-worker-w2-a0bcbc-openshift-default-65dc9d9b6c-m97sr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.49:15021/healthz/ready\": dial tcp 10.134.0.49:15021: connect: connection refused" start-of-body= Nov 04 02:48:03.086661 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:48:03.086271 4405 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-a0bcbc-openshift-default-65dc9d9b6c-m97sr" podUID="9a748832-c6ef-484f-8635-8c9148a38149" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.49:15021/healthz/ready\": dial tcp 10.134.0.49:15021: connect: connection refused" Nov 04 02:48:04.086499 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:48:04.086466 4405 patch_prober.go:29] interesting pod/e2e-worker-w2-a0bcbc-openshift-default-65dc9d9b6c-m97sr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.49:15021/healthz/ready\": dial tcp 10.134.0.49:15021: connect: connection refused" start-of-body= Nov 04 02:48:04.087029 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:48:04.086515 4405 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-a0bcbc-openshift-default-65dc9d9b6c-m97sr" podUID="9a748832-c6ef-484f-8635-8c9148a38149" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.49:15021/healthz/ready\": dial tcp 10.134.0.49:15021: connect: connection refused" Nov 04 02:48:05.086543 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:48:05.086505 4405 patch_prober.go:29] interesting pod/e2e-worker-w2-a0bcbc-openshift-default-65dc9d9b6c-m97sr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.49:15021/healthz/ready\": dial tcp 10.134.0.49:15021: connect: connection refused" start-of-body= Nov 04 02:48:05.087082 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:48:05.086566 4405 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-a0bcbc-openshift-default-65dc9d9b6c-m97sr" podUID="9a748832-c6ef-484f-8635-8c9148a38149" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.49:15021/healthz/ready\": dial tcp 10.134.0.49:15021: connect: connection refused" Nov 04 02:48:06.086596 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:48:06.086551 4405 patch_prober.go:29] interesting pod/e2e-worker-w2-a0bcbc-openshift-default-65dc9d9b6c-m97sr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.49:15021/healthz/ready\": dial tcp 10.134.0.49:15021: connect: connection refused" start-of-body= Nov 04 02:48:06.087121 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:48:06.086618 4405 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-a0bcbc-openshift-default-65dc9d9b6c-m97sr" podUID="9a748832-c6ef-484f-8635-8c9148a38149" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.49:15021/healthz/ready\": dial tcp 10.134.0.49:15021: connect: connection refused" Nov 04 02:48:06.882960 ip-10-0-134-125 kubenswrapper[4405]: E1104 02:48:06.882927 4405 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5f2vb" containerName="extractor" Nov 04 02:48:07.086276 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:48:07.086237 4405 patch_prober.go:29] interesting pod/e2e-worker-w2-a0bcbc-openshift-default-65dc9d9b6c-m97sr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.49:15021/healthz/ready\": dial tcp 10.134.0.49:15021: connect: connection refused" start-of-body= Nov 04 02:48:07.086456 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:48:07.086297 4405 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-a0bcbc-openshift-default-65dc9d9b6c-m97sr" podUID="9a748832-c6ef-484f-8635-8c9148a38149" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.49:15021/healthz/ready\": dial tcp 10.134.0.49:15021: connect: connection refused" Nov 04 02:48:08.086345 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:48:08.086316 4405 patch_prober.go:29] interesting pod/e2e-worker-w2-a0bcbc-openshift-default-65dc9d9b6c-m97sr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.49:15021/healthz/ready\": dial tcp 10.134.0.49:15021: connect: connection refused" start-of-body= Nov 04 02:48:08.086859 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:48:08.086374 4405 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-a0bcbc-openshift-default-65dc9d9b6c-m97sr" podUID="9a748832-c6ef-484f-8635-8c9148a38149" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.49:15021/healthz/ready\": dial tcp 10.134.0.49:15021: connect: connection refused" Nov 04 02:48:09.091476 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:48:09.091445 4405 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-a0bcbc-openshift-default-65dc9d9b6c-m97sr" Nov 04 02:48:09.091986 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:48:09.091798 4405 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-a0bcbc-openshift-default-65dc9d9b6c-m97sr" Nov 04 02:48:09.092529 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:48:09.092509 4405 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-a0bcbc-openshift-default-65dc9d9b6c-m97sr" Nov 04 02:48:14.325557 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:48:14.325514 4405 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-bl5mg"] Nov 04 02:48:36.883569 ip-10-0-134-125 kubenswrapper[4405]: E1104 02:48:36.883535 4405 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5f2vb" containerName="extractor" Nov 04 02:48:59.915252 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:48:59.915218 4405 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-bl5mg"] Nov 04 02:49:02.884217 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:49:02.884184 4405 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-76597d6855-4ndqx"] Nov 04 02:49:02.884837 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:49:02.884388 4405 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-4ndqx" podUID="a236d45b-5615-4d69-9043-52de98c2ab38" containerName="istio-proxy" containerID="cri-o://9f736a640f6a81dee067ff6beb11f61696cd002aa50f12332518b080fef961c0" gracePeriod=30 Nov 04 02:49:02.933900 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:49:02.933865 4405 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-76597d6855-tlkdl"] Nov 04 02:49:02.934109 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:49:02.934083 4405 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-tlkdl" podUID="45bb7186-84f1-4586-b4c1-50250365ff2b" containerName="istio-proxy" containerID="cri-o://dd73bfbe496cb2981cf8933215430460d8e7d58d9eda1d0d88dc5bdfcb4ceb2e" gracePeriod=30 Nov 04 02:49:03.108230 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:49:03.108195 4405 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w6-ebb7fd-openshift-default-544f5768f-w6tt7"] Nov 04 02:49:03.112434 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:49:03.112410 4405 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-ebb7fd-openshift-default-544f5768f-w6tt7" Nov 04 02:49:03.170191 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:49:03.170106 4405 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w6-ebb7fd-openshift-default-dockercfg-k9smx\"" Nov 04 02:49:03.177160 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:49:03.177124 4405 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-ebb7fd-openshift-default-544f5768f-w6tt7"] Nov 04 02:49:03.191637 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:49:03.191606 4405 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2522c184-4d83-40b4-9a42-cbc55b7a5a37-workload-socket\") pod \"e2e-worker-w6-ebb7fd-openshift-default-544f5768f-w6tt7\" (UID: \"2522c184-4d83-40b4-9a42-cbc55b7a5a37\") " pod="openshift-ingress/e2e-worker-w6-ebb7fd-openshift-default-544f5768f-w6tt7" Nov 04 02:49:03.191824 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:49:03.191685 4405 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2522c184-4d83-40b4-9a42-cbc55b7a5a37-istio-token\") pod \"e2e-worker-w6-ebb7fd-openshift-default-544f5768f-w6tt7\" (UID: \"2522c184-4d83-40b4-9a42-cbc55b7a5a37\") " pod="openshift-ingress/e2e-worker-w6-ebb7fd-openshift-default-544f5768f-w6tt7" Nov 04 02:49:03.191824 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:49:03.191781 4405 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2522c184-4d83-40b4-9a42-cbc55b7a5a37-istio-envoy\") pod \"e2e-worker-w6-ebb7fd-openshift-default-544f5768f-w6tt7\" (UID: \"2522c184-4d83-40b4-9a42-cbc55b7a5a37\") " pod="openshift-ingress/e2e-worker-w6-ebb7fd-openshift-default-544f5768f-w6tt7" Nov 04 02:49:03.191952 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:49:03.191851 4405 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2522c184-4d83-40b4-9a42-cbc55b7a5a37-workload-certs\") pod \"e2e-worker-w6-ebb7fd-openshift-default-544f5768f-w6tt7\" (UID: \"2522c184-4d83-40b4-9a42-cbc55b7a5a37\") " pod="openshift-ingress/e2e-worker-w6-ebb7fd-openshift-default-544f5768f-w6tt7" Nov 04 02:49:03.191952 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:49:03.191875 4405 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2522c184-4d83-40b4-9a42-cbc55b7a5a37-credential-socket\") pod \"e2e-worker-w6-ebb7fd-openshift-default-544f5768f-w6tt7\" (UID: \"2522c184-4d83-40b4-9a42-cbc55b7a5a37\") " pod="openshift-ingress/e2e-worker-w6-ebb7fd-openshift-default-544f5768f-w6tt7" Nov 04 02:49:03.192064 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:49:03.191939 4405 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2tvxc\" (UniqueName: \"kubernetes.io/projected/2522c184-4d83-40b4-9a42-cbc55b7a5a37-kube-api-access-2tvxc\") pod \"e2e-worker-w6-ebb7fd-openshift-default-544f5768f-w6tt7\" (UID: \"2522c184-4d83-40b4-9a42-cbc55b7a5a37\") " pod="openshift-ingress/e2e-worker-w6-ebb7fd-openshift-default-544f5768f-w6tt7" Nov 04 02:49:03.192119 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:49:03.192080 4405 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2522c184-4d83-40b4-9a42-cbc55b7a5a37-istiod-ca-cert\") pod \"e2e-worker-w6-ebb7fd-openshift-default-544f5768f-w6tt7\" (UID: \"2522c184-4d83-40b4-9a42-cbc55b7a5a37\") " pod="openshift-ingress/e2e-worker-w6-ebb7fd-openshift-default-544f5768f-w6tt7" Nov 04 02:49:03.192175 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:49:03.192122 4405 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2522c184-4d83-40b4-9a42-cbc55b7a5a37-istio-data\") pod \"e2e-worker-w6-ebb7fd-openshift-default-544f5768f-w6tt7\" (UID: \"2522c184-4d83-40b4-9a42-cbc55b7a5a37\") " pod="openshift-ingress/e2e-worker-w6-ebb7fd-openshift-default-544f5768f-w6tt7" Nov 04 02:49:03.192232 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:49:03.192173 4405 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2522c184-4d83-40b4-9a42-cbc55b7a5a37-istio-podinfo\") pod \"e2e-worker-w6-ebb7fd-openshift-default-544f5768f-w6tt7\" (UID: \"2522c184-4d83-40b4-9a42-cbc55b7a5a37\") " pod="openshift-ingress/e2e-worker-w6-ebb7fd-openshift-default-544f5768f-w6tt7" Nov 04 02:49:03.293565 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:49:03.293524 4405 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2522c184-4d83-40b4-9a42-cbc55b7a5a37-istio-token\") pod \"e2e-worker-w6-ebb7fd-openshift-default-544f5768f-w6tt7\" (UID: \"2522c184-4d83-40b4-9a42-cbc55b7a5a37\") " pod="openshift-ingress/e2e-worker-w6-ebb7fd-openshift-default-544f5768f-w6tt7" Nov 04 02:49:03.293565 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:49:03.293572 4405 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2522c184-4d83-40b4-9a42-cbc55b7a5a37-istio-envoy\") pod \"e2e-worker-w6-ebb7fd-openshift-default-544f5768f-w6tt7\" (UID: \"2522c184-4d83-40b4-9a42-cbc55b7a5a37\") " pod="openshift-ingress/e2e-worker-w6-ebb7fd-openshift-default-544f5768f-w6tt7" Nov 04 02:49:03.293855 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:49:03.293641 4405 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2522c184-4d83-40b4-9a42-cbc55b7a5a37-workload-certs\") pod \"e2e-worker-w6-ebb7fd-openshift-default-544f5768f-w6tt7\" (UID: \"2522c184-4d83-40b4-9a42-cbc55b7a5a37\") " pod="openshift-ingress/e2e-worker-w6-ebb7fd-openshift-default-544f5768f-w6tt7" Nov 04 02:49:03.293855 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:49:03.293662 4405 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2522c184-4d83-40b4-9a42-cbc55b7a5a37-credential-socket\") pod \"e2e-worker-w6-ebb7fd-openshift-default-544f5768f-w6tt7\" (UID: \"2522c184-4d83-40b4-9a42-cbc55b7a5a37\") " pod="openshift-ingress/e2e-worker-w6-ebb7fd-openshift-default-544f5768f-w6tt7" Nov 04 02:49:03.293855 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:49:03.293705 4405 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2tvxc\" (UniqueName: \"kubernetes.io/projected/2522c184-4d83-40b4-9a42-cbc55b7a5a37-kube-api-access-2tvxc\") pod \"e2e-worker-w6-ebb7fd-openshift-default-544f5768f-w6tt7\" (UID: \"2522c184-4d83-40b4-9a42-cbc55b7a5a37\") " pod="openshift-ingress/e2e-worker-w6-ebb7fd-openshift-default-544f5768f-w6tt7" Nov 04 02:49:03.293855 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:49:03.293761 4405 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2522c184-4d83-40b4-9a42-cbc55b7a5a37-istiod-ca-cert\") pod \"e2e-worker-w6-ebb7fd-openshift-default-544f5768f-w6tt7\" (UID: \"2522c184-4d83-40b4-9a42-cbc55b7a5a37\") " pod="openshift-ingress/e2e-worker-w6-ebb7fd-openshift-default-544f5768f-w6tt7" Nov 04 02:49:03.293855 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:49:03.293811 4405 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2522c184-4d83-40b4-9a42-cbc55b7a5a37-istio-data\") pod \"e2e-worker-w6-ebb7fd-openshift-default-544f5768f-w6tt7\" (UID: \"2522c184-4d83-40b4-9a42-cbc55b7a5a37\") " pod="openshift-ingress/e2e-worker-w6-ebb7fd-openshift-default-544f5768f-w6tt7" Nov 04 02:49:03.293855 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:49:03.293836 4405 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2522c184-4d83-40b4-9a42-cbc55b7a5a37-istio-podinfo\") pod \"e2e-worker-w6-ebb7fd-openshift-default-544f5768f-w6tt7\" (UID: \"2522c184-4d83-40b4-9a42-cbc55b7a5a37\") " pod="openshift-ingress/e2e-worker-w6-ebb7fd-openshift-default-544f5768f-w6tt7" Nov 04 02:49:03.294196 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:49:03.293875 4405 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2522c184-4d83-40b4-9a42-cbc55b7a5a37-workload-socket\") pod \"e2e-worker-w6-ebb7fd-openshift-default-544f5768f-w6tt7\" (UID: \"2522c184-4d83-40b4-9a42-cbc55b7a5a37\") " pod="openshift-ingress/e2e-worker-w6-ebb7fd-openshift-default-544f5768f-w6tt7" Nov 04 02:49:03.294196 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:49:03.294106 4405 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2522c184-4d83-40b4-9a42-cbc55b7a5a37-credential-socket\") pod \"e2e-worker-w6-ebb7fd-openshift-default-544f5768f-w6tt7\" (UID: \"2522c184-4d83-40b4-9a42-cbc55b7a5a37\") " pod="openshift-ingress/e2e-worker-w6-ebb7fd-openshift-default-544f5768f-w6tt7" Nov 04 02:49:03.294288 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:49:03.294198 4405 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2522c184-4d83-40b4-9a42-cbc55b7a5a37-workload-certs\") pod \"e2e-worker-w6-ebb7fd-openshift-default-544f5768f-w6tt7\" (UID: \"2522c184-4d83-40b4-9a42-cbc55b7a5a37\") " pod="openshift-ingress/e2e-worker-w6-ebb7fd-openshift-default-544f5768f-w6tt7" Nov 04 02:49:03.294288 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:49:03.294261 4405 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2522c184-4d83-40b4-9a42-cbc55b7a5a37-workload-socket\") pod \"e2e-worker-w6-ebb7fd-openshift-default-544f5768f-w6tt7\" (UID: \"2522c184-4d83-40b4-9a42-cbc55b7a5a37\") " pod="openshift-ingress/e2e-worker-w6-ebb7fd-openshift-default-544f5768f-w6tt7" Nov 04 02:49:03.294288 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:49:03.294282 4405 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2522c184-4d83-40b4-9a42-cbc55b7a5a37-istio-data\") pod \"e2e-worker-w6-ebb7fd-openshift-default-544f5768f-w6tt7\" (UID: \"2522c184-4d83-40b4-9a42-cbc55b7a5a37\") " pod="openshift-ingress/e2e-worker-w6-ebb7fd-openshift-default-544f5768f-w6tt7" Nov 04 02:49:03.294522 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:49:03.294502 4405 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2522c184-4d83-40b4-9a42-cbc55b7a5a37-istiod-ca-cert\") pod \"e2e-worker-w6-ebb7fd-openshift-default-544f5768f-w6tt7\" (UID: \"2522c184-4d83-40b4-9a42-cbc55b7a5a37\") " pod="openshift-ingress/e2e-worker-w6-ebb7fd-openshift-default-544f5768f-w6tt7" Nov 04 02:49:03.296261 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:49:03.296235 4405 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2522c184-4d83-40b4-9a42-cbc55b7a5a37-istio-envoy\") pod \"e2e-worker-w6-ebb7fd-openshift-default-544f5768f-w6tt7\" (UID: \"2522c184-4d83-40b4-9a42-cbc55b7a5a37\") " pod="openshift-ingress/e2e-worker-w6-ebb7fd-openshift-default-544f5768f-w6tt7" Nov 04 02:49:03.296453 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:49:03.296428 4405 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2522c184-4d83-40b4-9a42-cbc55b7a5a37-istio-podinfo\") pod \"e2e-worker-w6-ebb7fd-openshift-default-544f5768f-w6tt7\" (UID: \"2522c184-4d83-40b4-9a42-cbc55b7a5a37\") " pod="openshift-ingress/e2e-worker-w6-ebb7fd-openshift-default-544f5768f-w6tt7" Nov 04 02:49:03.355734 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:49:03.355700 4405 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2522c184-4d83-40b4-9a42-cbc55b7a5a37-istio-token\") pod \"e2e-worker-w6-ebb7fd-openshift-default-544f5768f-w6tt7\" (UID: \"2522c184-4d83-40b4-9a42-cbc55b7a5a37\") " pod="openshift-ingress/e2e-worker-w6-ebb7fd-openshift-default-544f5768f-w6tt7" Nov 04 02:49:03.355887 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:49:03.355776 4405 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2tvxc\" (UniqueName: \"kubernetes.io/projected/2522c184-4d83-40b4-9a42-cbc55b7a5a37-kube-api-access-2tvxc\") pod \"e2e-worker-w6-ebb7fd-openshift-default-544f5768f-w6tt7\" (UID: \"2522c184-4d83-40b4-9a42-cbc55b7a5a37\") " pod="openshift-ingress/e2e-worker-w6-ebb7fd-openshift-default-544f5768f-w6tt7" Nov 04 02:49:03.426476 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:49:03.426399 4405 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-ebb7fd-openshift-default-544f5768f-w6tt7" Nov 04 02:49:03.857456 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:49:03.857427 4405 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-ebb7fd-openshift-default-544f5768f-w6tt7"] Nov 04 02:49:03.858974 ip-10-0-134-125 kubenswrapper[4405]: W1104 02:49:03.858935 4405 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2522c184_4d83_40b4_9a42_cbc55b7a5a37.slice/crio-a4c0ac9aed61eb6e9c510ed2ebbd24fa93f2119d68340e041e91047a1ab68cf4 WatchSource:0}: Error finding container a4c0ac9aed61eb6e9c510ed2ebbd24fa93f2119d68340e041e91047a1ab68cf4: Status 404 returned error can't find the container with id a4c0ac9aed61eb6e9c510ed2ebbd24fa93f2119d68340e041e91047a1ab68cf4 Nov 04 02:49:03.861252 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:49:03.861220 4405 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 04 02:49:03.861333 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:49:03.861287 4405 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 04 02:49:03.861333 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:49:03.861330 4405 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 04 02:49:04.251375 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:49:04.251285 4405 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-ebb7fd-openshift-default-544f5768f-w6tt7" event={"ID":"2522c184-4d83-40b4-9a42-cbc55b7a5a37","Type":"ContainerStarted","Data":"83db20eac508cc7d5a8e7a6d62fdbd1c4ef770a08b559688768c7610becdf04f"} Nov 04 02:49:04.251375 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:49:04.251345 4405 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-ebb7fd-openshift-default-544f5768f-w6tt7" event={"ID":"2522c184-4d83-40b4-9a42-cbc55b7a5a37","Type":"ContainerStarted","Data":"a4c0ac9aed61eb6e9c510ed2ebbd24fa93f2119d68340e041e91047a1ab68cf4"} Nov 04 02:49:04.378297 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:49:04.378242 4405 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w6-ebb7fd-openshift-default-544f5768f-w6tt7" podStartSLOduration=1.378227171 podStartE2EDuration="1.378227171s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 02:49:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 02:49:04.376700866 +0000 UTC m=+990.807400985" watchObservedRunningTime="2026-11-04 02:49:04.378227171 +0000 UTC m=+990.808927331" Nov 04 02:49:04.427057 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:49:04.427026 4405 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w6-ebb7fd-openshift-default-544f5768f-w6tt7" Nov 04 02:49:04.428362 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:49:04.428339 4405 patch_prober.go:29] interesting pod/e2e-worker-w6-ebb7fd-openshift-default-544f5768f-w6tt7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.50:15021/healthz/ready\": dial tcp 10.134.0.50:15021: connect: connection refused" start-of-body= Nov 04 02:49:04.428417 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:49:04.428388 4405 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-ebb7fd-openshift-default-544f5768f-w6tt7" podUID="2522c184-4d83-40b4-9a42-cbc55b7a5a37" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.50:15021/healthz/ready\": dial tcp 10.134.0.50:15021: connect: connection refused" Nov 04 02:49:05.427727 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:49:05.427695 4405 patch_prober.go:29] interesting pod/e2e-worker-w6-ebb7fd-openshift-default-544f5768f-w6tt7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.50:15021/healthz/ready\": dial tcp 10.134.0.50:15021: connect: connection refused" start-of-body= Nov 04 02:49:05.428135 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:49:05.427751 4405 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-ebb7fd-openshift-default-544f5768f-w6tt7" podUID="2522c184-4d83-40b4-9a42-cbc55b7a5a37" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.50:15021/healthz/ready\": dial tcp 10.134.0.50:15021: connect: connection refused" Nov 04 02:49:06.427684 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:49:06.427626 4405 patch_prober.go:29] interesting pod/e2e-worker-w6-ebb7fd-openshift-default-544f5768f-w6tt7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.50:15021/healthz/ready\": dial tcp 10.134.0.50:15021: connect: connection refused" start-of-body= Nov 04 02:49:06.427988 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:49:06.427950 4405 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-ebb7fd-openshift-default-544f5768f-w6tt7" podUID="2522c184-4d83-40b4-9a42-cbc55b7a5a37" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.50:15021/healthz/ready\": dial tcp 10.134.0.50:15021: connect: connection refused" Nov 04 02:49:06.883474 ip-10-0-134-125 kubenswrapper[4405]: E1104 02:49:06.883436 4405 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5f2vb" containerName="extractor" Nov 04 02:49:07.427288 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:49:07.427247 4405 patch_prober.go:29] interesting pod/e2e-worker-w6-ebb7fd-openshift-default-544f5768f-w6tt7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.50:15021/healthz/ready\": dial tcp 10.134.0.50:15021: connect: connection refused" start-of-body= Nov 04 02:49:07.427476 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:49:07.427332 4405 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-ebb7fd-openshift-default-544f5768f-w6tt7" podUID="2522c184-4d83-40b4-9a42-cbc55b7a5a37" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.50:15021/healthz/ready\": dial tcp 10.134.0.50:15021: connect: connection refused" Nov 04 02:49:08.265194 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:49:08.265157 4405 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-4ndqx" Nov 04 02:49:08.274661 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:49:08.274632 4405 generic.go:356] "Generic (PLEG): container finished" podID="45bb7186-84f1-4586-b4c1-50250365ff2b" containerID="dd73bfbe496cb2981cf8933215430460d8e7d58d9eda1d0d88dc5bdfcb4ceb2e" exitCode=0 Nov 04 02:49:08.274834 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:49:08.274716 4405 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-tlkdl" event={"ID":"45bb7186-84f1-4586-b4c1-50250365ff2b","Type":"ContainerDied","Data":"dd73bfbe496cb2981cf8933215430460d8e7d58d9eda1d0d88dc5bdfcb4ceb2e"} Nov 04 02:49:08.276179 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:49:08.276153 4405 generic.go:356] "Generic (PLEG): container finished" podID="a236d45b-5615-4d69-9043-52de98c2ab38" containerID="9f736a640f6a81dee067ff6beb11f61696cd002aa50f12332518b080fef961c0" exitCode=0 Nov 04 02:49:08.276302 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:49:08.276203 4405 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-4ndqx" Nov 04 02:49:08.276302 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:49:08.276254 4405 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-4ndqx" event={"ID":"a236d45b-5615-4d69-9043-52de98c2ab38","Type":"ContainerDied","Data":"9f736a640f6a81dee067ff6beb11f61696cd002aa50f12332518b080fef961c0"} Nov 04 02:49:08.276415 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:49:08.276320 4405 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-4ndqx" event={"ID":"a236d45b-5615-4d69-9043-52de98c2ab38","Type":"ContainerDied","Data":"96145a2af9a45df2eb9e921bf7bc74b67bac365da8be1af8ff3e2587bedbb58e"} Nov 04 02:49:08.276415 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:49:08.276333 4405 scope.go:122] "RemoveContainer" containerID="9f736a640f6a81dee067ff6beb11f61696cd002aa50f12332518b080fef961c0" Nov 04 02:49:08.330328 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:49:08.330243 4405 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-tlkdl" Nov 04 02:49:08.331604 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:49:08.331568 4405 scope.go:122] "RemoveContainer" containerID="9f736a640f6a81dee067ff6beb11f61696cd002aa50f12332518b080fef961c0" Nov 04 02:49:08.332122 ip-10-0-134-125 kubenswrapper[4405]: E1104 02:49:08.332096 4405 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9f736a640f6a81dee067ff6beb11f61696cd002aa50f12332518b080fef961c0\": container with ID starting with 9f736a640f6a81dee067ff6beb11f61696cd002aa50f12332518b080fef961c0 not found: ID does not exist" containerID="9f736a640f6a81dee067ff6beb11f61696cd002aa50f12332518b080fef961c0" Nov 04 02:49:08.332211 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:49:08.332135 4405 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9f736a640f6a81dee067ff6beb11f61696cd002aa50f12332518b080fef961c0"} err="failed to get container status \"9f736a640f6a81dee067ff6beb11f61696cd002aa50f12332518b080fef961c0\": rpc error: code = NotFound desc = could not find container \"9f736a640f6a81dee067ff6beb11f61696cd002aa50f12332518b080fef961c0\": container with ID starting with 9f736a640f6a81dee067ff6beb11f61696cd002aa50f12332518b080fef961c0 not found: ID does not exist" Nov 04 02:49:08.369795 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:49:08.369759 4405 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/a236d45b-5615-4d69-9043-52de98c2ab38-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a236d45b-5615-4d69-9043-52de98c2ab38-istio-podinfo\") pod \"a236d45b-5615-4d69-9043-52de98c2ab38\" (UID: \"a236d45b-5615-4d69-9043-52de98c2ab38\") " Nov 04 02:49:08.369945 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:49:08.369804 4405 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/45bb7186-84f1-4586-b4c1-50250365ff2b-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/45bb7186-84f1-4586-b4c1-50250365ff2b-istio-envoy\") pod \"45bb7186-84f1-4586-b4c1-50250365ff2b\" (UID: \"45bb7186-84f1-4586-b4c1-50250365ff2b\") " Nov 04 02:49:08.369945 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:49:08.369854 4405 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/45bb7186-84f1-4586-b4c1-50250365ff2b-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/45bb7186-84f1-4586-b4c1-50250365ff2b-credential-socket\") pod \"45bb7186-84f1-4586-b4c1-50250365ff2b\" (UID: \"45bb7186-84f1-4586-b4c1-50250365ff2b\") " Nov 04 02:49:08.369945 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:49:08.369915 4405 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/45bb7186-84f1-4586-b4c1-50250365ff2b-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/45bb7186-84f1-4586-b4c1-50250365ff2b-istio-data\") pod \"45bb7186-84f1-4586-b4c1-50250365ff2b\" (UID: \"45bb7186-84f1-4586-b4c1-50250365ff2b\") " Nov 04 02:49:08.370067 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:49:08.369955 4405 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/45bb7186-84f1-4586-b4c1-50250365ff2b-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/45bb7186-84f1-4586-b4c1-50250365ff2b-workload-certs\") pod \"45bb7186-84f1-4586-b4c1-50250365ff2b\" (UID: \"45bb7186-84f1-4586-b4c1-50250365ff2b\") " Nov 04 02:49:08.370067 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:49:08.369995 4405 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a236d45b-5615-4d69-9043-52de98c2ab38-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a236d45b-5615-4d69-9043-52de98c2ab38-istio-envoy\") pod \"a236d45b-5615-4d69-9043-52de98c2ab38\" (UID: \"a236d45b-5615-4d69-9043-52de98c2ab38\") " Nov 04 02:49:08.370067 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:49:08.370019 4405 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a236d45b-5615-4d69-9043-52de98c2ab38-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a236d45b-5615-4d69-9043-52de98c2ab38-istiod-ca-cert\") pod \"a236d45b-5615-4d69-9043-52de98c2ab38\" (UID: \"a236d45b-5615-4d69-9043-52de98c2ab38\") " Nov 04 02:49:08.370067 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:49:08.370061 4405 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/45bb7186-84f1-4586-b4c1-50250365ff2b-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/45bb7186-84f1-4586-b4c1-50250365ff2b-istiod-ca-cert\") pod \"45bb7186-84f1-4586-b4c1-50250365ff2b\" (UID: \"45bb7186-84f1-4586-b4c1-50250365ff2b\") " Nov 04 02:49:08.370208 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:49:08.370109 4405 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/45bb7186-84f1-4586-b4c1-50250365ff2b-kube-api-access-2lllb\" (UniqueName: \"kubernetes.io/projected/45bb7186-84f1-4586-b4c1-50250365ff2b-kube-api-access-2lllb\") pod \"45bb7186-84f1-4586-b4c1-50250365ff2b\" (UID: \"45bb7186-84f1-4586-b4c1-50250365ff2b\") " Nov 04 02:49:08.370208 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:49:08.370132 4405 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/45bb7186-84f1-4586-b4c1-50250365ff2b-credential-socket" pod "45bb7186-84f1-4586-b4c1-50250365ff2b" (UID: "45bb7186-84f1-4586-b4c1-50250365ff2b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 02:49:08.370208 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:49:08.370151 4405 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/45bb7186-84f1-4586-b4c1-50250365ff2b-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/45bb7186-84f1-4586-b4c1-50250365ff2b-istio-podinfo\") pod \"45bb7186-84f1-4586-b4c1-50250365ff2b\" (UID: \"45bb7186-84f1-4586-b4c1-50250365ff2b\") " Nov 04 02:49:08.370208 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:49:08.370190 4405 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a236d45b-5615-4d69-9043-52de98c2ab38-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a236d45b-5615-4d69-9043-52de98c2ab38-workload-socket\") pod \"a236d45b-5615-4d69-9043-52de98c2ab38\" (UID: \"a236d45b-5615-4d69-9043-52de98c2ab38\") " Nov 04 02:49:08.370402 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:49:08.370245 4405 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/45bb7186-84f1-4586-b4c1-50250365ff2b-istio-token\" (UniqueName: \"kubernetes.io/projected/45bb7186-84f1-4586-b4c1-50250365ff2b-istio-token\") pod \"45bb7186-84f1-4586-b4c1-50250365ff2b\" (UID: \"45bb7186-84f1-4586-b4c1-50250365ff2b\") " Nov 04 02:49:08.370402 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:49:08.370274 4405 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a236d45b-5615-4d69-9043-52de98c2ab38-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a236d45b-5615-4d69-9043-52de98c2ab38-credential-socket\") pod \"a236d45b-5615-4d69-9043-52de98c2ab38\" (UID: \"a236d45b-5615-4d69-9043-52de98c2ab38\") " Nov 04 02:49:08.370402 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:49:08.370309 4405 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a236d45b-5615-4d69-9043-52de98c2ab38-istio-token\" (UniqueName: \"kubernetes.io/projected/a236d45b-5615-4d69-9043-52de98c2ab38-istio-token\") pod \"a236d45b-5615-4d69-9043-52de98c2ab38\" (UID: \"a236d45b-5615-4d69-9043-52de98c2ab38\") " Nov 04 02:49:08.370402 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:49:08.370328 4405 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/45bb7186-84f1-4586-b4c1-50250365ff2b-istio-data" pod "45bb7186-84f1-4586-b4c1-50250365ff2b" (UID: "45bb7186-84f1-4586-b4c1-50250365ff2b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 02:49:08.370402 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:49:08.370338 4405 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a236d45b-5615-4d69-9043-52de98c2ab38-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a236d45b-5615-4d69-9043-52de98c2ab38-istio-data\") pod \"a236d45b-5615-4d69-9043-52de98c2ab38\" (UID: \"a236d45b-5615-4d69-9043-52de98c2ab38\") " Nov 04 02:49:08.370855 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:49:08.370418 4405 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a236d45b-5615-4d69-9043-52de98c2ab38-kube-api-access-l2dr4\" (UniqueName: \"kubernetes.io/projected/a236d45b-5615-4d69-9043-52de98c2ab38-kube-api-access-l2dr4\") pod \"a236d45b-5615-4d69-9043-52de98c2ab38\" (UID: \"a236d45b-5615-4d69-9043-52de98c2ab38\") " Nov 04 02:49:08.370855 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:49:08.370461 4405 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/45bb7186-84f1-4586-b4c1-50250365ff2b-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/45bb7186-84f1-4586-b4c1-50250365ff2b-workload-socket\") pod \"45bb7186-84f1-4586-b4c1-50250365ff2b\" (UID: \"45bb7186-84f1-4586-b4c1-50250365ff2b\") " Nov 04 02:49:08.370855 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:49:08.370485 4405 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/45bb7186-84f1-4586-b4c1-50250365ff2b-workload-certs" pod "45bb7186-84f1-4586-b4c1-50250365ff2b" (UID: "45bb7186-84f1-4586-b4c1-50250365ff2b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 02:49:08.370855 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:49:08.370493 4405 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a236d45b-5615-4d69-9043-52de98c2ab38-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a236d45b-5615-4d69-9043-52de98c2ab38-workload-certs\") pod \"a236d45b-5615-4d69-9043-52de98c2ab38\" (UID: \"a236d45b-5615-4d69-9043-52de98c2ab38\") " Nov 04 02:49:08.370855 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:49:08.370617 4405 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/45bb7186-84f1-4586-b4c1-50250365ff2b-istiod-ca-cert" pod "45bb7186-84f1-4586-b4c1-50250365ff2b" (UID: "45bb7186-84f1-4586-b4c1-50250365ff2b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 02:49:08.370855 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:49:08.370766 4405 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a236d45b-5615-4d69-9043-52de98c2ab38-istio-data" pod "a236d45b-5615-4d69-9043-52de98c2ab38" (UID: "a236d45b-5615-4d69-9043-52de98c2ab38"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 02:49:08.370855 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:49:08.370789 4405 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a236d45b-5615-4d69-9043-52de98c2ab38-workload-certs" pod "a236d45b-5615-4d69-9043-52de98c2ab38" (UID: "a236d45b-5615-4d69-9043-52de98c2ab38"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 02:49:08.370855 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:49:08.370651 4405 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a236d45b-5615-4d69-9043-52de98c2ab38-istiod-ca-cert" pod "a236d45b-5615-4d69-9043-52de98c2ab38" (UID: "a236d45b-5615-4d69-9043-52de98c2ab38"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 02:49:08.371649 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:49:08.370988 4405 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a236d45b-5615-4d69-9043-52de98c2ab38-workload-socket" pod "a236d45b-5615-4d69-9043-52de98c2ab38" (UID: "a236d45b-5615-4d69-9043-52de98c2ab38"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 02:49:08.371649 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:49:08.370991 4405 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/45bb7186-84f1-4586-b4c1-50250365ff2b-workload-socket" pod "45bb7186-84f1-4586-b4c1-50250365ff2b" (UID: "45bb7186-84f1-4586-b4c1-50250365ff2b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 02:49:08.371649 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:49:08.371152 4405 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/45bb7186-84f1-4586-b4c1-50250365ff2b-istio-data\") on node \"ip-10-0-134-125.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 02:49:08.371649 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:49:08.371172 4405 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/45bb7186-84f1-4586-b4c1-50250365ff2b-workload-certs\") on node \"ip-10-0-134-125.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 02:49:08.371649 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:49:08.371187 4405 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a236d45b-5615-4d69-9043-52de98c2ab38-istiod-ca-cert\") on node \"ip-10-0-134-125.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 02:49:08.371649 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:49:08.371201 4405 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/45bb7186-84f1-4586-b4c1-50250365ff2b-istiod-ca-cert\") on node \"ip-10-0-134-125.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 02:49:08.371649 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:49:08.371215 4405 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a236d45b-5615-4d69-9043-52de98c2ab38-workload-socket\") on node \"ip-10-0-134-125.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 02:49:08.371649 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:49:08.371225 4405 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a236d45b-5615-4d69-9043-52de98c2ab38-istio-data\") on node \"ip-10-0-134-125.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 02:49:08.371649 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:49:08.371234 4405 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/45bb7186-84f1-4586-b4c1-50250365ff2b-workload-socket\") on node \"ip-10-0-134-125.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 02:49:08.371649 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:49:08.371243 4405 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a236d45b-5615-4d69-9043-52de98c2ab38-workload-certs\") on node \"ip-10-0-134-125.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 02:49:08.371649 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:49:08.371253 4405 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/45bb7186-84f1-4586-b4c1-50250365ff2b-credential-socket\") on node \"ip-10-0-134-125.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 02:49:08.371649 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:49:08.371185 4405 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a236d45b-5615-4d69-9043-52de98c2ab38-credential-socket" pod "a236d45b-5615-4d69-9043-52de98c2ab38" (UID: "a236d45b-5615-4d69-9043-52de98c2ab38"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 02:49:08.372869 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:49:08.372822 4405 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a236d45b-5615-4d69-9043-52de98c2ab38-istio-podinfo" pod "a236d45b-5615-4d69-9043-52de98c2ab38" (UID: "a236d45b-5615-4d69-9043-52de98c2ab38"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 04 02:49:08.373099 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:49:08.373073 4405 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/45bb7186-84f1-4586-b4c1-50250365ff2b-istio-envoy" pod "45bb7186-84f1-4586-b4c1-50250365ff2b" (UID: "45bb7186-84f1-4586-b4c1-50250365ff2b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 02:49:08.373661 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:49:08.373623 4405 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/45bb7186-84f1-4586-b4c1-50250365ff2b-istio-podinfo" pod "45bb7186-84f1-4586-b4c1-50250365ff2b" (UID: "45bb7186-84f1-4586-b4c1-50250365ff2b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 04 02:49:08.373766 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:49:08.373736 4405 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a236d45b-5615-4d69-9043-52de98c2ab38-istio-token" pod "a236d45b-5615-4d69-9043-52de98c2ab38" (UID: "a236d45b-5615-4d69-9043-52de98c2ab38"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 02:49:08.373902 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:49:08.373872 4405 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/45bb7186-84f1-4586-b4c1-50250365ff2b-kube-api-access-2lllb" pod "45bb7186-84f1-4586-b4c1-50250365ff2b" (UID: "45bb7186-84f1-4586-b4c1-50250365ff2b"). InnerVolumeSpecName "kube-api-access-2lllb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 02:49:08.374225 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:49:08.374197 4405 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a236d45b-5615-4d69-9043-52de98c2ab38-istio-envoy" pod "a236d45b-5615-4d69-9043-52de98c2ab38" (UID: "a236d45b-5615-4d69-9043-52de98c2ab38"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 02:49:08.374351 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:49:08.374332 4405 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a236d45b-5615-4d69-9043-52de98c2ab38-kube-api-access-l2dr4" pod "a236d45b-5615-4d69-9043-52de98c2ab38" (UID: "a236d45b-5615-4d69-9043-52de98c2ab38"). InnerVolumeSpecName "kube-api-access-l2dr4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 02:49:08.375046 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:49:08.375026 4405 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/45bb7186-84f1-4586-b4c1-50250365ff2b-istio-token" pod "45bb7186-84f1-4586-b4c1-50250365ff2b" (UID: "45bb7186-84f1-4586-b4c1-50250365ff2b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 02:49:08.427312 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:49:08.427272 4405 patch_prober.go:29] interesting pod/e2e-worker-w6-ebb7fd-openshift-default-544f5768f-w6tt7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.50:15021/healthz/ready\": dial tcp 10.134.0.50:15021: connect: connection refused" start-of-body= Nov 04 02:49:08.427484 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:49:08.427340 4405 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-ebb7fd-openshift-default-544f5768f-w6tt7" podUID="2522c184-4d83-40b4-9a42-cbc55b7a5a37" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.50:15021/healthz/ready\": dial tcp 10.134.0.50:15021: connect: connection refused" Nov 04 02:49:08.473736 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:49:08.472083 4405 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a236d45b-5615-4d69-9043-52de98c2ab38-istio-envoy\") on node \"ip-10-0-134-125.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 02:49:08.473736 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:49:08.472113 4405 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2lllb\" (UniqueName: \"kubernetes.io/projected/45bb7186-84f1-4586-b4c1-50250365ff2b-kube-api-access-2lllb\") on node \"ip-10-0-134-125.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 02:49:08.473736 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:49:08.472123 4405 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/45bb7186-84f1-4586-b4c1-50250365ff2b-istio-podinfo\") on node \"ip-10-0-134-125.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 02:49:08.473736 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:49:08.472132 4405 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/45bb7186-84f1-4586-b4c1-50250365ff2b-istio-token\") on node \"ip-10-0-134-125.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 02:49:08.473736 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:49:08.472140 4405 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a236d45b-5615-4d69-9043-52de98c2ab38-credential-socket\") on node \"ip-10-0-134-125.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 02:49:08.473736 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:49:08.472149 4405 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a236d45b-5615-4d69-9043-52de98c2ab38-istio-token\") on node \"ip-10-0-134-125.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 02:49:08.473736 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:49:08.472158 4405 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l2dr4\" (UniqueName: \"kubernetes.io/projected/a236d45b-5615-4d69-9043-52de98c2ab38-kube-api-access-l2dr4\") on node \"ip-10-0-134-125.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 02:49:08.473736 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:49:08.472167 4405 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a236d45b-5615-4d69-9043-52de98c2ab38-istio-podinfo\") on node \"ip-10-0-134-125.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 02:49:08.473736 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:49:08.472175 4405 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/45bb7186-84f1-4586-b4c1-50250365ff2b-istio-envoy\") on node \"ip-10-0-134-125.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 02:49:08.740028 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:49:08.739920 4405 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-76597d6855-4ndqx"] Nov 04 02:49:08.744050 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:49:08.744021 4405 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-76597d6855-4ndqx"] Nov 04 02:49:09.283210 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:49:09.283179 4405 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-tlkdl" event={"ID":"45bb7186-84f1-4586-b4c1-50250365ff2b","Type":"ContainerDied","Data":"8cfe87dfef8cfc9190a002e22ce9bbf669809e1322d60af781ae027c5b1d3bb3"} Nov 04 02:49:09.283210 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:49:09.283196 4405 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-tlkdl" Nov 04 02:49:09.283808 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:49:09.283233 4405 scope.go:122] "RemoveContainer" containerID="dd73bfbe496cb2981cf8933215430460d8e7d58d9eda1d0d88dc5bdfcb4ceb2e" Nov 04 02:49:09.427916 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:49:09.427881 4405 patch_prober.go:29] interesting pod/e2e-worker-w6-ebb7fd-openshift-default-544f5768f-w6tt7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.50:15021/healthz/ready\": dial tcp 10.134.0.50:15021: connect: connection refused" start-of-body= Nov 04 02:49:09.428159 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:49:09.428127 4405 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-ebb7fd-openshift-default-544f5768f-w6tt7" podUID="2522c184-4d83-40b4-9a42-cbc55b7a5a37" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.50:15021/healthz/ready\": dial tcp 10.134.0.50:15021: connect: connection refused" Nov 04 02:49:09.449886 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:49:09.449821 4405 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-76597d6855-tlkdl"] Nov 04 02:49:09.528696 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:49:09.514476 4405 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-76597d6855-tlkdl"] Nov 04 02:49:09.724626 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:49:09.724586 4405 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="45bb7186-84f1-4586-b4c1-50250365ff2b" path="/var/lib/kubelet/pods/45bb7186-84f1-4586-b4c1-50250365ff2b/volumes" Nov 04 02:49:09.725326 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:49:09.725303 4405 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a236d45b-5615-4d69-9043-52de98c2ab38" path="/var/lib/kubelet/pods/a236d45b-5615-4d69-9043-52de98c2ab38/volumes" Nov 04 02:49:10.427385 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:49:10.427350 4405 patch_prober.go:29] interesting pod/e2e-worker-w6-ebb7fd-openshift-default-544f5768f-w6tt7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.50:15021/healthz/ready\": dial tcp 10.134.0.50:15021: connect: connection refused" start-of-body= Nov 04 02:49:10.427777 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:49:10.427411 4405 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-ebb7fd-openshift-default-544f5768f-w6tt7" podUID="2522c184-4d83-40b4-9a42-cbc55b7a5a37" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.50:15021/healthz/ready\": dial tcp 10.134.0.50:15021: connect: connection refused" Nov 04 02:49:11.432445 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:49:11.432412 4405 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w6-ebb7fd-openshift-default-544f5768f-w6tt7" Nov 04 02:49:11.432940 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:49:11.432704 4405 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w6-ebb7fd-openshift-default-544f5768f-w6tt7" Nov 04 02:49:11.433557 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:49:11.433539 4405 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w6-ebb7fd-openshift-default-544f5768f-w6tt7" Nov 04 02:49:32.677892 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:49:32.677855 4405 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-203c35-openshift-default-56497cdcc-2b66r"] Nov 04 02:49:32.678590 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:49:32.678318 4405 cpu_manager.go:414] "containerMap: removing container" podUID="45bb7186-84f1-4586-b4c1-50250365ff2b" containerName="istio-proxy" Nov 04 02:49:32.678590 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:49:32.678329 4405 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="45bb7186-84f1-4586-b4c1-50250365ff2b" containerName="istio-proxy" Nov 04 02:49:32.678590 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:49:32.678340 4405 cpu_manager.go:414] "containerMap: removing container" podUID="a236d45b-5615-4d69-9043-52de98c2ab38" containerName="istio-proxy" Nov 04 02:49:32.678590 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:49:32.678345 4405 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a236d45b-5615-4d69-9043-52de98c2ab38" containerName="istio-proxy" Nov 04 02:49:32.678590 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:49:32.678431 4405 memory_manager.go:365] "RemoveStaleState removing state" podUID="45bb7186-84f1-4586-b4c1-50250365ff2b" containerName="istio-proxy" Nov 04 02:49:32.678590 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:49:32.678443 4405 memory_manager.go:365] "RemoveStaleState removing state" podUID="a236d45b-5615-4d69-9043-52de98c2ab38" containerName="istio-proxy" Nov 04 02:49:32.681812 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:49:32.681796 4405 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-203c35-openshift-default-56497cdcc-2b66r" Nov 04 02:49:32.734053 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:49:32.734012 4405 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w4-203c35-openshift-default-dockercfg-ggkdg\"" Nov 04 02:49:32.739539 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:49:32.739498 4405 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/147e25a7-dda1-4170-9453-44896e44e2bc-istio-podinfo\") pod \"e2e-worker-w4-203c35-openshift-default-56497cdcc-2b66r\" (UID: \"147e25a7-dda1-4170-9453-44896e44e2bc\") " pod="openshift-ingress/e2e-worker-w4-203c35-openshift-default-56497cdcc-2b66r" Nov 04 02:49:32.740348 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:49:32.740316 4405 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/147e25a7-dda1-4170-9453-44896e44e2bc-istio-token\") pod \"e2e-worker-w4-203c35-openshift-default-56497cdcc-2b66r\" (UID: \"147e25a7-dda1-4170-9453-44896e44e2bc\") " pod="openshift-ingress/e2e-worker-w4-203c35-openshift-default-56497cdcc-2b66r" Nov 04 02:49:32.740692 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:49:32.740640 4405 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/147e25a7-dda1-4170-9453-44896e44e2bc-workload-socket\") pod \"e2e-worker-w4-203c35-openshift-default-56497cdcc-2b66r\" (UID: \"147e25a7-dda1-4170-9453-44896e44e2bc\") " pod="openshift-ingress/e2e-worker-w4-203c35-openshift-default-56497cdcc-2b66r" Nov 04 02:49:32.740839 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:49:32.740729 4405 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jf2xt\" (UniqueName: \"kubernetes.io/projected/147e25a7-dda1-4170-9453-44896e44e2bc-kube-api-access-jf2xt\") pod \"e2e-worker-w4-203c35-openshift-default-56497cdcc-2b66r\" (UID: \"147e25a7-dda1-4170-9453-44896e44e2bc\") " pod="openshift-ingress/e2e-worker-w4-203c35-openshift-default-56497cdcc-2b66r" Nov 04 02:49:32.740839 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:49:32.740792 4405 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/147e25a7-dda1-4170-9453-44896e44e2bc-istiod-ca-cert\") pod \"e2e-worker-w4-203c35-openshift-default-56497cdcc-2b66r\" (UID: \"147e25a7-dda1-4170-9453-44896e44e2bc\") " pod="openshift-ingress/e2e-worker-w4-203c35-openshift-default-56497cdcc-2b66r" Nov 04 02:49:32.740957 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:49:32.740914 4405 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/147e25a7-dda1-4170-9453-44896e44e2bc-credential-socket\") pod \"e2e-worker-w4-203c35-openshift-default-56497cdcc-2b66r\" (UID: \"147e25a7-dda1-4170-9453-44896e44e2bc\") " pod="openshift-ingress/e2e-worker-w4-203c35-openshift-default-56497cdcc-2b66r" Nov 04 02:49:32.740957 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:49:32.740936 4405 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/147e25a7-dda1-4170-9453-44896e44e2bc-istio-envoy\") pod \"e2e-worker-w4-203c35-openshift-default-56497cdcc-2b66r\" (UID: \"147e25a7-dda1-4170-9453-44896e44e2bc\") " pod="openshift-ingress/e2e-worker-w4-203c35-openshift-default-56497cdcc-2b66r" Nov 04 02:49:32.741062 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:49:32.740968 4405 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/147e25a7-dda1-4170-9453-44896e44e2bc-workload-certs\") pod \"e2e-worker-w4-203c35-openshift-default-56497cdcc-2b66r\" (UID: \"147e25a7-dda1-4170-9453-44896e44e2bc\") " pod="openshift-ingress/e2e-worker-w4-203c35-openshift-default-56497cdcc-2b66r" Nov 04 02:49:32.741062 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:49:32.740984 4405 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/147e25a7-dda1-4170-9453-44896e44e2bc-istio-data\") pod \"e2e-worker-w4-203c35-openshift-default-56497cdcc-2b66r\" (UID: \"147e25a7-dda1-4170-9453-44896e44e2bc\") " pod="openshift-ingress/e2e-worker-w4-203c35-openshift-default-56497cdcc-2b66r" Nov 04 02:49:32.785763 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:49:32.785736 4405 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-203c35-openshift-default-56497cdcc-2b66r"] Nov 04 02:49:32.842081 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:49:32.842036 4405 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/147e25a7-dda1-4170-9453-44896e44e2bc-istio-podinfo\") pod \"e2e-worker-w4-203c35-openshift-default-56497cdcc-2b66r\" (UID: \"147e25a7-dda1-4170-9453-44896e44e2bc\") " pod="openshift-ingress/e2e-worker-w4-203c35-openshift-default-56497cdcc-2b66r" Nov 04 02:49:32.842283 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:49:32.842088 4405 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/147e25a7-dda1-4170-9453-44896e44e2bc-istio-token\") pod \"e2e-worker-w4-203c35-openshift-default-56497cdcc-2b66r\" (UID: \"147e25a7-dda1-4170-9453-44896e44e2bc\") " pod="openshift-ingress/e2e-worker-w4-203c35-openshift-default-56497cdcc-2b66r" Nov 04 02:49:32.842283 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:49:32.842141 4405 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/147e25a7-dda1-4170-9453-44896e44e2bc-workload-socket\") pod \"e2e-worker-w4-203c35-openshift-default-56497cdcc-2b66r\" (UID: \"147e25a7-dda1-4170-9453-44896e44e2bc\") " pod="openshift-ingress/e2e-worker-w4-203c35-openshift-default-56497cdcc-2b66r" Nov 04 02:49:32.842283 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:49:32.842180 4405 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jf2xt\" (UniqueName: \"kubernetes.io/projected/147e25a7-dda1-4170-9453-44896e44e2bc-kube-api-access-jf2xt\") pod \"e2e-worker-w4-203c35-openshift-default-56497cdcc-2b66r\" (UID: \"147e25a7-dda1-4170-9453-44896e44e2bc\") " pod="openshift-ingress/e2e-worker-w4-203c35-openshift-default-56497cdcc-2b66r" Nov 04 02:49:32.842283 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:49:32.842249 4405 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/147e25a7-dda1-4170-9453-44896e44e2bc-istiod-ca-cert\") pod \"e2e-worker-w4-203c35-openshift-default-56497cdcc-2b66r\" (UID: \"147e25a7-dda1-4170-9453-44896e44e2bc\") " pod="openshift-ingress/e2e-worker-w4-203c35-openshift-default-56497cdcc-2b66r" Nov 04 02:49:32.842507 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:49:32.842346 4405 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/147e25a7-dda1-4170-9453-44896e44e2bc-credential-socket\") pod \"e2e-worker-w4-203c35-openshift-default-56497cdcc-2b66r\" (UID: \"147e25a7-dda1-4170-9453-44896e44e2bc\") " pod="openshift-ingress/e2e-worker-w4-203c35-openshift-default-56497cdcc-2b66r" Nov 04 02:49:32.842507 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:49:32.842373 4405 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/147e25a7-dda1-4170-9453-44896e44e2bc-istio-envoy\") pod \"e2e-worker-w4-203c35-openshift-default-56497cdcc-2b66r\" (UID: \"147e25a7-dda1-4170-9453-44896e44e2bc\") " pod="openshift-ingress/e2e-worker-w4-203c35-openshift-default-56497cdcc-2b66r" Nov 04 02:49:32.842507 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:49:32.842421 4405 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/147e25a7-dda1-4170-9453-44896e44e2bc-workload-certs\") pod \"e2e-worker-w4-203c35-openshift-default-56497cdcc-2b66r\" (UID: \"147e25a7-dda1-4170-9453-44896e44e2bc\") " pod="openshift-ingress/e2e-worker-w4-203c35-openshift-default-56497cdcc-2b66r" Nov 04 02:49:32.842507 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:49:32.842444 4405 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/147e25a7-dda1-4170-9453-44896e44e2bc-istio-data\") pod \"e2e-worker-w4-203c35-openshift-default-56497cdcc-2b66r\" (UID: \"147e25a7-dda1-4170-9453-44896e44e2bc\") " pod="openshift-ingress/e2e-worker-w4-203c35-openshift-default-56497cdcc-2b66r" Nov 04 02:49:32.842736 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:49:32.842617 4405 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/147e25a7-dda1-4170-9453-44896e44e2bc-workload-socket\") pod \"e2e-worker-w4-203c35-openshift-default-56497cdcc-2b66r\" (UID: \"147e25a7-dda1-4170-9453-44896e44e2bc\") " pod="openshift-ingress/e2e-worker-w4-203c35-openshift-default-56497cdcc-2b66r" Nov 04 02:49:32.842981 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:49:32.842879 4405 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/147e25a7-dda1-4170-9453-44896e44e2bc-istio-data\") pod \"e2e-worker-w4-203c35-openshift-default-56497cdcc-2b66r\" (UID: \"147e25a7-dda1-4170-9453-44896e44e2bc\") " pod="openshift-ingress/e2e-worker-w4-203c35-openshift-default-56497cdcc-2b66r" Nov 04 02:49:32.842981 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:49:32.842937 4405 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/147e25a7-dda1-4170-9453-44896e44e2bc-credential-socket\") pod \"e2e-worker-w4-203c35-openshift-default-56497cdcc-2b66r\" (UID: \"147e25a7-dda1-4170-9453-44896e44e2bc\") " pod="openshift-ingress/e2e-worker-w4-203c35-openshift-default-56497cdcc-2b66r" Nov 04 02:49:32.842981 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:49:32.842960 4405 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/147e25a7-dda1-4170-9453-44896e44e2bc-workload-certs\") pod \"e2e-worker-w4-203c35-openshift-default-56497cdcc-2b66r\" (UID: \"147e25a7-dda1-4170-9453-44896e44e2bc\") " pod="openshift-ingress/e2e-worker-w4-203c35-openshift-default-56497cdcc-2b66r" Nov 04 02:49:32.843131 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:49:32.843056 4405 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/147e25a7-dda1-4170-9453-44896e44e2bc-istiod-ca-cert\") pod \"e2e-worker-w4-203c35-openshift-default-56497cdcc-2b66r\" (UID: \"147e25a7-dda1-4170-9453-44896e44e2bc\") " pod="openshift-ingress/e2e-worker-w4-203c35-openshift-default-56497cdcc-2b66r" Nov 04 02:49:32.844762 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:49:32.844741 4405 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/147e25a7-dda1-4170-9453-44896e44e2bc-istio-podinfo\") pod \"e2e-worker-w4-203c35-openshift-default-56497cdcc-2b66r\" (UID: \"147e25a7-dda1-4170-9453-44896e44e2bc\") " pod="openshift-ingress/e2e-worker-w4-203c35-openshift-default-56497cdcc-2b66r" Nov 04 02:49:32.844858 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:49:32.844764 4405 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/147e25a7-dda1-4170-9453-44896e44e2bc-istio-envoy\") pod \"e2e-worker-w4-203c35-openshift-default-56497cdcc-2b66r\" (UID: \"147e25a7-dda1-4170-9453-44896e44e2bc\") " pod="openshift-ingress/e2e-worker-w4-203c35-openshift-default-56497cdcc-2b66r" Nov 04 02:49:32.903345 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:49:32.903304 4405 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/147e25a7-dda1-4170-9453-44896e44e2bc-istio-token\") pod \"e2e-worker-w4-203c35-openshift-default-56497cdcc-2b66r\" (UID: \"147e25a7-dda1-4170-9453-44896e44e2bc\") " pod="openshift-ingress/e2e-worker-w4-203c35-openshift-default-56497cdcc-2b66r" Nov 04 02:49:32.903496 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:49:32.903429 4405 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jf2xt\" (UniqueName: \"kubernetes.io/projected/147e25a7-dda1-4170-9453-44896e44e2bc-kube-api-access-jf2xt\") pod \"e2e-worker-w4-203c35-openshift-default-56497cdcc-2b66r\" (UID: \"147e25a7-dda1-4170-9453-44896e44e2bc\") " pod="openshift-ingress/e2e-worker-w4-203c35-openshift-default-56497cdcc-2b66r" Nov 04 02:49:33.002791 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:49:33.002708 4405 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-203c35-openshift-default-56497cdcc-2b66r" Nov 04 02:49:33.435173 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:49:33.435141 4405 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-203c35-openshift-default-56497cdcc-2b66r"] Nov 04 02:49:33.437383 ip-10-0-134-125 kubenswrapper[4405]: W1104 02:49:33.437342 4405 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod147e25a7_dda1_4170_9453_44896e44e2bc.slice/crio-c7c598b47e4fc9fb9d4530d8381ac46d074538393d9c17b8d34a86609b51689d WatchSource:0}: Error finding container c7c598b47e4fc9fb9d4530d8381ac46d074538393d9c17b8d34a86609b51689d: Status 404 returned error can't find the container with id c7c598b47e4fc9fb9d4530d8381ac46d074538393d9c17b8d34a86609b51689d Nov 04 02:49:33.439640 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:49:33.439606 4405 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 04 02:49:33.439779 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:49:33.439749 4405 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 04 02:49:33.439846 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:49:33.439803 4405 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 04 02:49:34.426987 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:49:34.426947 4405 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-203c35-openshift-default-56497cdcc-2b66r" event={"ID":"147e25a7-dda1-4170-9453-44896e44e2bc","Type":"ContainerStarted","Data":"9493dae8e6eaed626d077d1b19415b29a1cf533152cc9a5774624d4c6db883b5"} Nov 04 02:49:34.427393 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:49:34.427035 4405 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-203c35-openshift-default-56497cdcc-2b66r" event={"ID":"147e25a7-dda1-4170-9453-44896e44e2bc","Type":"ContainerStarted","Data":"c7c598b47e4fc9fb9d4530d8381ac46d074538393d9c17b8d34a86609b51689d"} Nov 04 02:49:34.647945 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:49:34.647893 4405 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-203c35-openshift-default-56497cdcc-2b66r" podStartSLOduration=2.647879243 podStartE2EDuration="2.647879243s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 02:49:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 02:49:34.598144094 +0000 UTC m=+1021.028844253" watchObservedRunningTime="2026-11-04 02:49:34.647879243 +0000 UTC m=+1021.078579363" Nov 04 02:49:35.005385 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:49:35.004193 4405 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-203c35-openshift-default-56497cdcc-2b66r" Nov 04 02:49:35.008692 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:49:35.007137 4405 patch_prober.go:29] interesting pod/e2e-worker-w4-203c35-openshift-default-56497cdcc-2b66r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.51:15021/healthz/ready\": dial tcp 10.134.0.51:15021: connect: connection refused" start-of-body= Nov 04 02:49:35.008692 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:49:35.007211 4405 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-203c35-openshift-default-56497cdcc-2b66r" podUID="147e25a7-dda1-4170-9453-44896e44e2bc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.51:15021/healthz/ready\": dial tcp 10.134.0.51:15021: connect: connection refused" Nov 04 02:49:36.007657 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:49:36.007620 4405 patch_prober.go:29] interesting pod/e2e-worker-w4-203c35-openshift-default-56497cdcc-2b66r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.51:15021/healthz/ready\": dial tcp 10.134.0.51:15021: connect: connection refused" start-of-body= Nov 04 02:49:36.008309 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:49:36.008276 4405 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-203c35-openshift-default-56497cdcc-2b66r" podUID="147e25a7-dda1-4170-9453-44896e44e2bc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.51:15021/healthz/ready\": dial tcp 10.134.0.51:15021: connect: connection refused" Nov 04 02:49:36.882985 ip-10-0-134-125 kubenswrapper[4405]: E1104 02:49:36.882945 4405 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5f2vb" containerName="extractor" Nov 04 02:49:37.006696 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:49:37.004028 4405 patch_prober.go:29] interesting pod/e2e-worker-w4-203c35-openshift-default-56497cdcc-2b66r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.51:15021/healthz/ready\": dial tcp 10.134.0.51:15021: connect: connection refused" start-of-body= Nov 04 02:49:37.006696 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:49:37.004087 4405 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-203c35-openshift-default-56497cdcc-2b66r" podUID="147e25a7-dda1-4170-9453-44896e44e2bc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.51:15021/healthz/ready\": dial tcp 10.134.0.51:15021: connect: connection refused" Nov 04 02:49:38.012694 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:49:38.005000 4405 patch_prober.go:29] interesting pod/e2e-worker-w4-203c35-openshift-default-56497cdcc-2b66r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.51:15021/healthz/ready\": dial tcp 10.134.0.51:15021: connect: connection refused" start-of-body= Nov 04 02:49:38.012694 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:49:38.005068 4405 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-203c35-openshift-default-56497cdcc-2b66r" podUID="147e25a7-dda1-4170-9453-44896e44e2bc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.51:15021/healthz/ready\": dial tcp 10.134.0.51:15021: connect: connection refused" Nov 04 02:49:39.005693 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:49:39.005005 4405 patch_prober.go:29] interesting pod/e2e-worker-w4-203c35-openshift-default-56497cdcc-2b66r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.51:15021/healthz/ready\": dial tcp 10.134.0.51:15021: connect: connection refused" start-of-body= Nov 04 02:49:39.005693 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:49:39.005073 4405 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-203c35-openshift-default-56497cdcc-2b66r" podUID="147e25a7-dda1-4170-9453-44896e44e2bc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.51:15021/healthz/ready\": dial tcp 10.134.0.51:15021: connect: connection refused" Nov 04 02:49:40.003573 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:49:40.003530 4405 patch_prober.go:29] interesting pod/e2e-worker-w4-203c35-openshift-default-56497cdcc-2b66r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.51:15021/healthz/ready\": dial tcp 10.134.0.51:15021: connect: connection refused" start-of-body= Nov 04 02:49:40.003996 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:49:40.003607 4405 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-203c35-openshift-default-56497cdcc-2b66r" podUID="147e25a7-dda1-4170-9453-44896e44e2bc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.51:15021/healthz/ready\": dial tcp 10.134.0.51:15021: connect: connection refused" Nov 04 02:49:41.008616 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:49:41.008587 4405 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-203c35-openshift-default-56497cdcc-2b66r" Nov 04 02:49:41.009136 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:49:41.008928 4405 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-203c35-openshift-default-56497cdcc-2b66r" Nov 04 02:49:41.009641 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:49:41.009622 4405 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-203c35-openshift-default-56497cdcc-2b66r" Nov 04 02:49:50.106561 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:49:50.106487 4405 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-bl5mg"] Nov 04 02:49:56.757056 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:49:56.757021 4405 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w2-a0bcbc/distinct-w2-a0bcbc-kserve-596f94c7cd-szl4g"] Nov 04 02:49:56.760982 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:49:56.760961 4405 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-a0bcbc/distinct-w2-a0bcbc-kserve-596f94c7cd-szl4g" Nov 04 02:49:56.865464 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:49:56.865432 4405 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-a0bcbc\"/\"distinct-w2-a0bcbc-kserve-self-signed-certs\"" Nov 04 02:49:56.867612 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:49:56.867592 4405 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-a0bcbc/distinct-w2-a0bcbc-kserve-596f94c7cd-szl4g"] Nov 04 02:49:56.881865 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:49:56.881827 4405 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bac32934-8a57-4919-b435-13f367cbea3e-tls-certs\") pod \"distinct-w2-a0bcbc-kserve-596f94c7cd-szl4g\" (UID: \"bac32934-8a57-4919-b435-13f367cbea3e\") " pod="e2e-models-e2e-worker-w2-a0bcbc/distinct-w2-a0bcbc-kserve-596f94c7cd-szl4g" Nov 04 02:49:56.882028 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:49:56.881913 4405 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tv4lr\" (UniqueName: \"kubernetes.io/projected/bac32934-8a57-4919-b435-13f367cbea3e-kube-api-access-tv4lr\") pod \"distinct-w2-a0bcbc-kserve-596f94c7cd-szl4g\" (UID: \"bac32934-8a57-4919-b435-13f367cbea3e\") " pod="e2e-models-e2e-worker-w2-a0bcbc/distinct-w2-a0bcbc-kserve-596f94c7cd-szl4g" Nov 04 02:49:56.882028 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:49:56.881981 4405 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bac32934-8a57-4919-b435-13f367cbea3e-dshm\") pod \"distinct-w2-a0bcbc-kserve-596f94c7cd-szl4g\" (UID: \"bac32934-8a57-4919-b435-13f367cbea3e\") " pod="e2e-models-e2e-worker-w2-a0bcbc/distinct-w2-a0bcbc-kserve-596f94c7cd-szl4g" Nov 04 02:49:56.882028 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:49:56.882012 4405 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bac32934-8a57-4919-b435-13f367cbea3e-home\") pod \"distinct-w2-a0bcbc-kserve-596f94c7cd-szl4g\" (UID: \"bac32934-8a57-4919-b435-13f367cbea3e\") " pod="e2e-models-e2e-worker-w2-a0bcbc/distinct-w2-a0bcbc-kserve-596f94c7cd-szl4g" Nov 04 02:49:56.882134 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:49:56.882072 4405 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bac32934-8a57-4919-b435-13f367cbea3e-tmp-dir\") pod \"distinct-w2-a0bcbc-kserve-596f94c7cd-szl4g\" (UID: \"bac32934-8a57-4919-b435-13f367cbea3e\") " pod="e2e-models-e2e-worker-w2-a0bcbc/distinct-w2-a0bcbc-kserve-596f94c7cd-szl4g" Nov 04 02:49:56.882134 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:49:56.882119 4405 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bac32934-8a57-4919-b435-13f367cbea3e-model-cache\") pod \"distinct-w2-a0bcbc-kserve-596f94c7cd-szl4g\" (UID: \"bac32934-8a57-4919-b435-13f367cbea3e\") " pod="e2e-models-e2e-worker-w2-a0bcbc/distinct-w2-a0bcbc-kserve-596f94c7cd-szl4g" Nov 04 02:49:56.982817 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:49:56.982773 4405 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bac32934-8a57-4919-b435-13f367cbea3e-tls-certs\") pod \"distinct-w2-a0bcbc-kserve-596f94c7cd-szl4g\" (UID: \"bac32934-8a57-4919-b435-13f367cbea3e\") " pod="e2e-models-e2e-worker-w2-a0bcbc/distinct-w2-a0bcbc-kserve-596f94c7cd-szl4g" Nov 04 02:49:56.983001 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:49:56.982826 4405 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tv4lr\" (UniqueName: \"kubernetes.io/projected/bac32934-8a57-4919-b435-13f367cbea3e-kube-api-access-tv4lr\") pod \"distinct-w2-a0bcbc-kserve-596f94c7cd-szl4g\" (UID: \"bac32934-8a57-4919-b435-13f367cbea3e\") " pod="e2e-models-e2e-worker-w2-a0bcbc/distinct-w2-a0bcbc-kserve-596f94c7cd-szl4g" Nov 04 02:49:56.983001 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:49:56.982877 4405 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bac32934-8a57-4919-b435-13f367cbea3e-dshm\") pod \"distinct-w2-a0bcbc-kserve-596f94c7cd-szl4g\" (UID: \"bac32934-8a57-4919-b435-13f367cbea3e\") " pod="e2e-models-e2e-worker-w2-a0bcbc/distinct-w2-a0bcbc-kserve-596f94c7cd-szl4g" Nov 04 02:49:56.983001 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:49:56.982917 4405 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bac32934-8a57-4919-b435-13f367cbea3e-home\") pod \"distinct-w2-a0bcbc-kserve-596f94c7cd-szl4g\" (UID: \"bac32934-8a57-4919-b435-13f367cbea3e\") " pod="e2e-models-e2e-worker-w2-a0bcbc/distinct-w2-a0bcbc-kserve-596f94c7cd-szl4g" Nov 04 02:49:56.983001 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:49:56.982968 4405 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bac32934-8a57-4919-b435-13f367cbea3e-tmp-dir\") pod \"distinct-w2-a0bcbc-kserve-596f94c7cd-szl4g\" (UID: \"bac32934-8a57-4919-b435-13f367cbea3e\") " pod="e2e-models-e2e-worker-w2-a0bcbc/distinct-w2-a0bcbc-kserve-596f94c7cd-szl4g" Nov 04 02:49:56.983224 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:49:56.983015 4405 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bac32934-8a57-4919-b435-13f367cbea3e-model-cache\") pod \"distinct-w2-a0bcbc-kserve-596f94c7cd-szl4g\" (UID: \"bac32934-8a57-4919-b435-13f367cbea3e\") " pod="e2e-models-e2e-worker-w2-a0bcbc/distinct-w2-a0bcbc-kserve-596f94c7cd-szl4g" Nov 04 02:49:56.983522 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:49:56.983492 4405 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bac32934-8a57-4919-b435-13f367cbea3e-tmp-dir\") pod \"distinct-w2-a0bcbc-kserve-596f94c7cd-szl4g\" (UID: \"bac32934-8a57-4919-b435-13f367cbea3e\") " pod="e2e-models-e2e-worker-w2-a0bcbc/distinct-w2-a0bcbc-kserve-596f94c7cd-szl4g" Nov 04 02:49:56.983522 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:49:56.983511 4405 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bac32934-8a57-4919-b435-13f367cbea3e-model-cache\") pod \"distinct-w2-a0bcbc-kserve-596f94c7cd-szl4g\" (UID: \"bac32934-8a57-4919-b435-13f367cbea3e\") " pod="e2e-models-e2e-worker-w2-a0bcbc/distinct-w2-a0bcbc-kserve-596f94c7cd-szl4g" Nov 04 02:49:56.984016 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:49:56.983992 4405 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bac32934-8a57-4919-b435-13f367cbea3e-home\") pod \"distinct-w2-a0bcbc-kserve-596f94c7cd-szl4g\" (UID: \"bac32934-8a57-4919-b435-13f367cbea3e\") " pod="e2e-models-e2e-worker-w2-a0bcbc/distinct-w2-a0bcbc-kserve-596f94c7cd-szl4g" Nov 04 02:49:56.985386 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:49:56.985360 4405 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bac32934-8a57-4919-b435-13f367cbea3e-dshm\") pod \"distinct-w2-a0bcbc-kserve-596f94c7cd-szl4g\" (UID: \"bac32934-8a57-4919-b435-13f367cbea3e\") " pod="e2e-models-e2e-worker-w2-a0bcbc/distinct-w2-a0bcbc-kserve-596f94c7cd-szl4g" Nov 04 02:49:56.985758 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:49:56.985737 4405 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bac32934-8a57-4919-b435-13f367cbea3e-tls-certs\") pod \"distinct-w2-a0bcbc-kserve-596f94c7cd-szl4g\" (UID: \"bac32934-8a57-4919-b435-13f367cbea3e\") " pod="e2e-models-e2e-worker-w2-a0bcbc/distinct-w2-a0bcbc-kserve-596f94c7cd-szl4g" Nov 04 02:49:57.045946 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:49:57.045869 4405 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tv4lr\" (UniqueName: \"kubernetes.io/projected/bac32934-8a57-4919-b435-13f367cbea3e-kube-api-access-tv4lr\") pod \"distinct-w2-a0bcbc-kserve-596f94c7cd-szl4g\" (UID: \"bac32934-8a57-4919-b435-13f367cbea3e\") " pod="e2e-models-e2e-worker-w2-a0bcbc/distinct-w2-a0bcbc-kserve-596f94c7cd-szl4g" Nov 04 02:49:57.073783 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:49:57.073744 4405 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-a0bcbc/distinct-w2-a0bcbc-kserve-596f94c7cd-szl4g" Nov 04 02:49:57.456414 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:49:57.456391 4405 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-a0bcbc/distinct-w2-a0bcbc-kserve-596f94c7cd-szl4g"] Nov 04 02:49:57.458400 ip-10-0-134-125 kubenswrapper[4405]: W1104 02:49:57.458364 4405 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podbac32934_8a57_4919_b435_13f367cbea3e.slice/crio-b4f4a6104efcef2678635173d0214e0719a8cced4f0bfb6bb0893d54defe05e1 WatchSource:0}: Error finding container b4f4a6104efcef2678635173d0214e0719a8cced4f0bfb6bb0893d54defe05e1: Status 404 returned error can't find the container with id b4f4a6104efcef2678635173d0214e0719a8cced4f0bfb6bb0893d54defe05e1 Nov 04 02:49:57.566880 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:49:57.566846 4405 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-a0bcbc/distinct-w2-a0bcbc-kserve-596f94c7cd-szl4g" event={"ID":"bac32934-8a57-4919-b435-13f367cbea3e","Type":"ContainerStarted","Data":"1ca1b0a45ab98ebeb37408d4b3f67c175e2691ff9f4b7cfd0b75093140603b9f"} Nov 04 02:49:57.567049 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:49:57.566891 4405 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-a0bcbc/distinct-w2-a0bcbc-kserve-596f94c7cd-szl4g" event={"ID":"bac32934-8a57-4919-b435-13f367cbea3e","Type":"ContainerStarted","Data":"b4f4a6104efcef2678635173d0214e0719a8cced4f0bfb6bb0893d54defe05e1"} Nov 04 02:49:57.686355 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:49:57.686299 4405 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w2-a0bcbc/distinct-w2-a0bcbc-kserve-596f94c7cd-szl4g" podStartSLOduration=1.686284341 podStartE2EDuration="1.686284341s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 02:49:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 02:49:57.685783614 +0000 UTC m=+1044.116483735" watchObservedRunningTime="2026-11-04 02:49:57.686284341 +0000 UTC m=+1044.116984461" Nov 04 02:49:58.074745 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:49:58.074717 4405 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-a0bcbc/distinct-w2-a0bcbc-kserve-596f94c7cd-szl4g" Nov 04 02:50:01.961714 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:50:01.961644 4405 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-cdcf67-openshift-default-6ff8f5cbc7-vbc52"] Nov 04 02:50:01.966020 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:50:01.965987 4405 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-cdcf67-openshift-default-6ff8f5cbc7-vbc52" Nov 04 02:50:02.038043 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:50:02.037997 4405 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vjznl\" (UniqueName: \"kubernetes.io/projected/8db652a6-107c-4696-a6f7-009cc7854aa3-kube-api-access-vjznl\") pod \"e2e-worker-w3-cdcf67-openshift-default-6ff8f5cbc7-vbc52\" (UID: \"8db652a6-107c-4696-a6f7-009cc7854aa3\") " pod="openshift-ingress/e2e-worker-w3-cdcf67-openshift-default-6ff8f5cbc7-vbc52" Nov 04 02:50:02.038207 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:50:02.038074 4405 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8db652a6-107c-4696-a6f7-009cc7854aa3-workload-certs\") pod \"e2e-worker-w3-cdcf67-openshift-default-6ff8f5cbc7-vbc52\" (UID: \"8db652a6-107c-4696-a6f7-009cc7854aa3\") " pod="openshift-ingress/e2e-worker-w3-cdcf67-openshift-default-6ff8f5cbc7-vbc52" Nov 04 02:50:02.038207 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:50:02.038130 4405 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8db652a6-107c-4696-a6f7-009cc7854aa3-istio-token\") pod \"e2e-worker-w3-cdcf67-openshift-default-6ff8f5cbc7-vbc52\" (UID: \"8db652a6-107c-4696-a6f7-009cc7854aa3\") " pod="openshift-ingress/e2e-worker-w3-cdcf67-openshift-default-6ff8f5cbc7-vbc52" Nov 04 02:50:02.038307 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:50:02.038229 4405 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8db652a6-107c-4696-a6f7-009cc7854aa3-credential-socket\") pod \"e2e-worker-w3-cdcf67-openshift-default-6ff8f5cbc7-vbc52\" (UID: \"8db652a6-107c-4696-a6f7-009cc7854aa3\") " pod="openshift-ingress/e2e-worker-w3-cdcf67-openshift-default-6ff8f5cbc7-vbc52" Nov 04 02:50:02.038307 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:50:02.038262 4405 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8db652a6-107c-4696-a6f7-009cc7854aa3-istiod-ca-cert\") pod \"e2e-worker-w3-cdcf67-openshift-default-6ff8f5cbc7-vbc52\" (UID: \"8db652a6-107c-4696-a6f7-009cc7854aa3\") " pod="openshift-ingress/e2e-worker-w3-cdcf67-openshift-default-6ff8f5cbc7-vbc52" Nov 04 02:50:02.038307 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:50:02.038301 4405 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8db652a6-107c-4696-a6f7-009cc7854aa3-workload-socket\") pod \"e2e-worker-w3-cdcf67-openshift-default-6ff8f5cbc7-vbc52\" (UID: \"8db652a6-107c-4696-a6f7-009cc7854aa3\") " pod="openshift-ingress/e2e-worker-w3-cdcf67-openshift-default-6ff8f5cbc7-vbc52" Nov 04 02:50:02.038395 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:50:02.038351 4405 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8db652a6-107c-4696-a6f7-009cc7854aa3-istio-podinfo\") pod \"e2e-worker-w3-cdcf67-openshift-default-6ff8f5cbc7-vbc52\" (UID: \"8db652a6-107c-4696-a6f7-009cc7854aa3\") " pod="openshift-ingress/e2e-worker-w3-cdcf67-openshift-default-6ff8f5cbc7-vbc52" Nov 04 02:50:02.038431 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:50:02.038406 4405 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8db652a6-107c-4696-a6f7-009cc7854aa3-istio-envoy\") pod \"e2e-worker-w3-cdcf67-openshift-default-6ff8f5cbc7-vbc52\" (UID: \"8db652a6-107c-4696-a6f7-009cc7854aa3\") " pod="openshift-ingress/e2e-worker-w3-cdcf67-openshift-default-6ff8f5cbc7-vbc52" Nov 04 02:50:02.038462 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:50:02.038451 4405 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8db652a6-107c-4696-a6f7-009cc7854aa3-istio-data\") pod \"e2e-worker-w3-cdcf67-openshift-default-6ff8f5cbc7-vbc52\" (UID: \"8db652a6-107c-4696-a6f7-009cc7854aa3\") " pod="openshift-ingress/e2e-worker-w3-cdcf67-openshift-default-6ff8f5cbc7-vbc52" Nov 04 02:50:02.061712 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:50:02.061635 4405 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-cdcf67-openshift-default-6ff8f5cbc7-vbc52"] Nov 04 02:50:02.139757 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:50:02.139713 4405 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8db652a6-107c-4696-a6f7-009cc7854aa3-istiod-ca-cert\") pod \"e2e-worker-w3-cdcf67-openshift-default-6ff8f5cbc7-vbc52\" (UID: \"8db652a6-107c-4696-a6f7-009cc7854aa3\") " pod="openshift-ingress/e2e-worker-w3-cdcf67-openshift-default-6ff8f5cbc7-vbc52" Nov 04 02:50:02.139757 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:50:02.139765 4405 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8db652a6-107c-4696-a6f7-009cc7854aa3-workload-socket\") pod \"e2e-worker-w3-cdcf67-openshift-default-6ff8f5cbc7-vbc52\" (UID: \"8db652a6-107c-4696-a6f7-009cc7854aa3\") " pod="openshift-ingress/e2e-worker-w3-cdcf67-openshift-default-6ff8f5cbc7-vbc52" Nov 04 02:50:02.140040 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:50:02.139791 4405 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8db652a6-107c-4696-a6f7-009cc7854aa3-istio-podinfo\") pod \"e2e-worker-w3-cdcf67-openshift-default-6ff8f5cbc7-vbc52\" (UID: \"8db652a6-107c-4696-a6f7-009cc7854aa3\") " pod="openshift-ingress/e2e-worker-w3-cdcf67-openshift-default-6ff8f5cbc7-vbc52" Nov 04 02:50:02.140040 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:50:02.139834 4405 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8db652a6-107c-4696-a6f7-009cc7854aa3-istio-envoy\") pod \"e2e-worker-w3-cdcf67-openshift-default-6ff8f5cbc7-vbc52\" (UID: \"8db652a6-107c-4696-a6f7-009cc7854aa3\") " pod="openshift-ingress/e2e-worker-w3-cdcf67-openshift-default-6ff8f5cbc7-vbc52" Nov 04 02:50:02.140040 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:50:02.139850 4405 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8db652a6-107c-4696-a6f7-009cc7854aa3-istio-data\") pod \"e2e-worker-w3-cdcf67-openshift-default-6ff8f5cbc7-vbc52\" (UID: \"8db652a6-107c-4696-a6f7-009cc7854aa3\") " pod="openshift-ingress/e2e-worker-w3-cdcf67-openshift-default-6ff8f5cbc7-vbc52" Nov 04 02:50:02.140040 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:50:02.139913 4405 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vjznl\" (UniqueName: \"kubernetes.io/projected/8db652a6-107c-4696-a6f7-009cc7854aa3-kube-api-access-vjznl\") pod \"e2e-worker-w3-cdcf67-openshift-default-6ff8f5cbc7-vbc52\" (UID: \"8db652a6-107c-4696-a6f7-009cc7854aa3\") " pod="openshift-ingress/e2e-worker-w3-cdcf67-openshift-default-6ff8f5cbc7-vbc52" Nov 04 02:50:02.140040 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:50:02.139979 4405 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8db652a6-107c-4696-a6f7-009cc7854aa3-workload-certs\") pod \"e2e-worker-w3-cdcf67-openshift-default-6ff8f5cbc7-vbc52\" (UID: \"8db652a6-107c-4696-a6f7-009cc7854aa3\") " pod="openshift-ingress/e2e-worker-w3-cdcf67-openshift-default-6ff8f5cbc7-vbc52" Nov 04 02:50:02.140040 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:50:02.140003 4405 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8db652a6-107c-4696-a6f7-009cc7854aa3-istio-token\") pod \"e2e-worker-w3-cdcf67-openshift-default-6ff8f5cbc7-vbc52\" (UID: \"8db652a6-107c-4696-a6f7-009cc7854aa3\") " pod="openshift-ingress/e2e-worker-w3-cdcf67-openshift-default-6ff8f5cbc7-vbc52" Nov 04 02:50:02.140329 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:50:02.140065 4405 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8db652a6-107c-4696-a6f7-009cc7854aa3-credential-socket\") pod \"e2e-worker-w3-cdcf67-openshift-default-6ff8f5cbc7-vbc52\" (UID: \"8db652a6-107c-4696-a6f7-009cc7854aa3\") " pod="openshift-ingress/e2e-worker-w3-cdcf67-openshift-default-6ff8f5cbc7-vbc52" Nov 04 02:50:02.140329 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:50:02.140275 4405 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8db652a6-107c-4696-a6f7-009cc7854aa3-istio-data\") pod \"e2e-worker-w3-cdcf67-openshift-default-6ff8f5cbc7-vbc52\" (UID: \"8db652a6-107c-4696-a6f7-009cc7854aa3\") " pod="openshift-ingress/e2e-worker-w3-cdcf67-openshift-default-6ff8f5cbc7-vbc52" Nov 04 02:50:02.140434 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:50:02.140345 4405 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8db652a6-107c-4696-a6f7-009cc7854aa3-workload-socket\") pod \"e2e-worker-w3-cdcf67-openshift-default-6ff8f5cbc7-vbc52\" (UID: \"8db652a6-107c-4696-a6f7-009cc7854aa3\") " pod="openshift-ingress/e2e-worker-w3-cdcf67-openshift-default-6ff8f5cbc7-vbc52" Nov 04 02:50:02.140478 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:50:02.140455 4405 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8db652a6-107c-4696-a6f7-009cc7854aa3-credential-socket\") pod \"e2e-worker-w3-cdcf67-openshift-default-6ff8f5cbc7-vbc52\" (UID: \"8db652a6-107c-4696-a6f7-009cc7854aa3\") " pod="openshift-ingress/e2e-worker-w3-cdcf67-openshift-default-6ff8f5cbc7-vbc52" Nov 04 02:50:02.140532 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:50:02.140481 4405 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8db652a6-107c-4696-a6f7-009cc7854aa3-workload-certs\") pod \"e2e-worker-w3-cdcf67-openshift-default-6ff8f5cbc7-vbc52\" (UID: \"8db652a6-107c-4696-a6f7-009cc7854aa3\") " pod="openshift-ingress/e2e-worker-w3-cdcf67-openshift-default-6ff8f5cbc7-vbc52" Nov 04 02:50:02.140597 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:50:02.140574 4405 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8db652a6-107c-4696-a6f7-009cc7854aa3-istiod-ca-cert\") pod \"e2e-worker-w3-cdcf67-openshift-default-6ff8f5cbc7-vbc52\" (UID: \"8db652a6-107c-4696-a6f7-009cc7854aa3\") " pod="openshift-ingress/e2e-worker-w3-cdcf67-openshift-default-6ff8f5cbc7-vbc52" Nov 04 02:50:02.142543 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:50:02.142519 4405 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8db652a6-107c-4696-a6f7-009cc7854aa3-istio-podinfo\") pod \"e2e-worker-w3-cdcf67-openshift-default-6ff8f5cbc7-vbc52\" (UID: \"8db652a6-107c-4696-a6f7-009cc7854aa3\") " pod="openshift-ingress/e2e-worker-w3-cdcf67-openshift-default-6ff8f5cbc7-vbc52" Nov 04 02:50:02.142763 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:50:02.142744 4405 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8db652a6-107c-4696-a6f7-009cc7854aa3-istio-envoy\") pod \"e2e-worker-w3-cdcf67-openshift-default-6ff8f5cbc7-vbc52\" (UID: \"8db652a6-107c-4696-a6f7-009cc7854aa3\") " pod="openshift-ingress/e2e-worker-w3-cdcf67-openshift-default-6ff8f5cbc7-vbc52" Nov 04 02:50:02.201197 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:50:02.201155 4405 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vjznl\" (UniqueName: \"kubernetes.io/projected/8db652a6-107c-4696-a6f7-009cc7854aa3-kube-api-access-vjznl\") pod \"e2e-worker-w3-cdcf67-openshift-default-6ff8f5cbc7-vbc52\" (UID: \"8db652a6-107c-4696-a6f7-009cc7854aa3\") " pod="openshift-ingress/e2e-worker-w3-cdcf67-openshift-default-6ff8f5cbc7-vbc52" Nov 04 02:50:02.201369 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:50:02.201234 4405 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8db652a6-107c-4696-a6f7-009cc7854aa3-istio-token\") pod \"e2e-worker-w3-cdcf67-openshift-default-6ff8f5cbc7-vbc52\" (UID: \"8db652a6-107c-4696-a6f7-009cc7854aa3\") " pod="openshift-ingress/e2e-worker-w3-cdcf67-openshift-default-6ff8f5cbc7-vbc52" Nov 04 02:50:02.282016 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:50:02.281928 4405 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-cdcf67-openshift-default-6ff8f5cbc7-vbc52" Nov 04 02:50:02.714519 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:50:02.714482 4405 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-cdcf67-openshift-default-6ff8f5cbc7-vbc52"] Nov 04 02:50:02.718986 ip-10-0-134-125 kubenswrapper[4405]: W1104 02:50:02.718951 4405 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8db652a6_107c_4696_a6f7_009cc7854aa3.slice/crio-ca77e9275f3792de686421eef979b09048d27628aa2f03d66a0639bec9176c2d WatchSource:0}: Error finding container ca77e9275f3792de686421eef979b09048d27628aa2f03d66a0639bec9176c2d: Status 404 returned error can't find the container with id ca77e9275f3792de686421eef979b09048d27628aa2f03d66a0639bec9176c2d Nov 04 02:50:02.721872 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:50:02.721841 4405 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 04 02:50:02.721948 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:50:02.721909 4405 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 04 02:50:02.721948 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:50:02.721942 4405 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 04 02:50:03.625840 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:50:03.625800 4405 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-cdcf67-openshift-default-6ff8f5cbc7-vbc52" event={"ID":"8db652a6-107c-4696-a6f7-009cc7854aa3","Type":"ContainerStarted","Data":"96ed3be429ca5affc319b3368c190a474ad89ee7b9a139b572f11ca848ae44b9"} Nov 04 02:50:03.626457 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:50:03.625871 4405 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-cdcf67-openshift-default-6ff8f5cbc7-vbc52" event={"ID":"8db652a6-107c-4696-a6f7-009cc7854aa3","Type":"ContainerStarted","Data":"ca77e9275f3792de686421eef979b09048d27628aa2f03d66a0639bec9176c2d"} Nov 04 02:50:03.789168 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:50:03.789115 4405 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-cdcf67-openshift-default-6ff8f5cbc7-vbc52" podStartSLOduration=2.789100597 podStartE2EDuration="2.789100597s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 02:50:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 02:50:03.786997252 +0000 UTC m=+1050.217697372" watchObservedRunningTime="2026-11-04 02:50:03.789100597 +0000 UTC m=+1050.219800716" Nov 04 02:50:04.282462 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:50:04.282417 4405 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-cdcf67-openshift-default-6ff8f5cbc7-vbc52" Nov 04 02:50:04.284420 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:50:04.284397 4405 patch_prober.go:29] interesting pod/e2e-worker-w3-cdcf67-openshift-default-6ff8f5cbc7-vbc52 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.53:15021/healthz/ready\": dial tcp 10.134.0.53:15021: connect: connection refused" start-of-body= Nov 04 02:50:04.284507 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:50:04.284462 4405 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-cdcf67-openshift-default-6ff8f5cbc7-vbc52" podUID="8db652a6-107c-4696-a6f7-009cc7854aa3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.53:15021/healthz/ready\": dial tcp 10.134.0.53:15021: connect: connection refused" Nov 04 02:50:05.282702 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:50:05.282464 4405 patch_prober.go:29] interesting pod/e2e-worker-w3-cdcf67-openshift-default-6ff8f5cbc7-vbc52 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.53:15021/healthz/ready\": dial tcp 10.134.0.53:15021: connect: connection refused" start-of-body= Nov 04 02:50:05.282702 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:50:05.282556 4405 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-cdcf67-openshift-default-6ff8f5cbc7-vbc52" podUID="8db652a6-107c-4696-a6f7-009cc7854aa3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.53:15021/healthz/ready\": dial tcp 10.134.0.53:15021: connect: connection refused" Nov 04 02:50:06.282756 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:50:06.282719 4405 patch_prober.go:29] interesting pod/e2e-worker-w3-cdcf67-openshift-default-6ff8f5cbc7-vbc52 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.53:15021/healthz/ready\": dial tcp 10.134.0.53:15021: connect: connection refused" start-of-body= Nov 04 02:50:06.283136 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:50:06.282775 4405 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-cdcf67-openshift-default-6ff8f5cbc7-vbc52" podUID="8db652a6-107c-4696-a6f7-009cc7854aa3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.53:15021/healthz/ready\": dial tcp 10.134.0.53:15021: connect: connection refused" Nov 04 02:50:06.890702 ip-10-0-134-125 kubenswrapper[4405]: E1104 02:50:06.888809 4405 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5f2vb" containerName="extractor" Nov 04 02:50:07.074075 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:50:07.074039 4405 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w2-a0bcbc/distinct-w2-a0bcbc-kserve-596f94c7cd-szl4g" Nov 04 02:50:07.089607 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:50:07.089576 4405 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w2-a0bcbc/distinct-w2-a0bcbc-kserve-596f94c7cd-szl4g" Nov 04 02:50:07.282624 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:50:07.282531 4405 patch_prober.go:29] interesting pod/e2e-worker-w3-cdcf67-openshift-default-6ff8f5cbc7-vbc52 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.53:15021/healthz/ready\": dial tcp 10.134.0.53:15021: connect: connection refused" start-of-body= Nov 04 02:50:07.282624 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:50:07.282583 4405 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-cdcf67-openshift-default-6ff8f5cbc7-vbc52" podUID="8db652a6-107c-4696-a6f7-009cc7854aa3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.53:15021/healthz/ready\": dial tcp 10.134.0.53:15021: connect: connection refused" Nov 04 02:50:07.670715 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:50:07.670648 4405 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-a0bcbc/distinct-w2-a0bcbc-kserve-596f94c7cd-szl4g" Nov 04 02:50:08.282372 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:50:08.282335 4405 patch_prober.go:29] interesting pod/e2e-worker-w3-cdcf67-openshift-default-6ff8f5cbc7-vbc52 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.53:15021/healthz/ready\": dial tcp 10.134.0.53:15021: connect: connection refused" start-of-body= Nov 04 02:50:08.282588 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:50:08.282392 4405 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-cdcf67-openshift-default-6ff8f5cbc7-vbc52" podUID="8db652a6-107c-4696-a6f7-009cc7854aa3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.53:15021/healthz/ready\": dial tcp 10.134.0.53:15021: connect: connection refused" Nov 04 02:50:09.284182 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:50:09.284142 4405 patch_prober.go:29] interesting pod/e2e-worker-w3-cdcf67-openshift-default-6ff8f5cbc7-vbc52 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.53:15021/healthz/ready\": dial tcp 10.134.0.53:15021: connect: connection refused" start-of-body= Nov 04 02:50:09.284806 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:50:09.284770 4405 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-cdcf67-openshift-default-6ff8f5cbc7-vbc52" podUID="8db652a6-107c-4696-a6f7-009cc7854aa3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.53:15021/healthz/ready\": dial tcp 10.134.0.53:15021: connect: connection refused" Nov 04 02:50:10.724761 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:50:10.720766 4405 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-cdcf67-openshift-default-6ff8f5cbc7-vbc52" Nov 04 02:50:10.724761 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:50:10.722177 4405 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-cdcf67-openshift-default-6ff8f5cbc7-vbc52" Nov 04 02:50:10.724761 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:50:10.723514 4405 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-cdcf67-openshift-default-6ff8f5cbc7-vbc52" Nov 04 02:50:16.834891 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:50:16.834854 4405 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-a0bcbc-openshift-default-65dc9d9b6c-j4n6p"] Nov 04 02:50:16.840270 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:50:16.840247 4405 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-a0bcbc-openshift-default-65dc9d9b6c-j4n6p" Nov 04 02:50:16.940252 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:50:16.940216 4405 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-a0bcbc-openshift-default-65dc9d9b6c-j4n6p"] Nov 04 02:50:16.942311 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:50:16.942277 4405 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/55473cc8-d292-4f5f-bb17-3f08c2e19e8c-workload-certs\") pod \"e2e-worker-w2-a0bcbc-openshift-default-65dc9d9b6c-j4n6p\" (UID: \"55473cc8-d292-4f5f-bb17-3f08c2e19e8c\") " pod="openshift-ingress/e2e-worker-w2-a0bcbc-openshift-default-65dc9d9b6c-j4n6p" Nov 04 02:50:16.942477 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:50:16.942441 4405 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/55473cc8-d292-4f5f-bb17-3f08c2e19e8c-credential-socket\") pod \"e2e-worker-w2-a0bcbc-openshift-default-65dc9d9b6c-j4n6p\" (UID: \"55473cc8-d292-4f5f-bb17-3f08c2e19e8c\") " pod="openshift-ingress/e2e-worker-w2-a0bcbc-openshift-default-65dc9d9b6c-j4n6p" Nov 04 02:50:16.942604 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:50:16.942577 4405 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/55473cc8-d292-4f5f-bb17-3f08c2e19e8c-workload-socket\") pod \"e2e-worker-w2-a0bcbc-openshift-default-65dc9d9b6c-j4n6p\" (UID: \"55473cc8-d292-4f5f-bb17-3f08c2e19e8c\") " pod="openshift-ingress/e2e-worker-w2-a0bcbc-openshift-default-65dc9d9b6c-j4n6p" Nov 04 02:50:16.942768 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:50:16.942644 4405 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/55473cc8-d292-4f5f-bb17-3f08c2e19e8c-istio-token\") pod \"e2e-worker-w2-a0bcbc-openshift-default-65dc9d9b6c-j4n6p\" (UID: \"55473cc8-d292-4f5f-bb17-3f08c2e19e8c\") " pod="openshift-ingress/e2e-worker-w2-a0bcbc-openshift-default-65dc9d9b6c-j4n6p" Nov 04 02:50:16.942768 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:50:16.942707 4405 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/55473cc8-d292-4f5f-bb17-3f08c2e19e8c-istio-envoy\") pod \"e2e-worker-w2-a0bcbc-openshift-default-65dc9d9b6c-j4n6p\" (UID: \"55473cc8-d292-4f5f-bb17-3f08c2e19e8c\") " pod="openshift-ingress/e2e-worker-w2-a0bcbc-openshift-default-65dc9d9b6c-j4n6p" Nov 04 02:50:16.942768 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:50:16.942738 4405 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/55473cc8-d292-4f5f-bb17-3f08c2e19e8c-istio-data\") pod \"e2e-worker-w2-a0bcbc-openshift-default-65dc9d9b6c-j4n6p\" (UID: \"55473cc8-d292-4f5f-bb17-3f08c2e19e8c\") " pod="openshift-ingress/e2e-worker-w2-a0bcbc-openshift-default-65dc9d9b6c-j4n6p" Nov 04 02:50:16.942906 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:50:16.942863 4405 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-28xj2\" (UniqueName: \"kubernetes.io/projected/55473cc8-d292-4f5f-bb17-3f08c2e19e8c-kube-api-access-28xj2\") pod \"e2e-worker-w2-a0bcbc-openshift-default-65dc9d9b6c-j4n6p\" (UID: \"55473cc8-d292-4f5f-bb17-3f08c2e19e8c\") " pod="openshift-ingress/e2e-worker-w2-a0bcbc-openshift-default-65dc9d9b6c-j4n6p" Nov 04 02:50:16.942962 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:50:16.942923 4405 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/55473cc8-d292-4f5f-bb17-3f08c2e19e8c-istio-podinfo\") pod \"e2e-worker-w2-a0bcbc-openshift-default-65dc9d9b6c-j4n6p\" (UID: \"55473cc8-d292-4f5f-bb17-3f08c2e19e8c\") " pod="openshift-ingress/e2e-worker-w2-a0bcbc-openshift-default-65dc9d9b6c-j4n6p" Nov 04 02:50:16.943019 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:50:16.942983 4405 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/55473cc8-d292-4f5f-bb17-3f08c2e19e8c-istiod-ca-cert\") pod \"e2e-worker-w2-a0bcbc-openshift-default-65dc9d9b6c-j4n6p\" (UID: \"55473cc8-d292-4f5f-bb17-3f08c2e19e8c\") " pod="openshift-ingress/e2e-worker-w2-a0bcbc-openshift-default-65dc9d9b6c-j4n6p" Nov 04 02:50:17.044314 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:50:17.044273 4405 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-28xj2\" (UniqueName: \"kubernetes.io/projected/55473cc8-d292-4f5f-bb17-3f08c2e19e8c-kube-api-access-28xj2\") pod \"e2e-worker-w2-a0bcbc-openshift-default-65dc9d9b6c-j4n6p\" (UID: \"55473cc8-d292-4f5f-bb17-3f08c2e19e8c\") " pod="openshift-ingress/e2e-worker-w2-a0bcbc-openshift-default-65dc9d9b6c-j4n6p" Nov 04 02:50:17.044531 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:50:17.044335 4405 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/55473cc8-d292-4f5f-bb17-3f08c2e19e8c-istio-podinfo\") pod \"e2e-worker-w2-a0bcbc-openshift-default-65dc9d9b6c-j4n6p\" (UID: \"55473cc8-d292-4f5f-bb17-3f08c2e19e8c\") " pod="openshift-ingress/e2e-worker-w2-a0bcbc-openshift-default-65dc9d9b6c-j4n6p" Nov 04 02:50:17.044531 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:50:17.044361 4405 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/55473cc8-d292-4f5f-bb17-3f08c2e19e8c-istiod-ca-cert\") pod \"e2e-worker-w2-a0bcbc-openshift-default-65dc9d9b6c-j4n6p\" (UID: \"55473cc8-d292-4f5f-bb17-3f08c2e19e8c\") " pod="openshift-ingress/e2e-worker-w2-a0bcbc-openshift-default-65dc9d9b6c-j4n6p" Nov 04 02:50:17.044531 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:50:17.044393 4405 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/55473cc8-d292-4f5f-bb17-3f08c2e19e8c-workload-certs\") pod \"e2e-worker-w2-a0bcbc-openshift-default-65dc9d9b6c-j4n6p\" (UID: \"55473cc8-d292-4f5f-bb17-3f08c2e19e8c\") " pod="openshift-ingress/e2e-worker-w2-a0bcbc-openshift-default-65dc9d9b6c-j4n6p" Nov 04 02:50:17.044531 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:50:17.044464 4405 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/55473cc8-d292-4f5f-bb17-3f08c2e19e8c-credential-socket\") pod \"e2e-worker-w2-a0bcbc-openshift-default-65dc9d9b6c-j4n6p\" (UID: \"55473cc8-d292-4f5f-bb17-3f08c2e19e8c\") " pod="openshift-ingress/e2e-worker-w2-a0bcbc-openshift-default-65dc9d9b6c-j4n6p" Nov 04 02:50:17.044531 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:50:17.044511 4405 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/55473cc8-d292-4f5f-bb17-3f08c2e19e8c-workload-socket\") pod \"e2e-worker-w2-a0bcbc-openshift-default-65dc9d9b6c-j4n6p\" (UID: \"55473cc8-d292-4f5f-bb17-3f08c2e19e8c\") " pod="openshift-ingress/e2e-worker-w2-a0bcbc-openshift-default-65dc9d9b6c-j4n6p" Nov 04 02:50:17.044865 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:50:17.044536 4405 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/55473cc8-d292-4f5f-bb17-3f08c2e19e8c-istio-token\") pod \"e2e-worker-w2-a0bcbc-openshift-default-65dc9d9b6c-j4n6p\" (UID: \"55473cc8-d292-4f5f-bb17-3f08c2e19e8c\") " pod="openshift-ingress/e2e-worker-w2-a0bcbc-openshift-default-65dc9d9b6c-j4n6p" Nov 04 02:50:17.044865 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:50:17.044721 4405 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/55473cc8-d292-4f5f-bb17-3f08c2e19e8c-istio-envoy\") pod \"e2e-worker-w2-a0bcbc-openshift-default-65dc9d9b6c-j4n6p\" (UID: \"55473cc8-d292-4f5f-bb17-3f08c2e19e8c\") " pod="openshift-ingress/e2e-worker-w2-a0bcbc-openshift-default-65dc9d9b6c-j4n6p" Nov 04 02:50:17.044865 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:50:17.044768 4405 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/55473cc8-d292-4f5f-bb17-3f08c2e19e8c-istio-data\") pod \"e2e-worker-w2-a0bcbc-openshift-default-65dc9d9b6c-j4n6p\" (UID: \"55473cc8-d292-4f5f-bb17-3f08c2e19e8c\") " pod="openshift-ingress/e2e-worker-w2-a0bcbc-openshift-default-65dc9d9b6c-j4n6p" Nov 04 02:50:17.045021 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:50:17.044915 4405 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/55473cc8-d292-4f5f-bb17-3f08c2e19e8c-credential-socket\") pod \"e2e-worker-w2-a0bcbc-openshift-default-65dc9d9b6c-j4n6p\" (UID: \"55473cc8-d292-4f5f-bb17-3f08c2e19e8c\") " pod="openshift-ingress/e2e-worker-w2-a0bcbc-openshift-default-65dc9d9b6c-j4n6p" Nov 04 02:50:17.045021 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:50:17.045002 4405 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/55473cc8-d292-4f5f-bb17-3f08c2e19e8c-workload-certs\") pod \"e2e-worker-w2-a0bcbc-openshift-default-65dc9d9b6c-j4n6p\" (UID: \"55473cc8-d292-4f5f-bb17-3f08c2e19e8c\") " pod="openshift-ingress/e2e-worker-w2-a0bcbc-openshift-default-65dc9d9b6c-j4n6p" Nov 04 02:50:17.045131 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:50:17.045064 4405 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/55473cc8-d292-4f5f-bb17-3f08c2e19e8c-workload-socket\") pod \"e2e-worker-w2-a0bcbc-openshift-default-65dc9d9b6c-j4n6p\" (UID: \"55473cc8-d292-4f5f-bb17-3f08c2e19e8c\") " pod="openshift-ingress/e2e-worker-w2-a0bcbc-openshift-default-65dc9d9b6c-j4n6p" Nov 04 02:50:17.045188 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:50:17.045159 4405 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/55473cc8-d292-4f5f-bb17-3f08c2e19e8c-istiod-ca-cert\") pod \"e2e-worker-w2-a0bcbc-openshift-default-65dc9d9b6c-j4n6p\" (UID: \"55473cc8-d292-4f5f-bb17-3f08c2e19e8c\") " pod="openshift-ingress/e2e-worker-w2-a0bcbc-openshift-default-65dc9d9b6c-j4n6p" Nov 04 02:50:17.045250 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:50:17.045227 4405 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/55473cc8-d292-4f5f-bb17-3f08c2e19e8c-istio-data\") pod \"e2e-worker-w2-a0bcbc-openshift-default-65dc9d9b6c-j4n6p\" (UID: \"55473cc8-d292-4f5f-bb17-3f08c2e19e8c\") " pod="openshift-ingress/e2e-worker-w2-a0bcbc-openshift-default-65dc9d9b6c-j4n6p" Nov 04 02:50:17.047028 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:50:17.047008 4405 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/55473cc8-d292-4f5f-bb17-3f08c2e19e8c-istio-podinfo\") pod \"e2e-worker-w2-a0bcbc-openshift-default-65dc9d9b6c-j4n6p\" (UID: \"55473cc8-d292-4f5f-bb17-3f08c2e19e8c\") " pod="openshift-ingress/e2e-worker-w2-a0bcbc-openshift-default-65dc9d9b6c-j4n6p" Nov 04 02:50:17.047162 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:50:17.047144 4405 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/55473cc8-d292-4f5f-bb17-3f08c2e19e8c-istio-envoy\") pod \"e2e-worker-w2-a0bcbc-openshift-default-65dc9d9b6c-j4n6p\" (UID: \"55473cc8-d292-4f5f-bb17-3f08c2e19e8c\") " pod="openshift-ingress/e2e-worker-w2-a0bcbc-openshift-default-65dc9d9b6c-j4n6p" Nov 04 02:50:17.107217 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:50:17.107126 4405 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/55473cc8-d292-4f5f-bb17-3f08c2e19e8c-istio-token\") pod \"e2e-worker-w2-a0bcbc-openshift-default-65dc9d9b6c-j4n6p\" (UID: \"55473cc8-d292-4f5f-bb17-3f08c2e19e8c\") " pod="openshift-ingress/e2e-worker-w2-a0bcbc-openshift-default-65dc9d9b6c-j4n6p" Nov 04 02:50:17.107382 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:50:17.107238 4405 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-28xj2\" (UniqueName: \"kubernetes.io/projected/55473cc8-d292-4f5f-bb17-3f08c2e19e8c-kube-api-access-28xj2\") pod \"e2e-worker-w2-a0bcbc-openshift-default-65dc9d9b6c-j4n6p\" (UID: \"55473cc8-d292-4f5f-bb17-3f08c2e19e8c\") " pod="openshift-ingress/e2e-worker-w2-a0bcbc-openshift-default-65dc9d9b6c-j4n6p" Nov 04 02:50:17.155559 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:50:17.155436 4405 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-a0bcbc-openshift-default-65dc9d9b6c-j4n6p" Nov 04 02:50:17.594943 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:50:17.594912 4405 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-a0bcbc-openshift-default-65dc9d9b6c-j4n6p"] Nov 04 02:50:17.596451 ip-10-0-134-125 kubenswrapper[4405]: W1104 02:50:17.596421 4405 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod55473cc8_d292_4f5f_bb17_3f08c2e19e8c.slice/crio-07ea17ab897809cbcbab549a3a4a53a38e10dd34ee720910bffcc1c8439260b7 WatchSource:0}: Error finding container 07ea17ab897809cbcbab549a3a4a53a38e10dd34ee720910bffcc1c8439260b7: Status 404 returned error can't find the container with id 07ea17ab897809cbcbab549a3a4a53a38e10dd34ee720910bffcc1c8439260b7 Nov 04 02:50:17.599473 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:50:17.599419 4405 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 04 02:50:17.599593 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:50:17.599527 4405 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 04 02:50:17.599593 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:50:17.599585 4405 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 04 02:50:17.722578 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:50:17.722536 4405 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-e516ff-openshift-default-6f4b957485-7pg2l"] Nov 04 02:50:17.727142 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:50:17.727111 4405 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-e516ff-openshift-default-6f4b957485-7pg2l" Nov 04 02:50:17.753364 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:50:17.753327 4405 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8fc7b14e-09f9-4630-bfc6-b98eab67753f-workload-certs\") pod \"e2e-worker-w1-e516ff-openshift-default-6f4b957485-7pg2l\" (UID: \"8fc7b14e-09f9-4630-bfc6-b98eab67753f\") " pod="openshift-ingress/e2e-worker-w1-e516ff-openshift-default-6f4b957485-7pg2l" Nov 04 02:50:17.753528 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:50:17.753370 4405 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qh6vr\" (UniqueName: \"kubernetes.io/projected/8fc7b14e-09f9-4630-bfc6-b98eab67753f-kube-api-access-qh6vr\") pod \"e2e-worker-w1-e516ff-openshift-default-6f4b957485-7pg2l\" (UID: \"8fc7b14e-09f9-4630-bfc6-b98eab67753f\") " pod="openshift-ingress/e2e-worker-w1-e516ff-openshift-default-6f4b957485-7pg2l" Nov 04 02:50:17.753528 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:50:17.753404 4405 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8fc7b14e-09f9-4630-bfc6-b98eab67753f-istio-data\") pod \"e2e-worker-w1-e516ff-openshift-default-6f4b957485-7pg2l\" (UID: \"8fc7b14e-09f9-4630-bfc6-b98eab67753f\") " pod="openshift-ingress/e2e-worker-w1-e516ff-openshift-default-6f4b957485-7pg2l" Nov 04 02:50:17.753528 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:50:17.753463 4405 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8fc7b14e-09f9-4630-bfc6-b98eab67753f-istiod-ca-cert\") pod \"e2e-worker-w1-e516ff-openshift-default-6f4b957485-7pg2l\" (UID: \"8fc7b14e-09f9-4630-bfc6-b98eab67753f\") " pod="openshift-ingress/e2e-worker-w1-e516ff-openshift-default-6f4b957485-7pg2l" Nov 04 02:50:17.753738 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:50:17.753601 4405 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8fc7b14e-09f9-4630-bfc6-b98eab67753f-istio-envoy\") pod \"e2e-worker-w1-e516ff-openshift-default-6f4b957485-7pg2l\" (UID: \"8fc7b14e-09f9-4630-bfc6-b98eab67753f\") " pod="openshift-ingress/e2e-worker-w1-e516ff-openshift-default-6f4b957485-7pg2l" Nov 04 02:50:17.753738 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:50:17.753729 4405 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8fc7b14e-09f9-4630-bfc6-b98eab67753f-workload-socket\") pod \"e2e-worker-w1-e516ff-openshift-default-6f4b957485-7pg2l\" (UID: \"8fc7b14e-09f9-4630-bfc6-b98eab67753f\") " pod="openshift-ingress/e2e-worker-w1-e516ff-openshift-default-6f4b957485-7pg2l" Nov 04 02:50:17.753865 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:50:17.753766 4405 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8fc7b14e-09f9-4630-bfc6-b98eab67753f-credential-socket\") pod \"e2e-worker-w1-e516ff-openshift-default-6f4b957485-7pg2l\" (UID: \"8fc7b14e-09f9-4630-bfc6-b98eab67753f\") " pod="openshift-ingress/e2e-worker-w1-e516ff-openshift-default-6f4b957485-7pg2l" Nov 04 02:50:17.753865 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:50:17.753796 4405 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8fc7b14e-09f9-4630-bfc6-b98eab67753f-istio-podinfo\") pod \"e2e-worker-w1-e516ff-openshift-default-6f4b957485-7pg2l\" (UID: \"8fc7b14e-09f9-4630-bfc6-b98eab67753f\") " pod="openshift-ingress/e2e-worker-w1-e516ff-openshift-default-6f4b957485-7pg2l" Nov 04 02:50:17.753865 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:50:17.753832 4405 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8fc7b14e-09f9-4630-bfc6-b98eab67753f-istio-token\") pod \"e2e-worker-w1-e516ff-openshift-default-6f4b957485-7pg2l\" (UID: \"8fc7b14e-09f9-4630-bfc6-b98eab67753f\") " pod="openshift-ingress/e2e-worker-w1-e516ff-openshift-default-6f4b957485-7pg2l" Nov 04 02:50:17.767012 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:50:17.766805 4405 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-a0bcbc-openshift-default-65dc9d9b6c-j4n6p" event={"ID":"55473cc8-d292-4f5f-bb17-3f08c2e19e8c","Type":"ContainerStarted","Data":"d0483ea48aad53b792dd83f987bec31e3af66acf77e8ca0b0d1c64621a92d5d4"} Nov 04 02:50:17.767012 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:50:17.766865 4405 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-a0bcbc-openshift-default-65dc9d9b6c-j4n6p" event={"ID":"55473cc8-d292-4f5f-bb17-3f08c2e19e8c","Type":"ContainerStarted","Data":"07ea17ab897809cbcbab549a3a4a53a38e10dd34ee720910bffcc1c8439260b7"} Nov 04 02:50:17.789559 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:50:17.789525 4405 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-e516ff-openshift-default-6f4b957485-7pg2l"] Nov 04 02:50:17.855159 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:50:17.855054 4405 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8fc7b14e-09f9-4630-bfc6-b98eab67753f-workload-certs\") pod \"e2e-worker-w1-e516ff-openshift-default-6f4b957485-7pg2l\" (UID: \"8fc7b14e-09f9-4630-bfc6-b98eab67753f\") " pod="openshift-ingress/e2e-worker-w1-e516ff-openshift-default-6f4b957485-7pg2l" Nov 04 02:50:17.855159 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:50:17.855102 4405 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qh6vr\" (UniqueName: \"kubernetes.io/projected/8fc7b14e-09f9-4630-bfc6-b98eab67753f-kube-api-access-qh6vr\") pod \"e2e-worker-w1-e516ff-openshift-default-6f4b957485-7pg2l\" (UID: \"8fc7b14e-09f9-4630-bfc6-b98eab67753f\") " pod="openshift-ingress/e2e-worker-w1-e516ff-openshift-default-6f4b957485-7pg2l" Nov 04 02:50:17.855159 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:50:17.855131 4405 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8fc7b14e-09f9-4630-bfc6-b98eab67753f-istio-data\") pod \"e2e-worker-w1-e516ff-openshift-default-6f4b957485-7pg2l\" (UID: \"8fc7b14e-09f9-4630-bfc6-b98eab67753f\") " pod="openshift-ingress/e2e-worker-w1-e516ff-openshift-default-6f4b957485-7pg2l" Nov 04 02:50:17.855808 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:50:17.855158 4405 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8fc7b14e-09f9-4630-bfc6-b98eab67753f-istiod-ca-cert\") pod \"e2e-worker-w1-e516ff-openshift-default-6f4b957485-7pg2l\" (UID: \"8fc7b14e-09f9-4630-bfc6-b98eab67753f\") " pod="openshift-ingress/e2e-worker-w1-e516ff-openshift-default-6f4b957485-7pg2l" Nov 04 02:50:17.855808 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:50:17.855548 4405 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8fc7b14e-09f9-4630-bfc6-b98eab67753f-istio-envoy\") pod \"e2e-worker-w1-e516ff-openshift-default-6f4b957485-7pg2l\" (UID: \"8fc7b14e-09f9-4630-bfc6-b98eab67753f\") " pod="openshift-ingress/e2e-worker-w1-e516ff-openshift-default-6f4b957485-7pg2l" Nov 04 02:50:17.855808 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:50:17.855659 4405 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8fc7b14e-09f9-4630-bfc6-b98eab67753f-workload-socket\") pod \"e2e-worker-w1-e516ff-openshift-default-6f4b957485-7pg2l\" (UID: \"8fc7b14e-09f9-4630-bfc6-b98eab67753f\") " pod="openshift-ingress/e2e-worker-w1-e516ff-openshift-default-6f4b957485-7pg2l" Nov 04 02:50:17.855808 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:50:17.855734 4405 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8fc7b14e-09f9-4630-bfc6-b98eab67753f-credential-socket\") pod \"e2e-worker-w1-e516ff-openshift-default-6f4b957485-7pg2l\" (UID: \"8fc7b14e-09f9-4630-bfc6-b98eab67753f\") " pod="openshift-ingress/e2e-worker-w1-e516ff-openshift-default-6f4b957485-7pg2l" Nov 04 02:50:17.855808 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:50:17.855767 4405 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8fc7b14e-09f9-4630-bfc6-b98eab67753f-istio-podinfo\") pod \"e2e-worker-w1-e516ff-openshift-default-6f4b957485-7pg2l\" (UID: \"8fc7b14e-09f9-4630-bfc6-b98eab67753f\") " pod="openshift-ingress/e2e-worker-w1-e516ff-openshift-default-6f4b957485-7pg2l" Nov 04 02:50:17.856071 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:50:17.855837 4405 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8fc7b14e-09f9-4630-bfc6-b98eab67753f-istio-token\") pod \"e2e-worker-w1-e516ff-openshift-default-6f4b957485-7pg2l\" (UID: \"8fc7b14e-09f9-4630-bfc6-b98eab67753f\") " pod="openshift-ingress/e2e-worker-w1-e516ff-openshift-default-6f4b957485-7pg2l" Nov 04 02:50:17.856071 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:50:17.855882 4405 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8fc7b14e-09f9-4630-bfc6-b98eab67753f-istio-data\") pod \"e2e-worker-w1-e516ff-openshift-default-6f4b957485-7pg2l\" (UID: \"8fc7b14e-09f9-4630-bfc6-b98eab67753f\") " pod="openshift-ingress/e2e-worker-w1-e516ff-openshift-default-6f4b957485-7pg2l" Nov 04 02:50:17.856071 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:50:17.855920 4405 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8fc7b14e-09f9-4630-bfc6-b98eab67753f-istiod-ca-cert\") pod \"e2e-worker-w1-e516ff-openshift-default-6f4b957485-7pg2l\" (UID: \"8fc7b14e-09f9-4630-bfc6-b98eab67753f\") " pod="openshift-ingress/e2e-worker-w1-e516ff-openshift-default-6f4b957485-7pg2l" Nov 04 02:50:17.856234 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:50:17.856195 4405 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8fc7b14e-09f9-4630-bfc6-b98eab67753f-workload-certs\") pod \"e2e-worker-w1-e516ff-openshift-default-6f4b957485-7pg2l\" (UID: \"8fc7b14e-09f9-4630-bfc6-b98eab67753f\") " pod="openshift-ingress/e2e-worker-w1-e516ff-openshift-default-6f4b957485-7pg2l" Nov 04 02:50:17.856291 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:50:17.856246 4405 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8fc7b14e-09f9-4630-bfc6-b98eab67753f-workload-socket\") pod \"e2e-worker-w1-e516ff-openshift-default-6f4b957485-7pg2l\" (UID: \"8fc7b14e-09f9-4630-bfc6-b98eab67753f\") " pod="openshift-ingress/e2e-worker-w1-e516ff-openshift-default-6f4b957485-7pg2l" Nov 04 02:50:17.857094 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:50:17.857072 4405 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8fc7b14e-09f9-4630-bfc6-b98eab67753f-credential-socket\") pod \"e2e-worker-w1-e516ff-openshift-default-6f4b957485-7pg2l\" (UID: \"8fc7b14e-09f9-4630-bfc6-b98eab67753f\") " pod="openshift-ingress/e2e-worker-w1-e516ff-openshift-default-6f4b957485-7pg2l" Nov 04 02:50:17.858603 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:50:17.858583 4405 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8fc7b14e-09f9-4630-bfc6-b98eab67753f-istio-podinfo\") pod \"e2e-worker-w1-e516ff-openshift-default-6f4b957485-7pg2l\" (UID: \"8fc7b14e-09f9-4630-bfc6-b98eab67753f\") " pod="openshift-ingress/e2e-worker-w1-e516ff-openshift-default-6f4b957485-7pg2l" Nov 04 02:50:17.859124 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:50:17.859106 4405 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8fc7b14e-09f9-4630-bfc6-b98eab67753f-istio-envoy\") pod \"e2e-worker-w1-e516ff-openshift-default-6f4b957485-7pg2l\" (UID: \"8fc7b14e-09f9-4630-bfc6-b98eab67753f\") " pod="openshift-ingress/e2e-worker-w1-e516ff-openshift-default-6f4b957485-7pg2l" Nov 04 02:50:17.917010 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:50:17.916976 4405 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8fc7b14e-09f9-4630-bfc6-b98eab67753f-istio-token\") pod \"e2e-worker-w1-e516ff-openshift-default-6f4b957485-7pg2l\" (UID: \"8fc7b14e-09f9-4630-bfc6-b98eab67753f\") " pod="openshift-ingress/e2e-worker-w1-e516ff-openshift-default-6f4b957485-7pg2l" Nov 04 02:50:17.917194 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:50:17.917031 4405 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qh6vr\" (UniqueName: \"kubernetes.io/projected/8fc7b14e-09f9-4630-bfc6-b98eab67753f-kube-api-access-qh6vr\") pod \"e2e-worker-w1-e516ff-openshift-default-6f4b957485-7pg2l\" (UID: \"8fc7b14e-09f9-4630-bfc6-b98eab67753f\") " pod="openshift-ingress/e2e-worker-w1-e516ff-openshift-default-6f4b957485-7pg2l" Nov 04 02:50:17.967547 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:50:17.967479 4405 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-a0bcbc-openshift-default-65dc9d9b6c-j4n6p" podStartSLOduration=1.967459872 podStartE2EDuration="1.967459872s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 02:50:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 02:50:17.966526278 +0000 UTC m=+1064.397226399" watchObservedRunningTime="2026-11-04 02:50:17.967459872 +0000 UTC m=+1064.398159993" Nov 04 02:50:18.042126 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:50:18.042088 4405 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-e516ff-openshift-default-6f4b957485-7pg2l" Nov 04 02:50:18.156758 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:50:18.156646 4405 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-a0bcbc-openshift-default-65dc9d9b6c-j4n6p" Nov 04 02:50:18.160711 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:50:18.160492 4405 patch_prober.go:29] interesting pod/e2e-worker-w2-a0bcbc-openshift-default-65dc9d9b6c-j4n6p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.54:15021/healthz/ready\": dial tcp 10.134.0.54:15021: connect: connection refused" start-of-body= Nov 04 02:50:18.160711 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:50:18.160550 4405 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-a0bcbc-openshift-default-65dc9d9b6c-j4n6p" podUID="55473cc8-d292-4f5f-bb17-3f08c2e19e8c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.54:15021/healthz/ready\": dial tcp 10.134.0.54:15021: connect: connection refused" Nov 04 02:50:18.521600 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:50:18.521451 4405 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-e516ff-openshift-default-6f4b957485-7pg2l"] Nov 04 02:50:18.527341 ip-10-0-134-125 kubenswrapper[4405]: W1104 02:50:18.527299 4405 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8fc7b14e_09f9_4630_bfc6_b98eab67753f.slice/crio-f7490e5285cf059ef915ad60914f6db9b77e4aeb088169a635e5fb82ef76b3da WatchSource:0}: Error finding container f7490e5285cf059ef915ad60914f6db9b77e4aeb088169a635e5fb82ef76b3da: Status 404 returned error can't find the container with id f7490e5285cf059ef915ad60914f6db9b77e4aeb088169a635e5fb82ef76b3da Nov 04 02:50:18.530283 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:50:18.530238 4405 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 04 02:50:18.530397 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:50:18.530337 4405 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 04 02:50:18.530397 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:50:18.530380 4405 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 04 02:50:18.774789 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:50:18.774690 4405 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-e516ff-openshift-default-6f4b957485-7pg2l" event={"ID":"8fc7b14e-09f9-4630-bfc6-b98eab67753f","Type":"ContainerStarted","Data":"5ff33d8a8d5b1b7f130a605e52248650a506b2ee611d906f466d983b36b612ac"} Nov 04 02:50:18.774789 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:50:18.774757 4405 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-e516ff-openshift-default-6f4b957485-7pg2l" event={"ID":"8fc7b14e-09f9-4630-bfc6-b98eab67753f","Type":"ContainerStarted","Data":"f7490e5285cf059ef915ad60914f6db9b77e4aeb088169a635e5fb82ef76b3da"} Nov 04 02:50:18.897805 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:50:18.897743 4405 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-e516ff-openshift-default-6f4b957485-7pg2l" podStartSLOduration=1.897722821 podStartE2EDuration="1.897722821s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 02:50:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 02:50:18.894231721 +0000 UTC m=+1065.324931842" watchObservedRunningTime="2026-11-04 02:50:18.897722821 +0000 UTC m=+1065.328422945" Nov 04 02:50:19.042423 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:50:19.042346 4405 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-e516ff-openshift-default-6f4b957485-7pg2l" Nov 04 02:50:19.044117 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:50:19.044090 4405 patch_prober.go:29] interesting pod/e2e-worker-w1-e516ff-openshift-default-6f4b957485-7pg2l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.55:15021/healthz/ready\": dial tcp 10.134.0.55:15021: connect: connection refused" start-of-body= Nov 04 02:50:19.044263 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:50:19.044150 4405 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-e516ff-openshift-default-6f4b957485-7pg2l" podUID="8fc7b14e-09f9-4630-bfc6-b98eab67753f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.55:15021/healthz/ready\": dial tcp 10.134.0.55:15021: connect: connection refused" Nov 04 02:50:19.156547 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:50:19.156198 4405 patch_prober.go:29] interesting pod/e2e-worker-w2-a0bcbc-openshift-default-65dc9d9b6c-j4n6p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.54:15021/healthz/ready\": dial tcp 10.134.0.54:15021: connect: connection refused" start-of-body= Nov 04 02:50:19.156547 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:50:19.156251 4405 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-a0bcbc-openshift-default-65dc9d9b6c-j4n6p" podUID="55473cc8-d292-4f5f-bb17-3f08c2e19e8c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.54:15021/healthz/ready\": dial tcp 10.134.0.54:15021: connect: connection refused" Nov 04 02:50:20.043689 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:50:20.043180 4405 patch_prober.go:29] interesting pod/e2e-worker-w1-e516ff-openshift-default-6f4b957485-7pg2l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.55:15021/healthz/ready\": dial tcp 10.134.0.55:15021: connect: connection refused" start-of-body= Nov 04 02:50:20.043689 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:50:20.043246 4405 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-e516ff-openshift-default-6f4b957485-7pg2l" podUID="8fc7b14e-09f9-4630-bfc6-b98eab67753f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.55:15021/healthz/ready\": dial tcp 10.134.0.55:15021: connect: connection refused" Nov 04 02:50:20.160693 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:50:20.159821 4405 patch_prober.go:29] interesting pod/e2e-worker-w2-a0bcbc-openshift-default-65dc9d9b6c-j4n6p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.54:15021/healthz/ready\": dial tcp 10.134.0.54:15021: connect: connection refused" start-of-body= Nov 04 02:50:20.160693 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:50:20.159894 4405 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-a0bcbc-openshift-default-65dc9d9b6c-j4n6p" podUID="55473cc8-d292-4f5f-bb17-3f08c2e19e8c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.54:15021/healthz/ready\": dial tcp 10.134.0.54:15021: connect: connection refused" Nov 04 02:50:21.048693 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:50:21.048106 4405 patch_prober.go:29] interesting pod/e2e-worker-w1-e516ff-openshift-default-6f4b957485-7pg2l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.55:15021/healthz/ready\": dial tcp 10.134.0.55:15021: connect: connection refused" start-of-body= Nov 04 02:50:21.048693 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:50:21.048174 4405 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-e516ff-openshift-default-6f4b957485-7pg2l" podUID="8fc7b14e-09f9-4630-bfc6-b98eab67753f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.55:15021/healthz/ready\": dial tcp 10.134.0.55:15021: connect: connection refused" Nov 04 02:50:21.156315 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:50:21.156249 4405 patch_prober.go:29] interesting pod/e2e-worker-w2-a0bcbc-openshift-default-65dc9d9b6c-j4n6p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.54:15021/healthz/ready\": dial tcp 10.134.0.54:15021: connect: connection refused" start-of-body= Nov 04 02:50:21.156539 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:50:21.156316 4405 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-a0bcbc-openshift-default-65dc9d9b6c-j4n6p" podUID="55473cc8-d292-4f5f-bb17-3f08c2e19e8c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.54:15021/healthz/ready\": dial tcp 10.134.0.54:15021: connect: connection refused" Nov 04 02:50:22.046288 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:50:22.046244 4405 patch_prober.go:29] interesting pod/e2e-worker-w1-e516ff-openshift-default-6f4b957485-7pg2l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.55:15021/healthz/ready\": dial tcp 10.134.0.55:15021: connect: connection refused" start-of-body= Nov 04 02:50:22.046623 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:50:22.046579 4405 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-e516ff-openshift-default-6f4b957485-7pg2l" podUID="8fc7b14e-09f9-4630-bfc6-b98eab67753f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.55:15021/healthz/ready\": dial tcp 10.134.0.55:15021: connect: connection refused" Nov 04 02:50:22.163872 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:50:22.163831 4405 patch_prober.go:29] interesting pod/e2e-worker-w2-a0bcbc-openshift-default-65dc9d9b6c-j4n6p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.54:15021/healthz/ready\": dial tcp 10.134.0.54:15021: connect: connection refused" start-of-body= Nov 04 02:50:22.164262 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:50:22.163897 4405 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-a0bcbc-openshift-default-65dc9d9b6c-j4n6p" podUID="55473cc8-d292-4f5f-bb17-3f08c2e19e8c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.54:15021/healthz/ready\": dial tcp 10.134.0.54:15021: connect: connection refused" Nov 04 02:50:23.046697 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:50:23.043934 4405 patch_prober.go:29] interesting pod/e2e-worker-w1-e516ff-openshift-default-6f4b957485-7pg2l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.55:15021/healthz/ready\": dial tcp 10.134.0.55:15021: connect: connection refused" start-of-body= Nov 04 02:50:23.046697 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:50:23.043998 4405 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-e516ff-openshift-default-6f4b957485-7pg2l" podUID="8fc7b14e-09f9-4630-bfc6-b98eab67753f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.55:15021/healthz/ready\": dial tcp 10.134.0.55:15021: connect: connection refused" Nov 04 02:50:23.156562 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:50:23.156060 4405 patch_prober.go:29] interesting pod/e2e-worker-w2-a0bcbc-openshift-default-65dc9d9b6c-j4n6p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.54:15021/healthz/ready\": dial tcp 10.134.0.54:15021: connect: connection refused" start-of-body= Nov 04 02:50:23.156562 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:50:23.156125 4405 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-a0bcbc-openshift-default-65dc9d9b6c-j4n6p" podUID="55473cc8-d292-4f5f-bb17-3f08c2e19e8c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.54:15021/healthz/ready\": dial tcp 10.134.0.54:15021: connect: connection refused" Nov 04 02:50:24.042740 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:50:24.042706 4405 patch_prober.go:29] interesting pod/e2e-worker-w1-e516ff-openshift-default-6f4b957485-7pg2l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.55:15021/healthz/ready\": dial tcp 10.134.0.55:15021: connect: connection refused" start-of-body= Nov 04 02:50:24.043233 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:50:24.043206 4405 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-e516ff-openshift-default-6f4b957485-7pg2l" podUID="8fc7b14e-09f9-4630-bfc6-b98eab67753f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.55:15021/healthz/ready\": dial tcp 10.134.0.55:15021: connect: connection refused" Nov 04 02:50:24.156451 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:50:24.156413 4405 patch_prober.go:29] interesting pod/e2e-worker-w2-a0bcbc-openshift-default-65dc9d9b6c-j4n6p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.54:15021/healthz/ready\": dial tcp 10.134.0.54:15021: connect: connection refused" start-of-body= Nov 04 02:50:24.156631 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:50:24.156482 4405 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-a0bcbc-openshift-default-65dc9d9b6c-j4n6p" podUID="55473cc8-d292-4f5f-bb17-3f08c2e19e8c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.54:15021/healthz/ready\": dial tcp 10.134.0.54:15021: connect: connection refused" Nov 04 02:50:25.042742 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:50:25.042616 4405 patch_prober.go:29] interesting pod/e2e-worker-w1-e516ff-openshift-default-6f4b957485-7pg2l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.55:15021/healthz/ready\": dial tcp 10.134.0.55:15021: connect: connection refused" start-of-body= Nov 04 02:50:25.042742 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:50:25.042728 4405 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-e516ff-openshift-default-6f4b957485-7pg2l" podUID="8fc7b14e-09f9-4630-bfc6-b98eab67753f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.55:15021/healthz/ready\": dial tcp 10.134.0.55:15021: connect: connection refused" Nov 04 02:50:25.255584 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:50:25.255536 4405 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-a0bcbc-openshift-default-65dc9d9b6c-j4n6p" Nov 04 02:50:25.256282 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:50:25.256253 4405 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-a0bcbc-openshift-default-65dc9d9b6c-j4n6p" Nov 04 02:50:25.257158 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:50:25.257136 4405 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-a0bcbc-openshift-default-65dc9d9b6c-j4n6p" Nov 04 02:50:25.490330 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:50:25.490289 4405 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-bl5mg"] Nov 04 02:50:26.046905 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:50:26.046873 4405 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-e516ff-openshift-default-6f4b957485-7pg2l" Nov 04 02:50:26.047351 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:50:26.047167 4405 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-e516ff-openshift-default-6f4b957485-7pg2l" Nov 04 02:50:26.048109 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:50:26.048092 4405 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-e516ff-openshift-default-6f4b957485-7pg2l" Nov 04 02:50:36.885106 ip-10-0-134-125 kubenswrapper[4405]: E1104 02:50:36.885057 4405 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5f2vb" containerName="extractor" Nov 04 02:50:47.331235 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:50:47.331201 4405 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-cdcf67-openshift-default-6ff8f5cbc7-975ss"] Nov 04 02:50:47.335272 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:50:47.335251 4405 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-cdcf67-openshift-default-6ff8f5cbc7-975ss" Nov 04 02:50:47.435802 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:50:47.435761 4405 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e5ccd15d-11fd-4f67-9bd5-7a4cfa35d602-workload-socket\") pod \"e2e-worker-w3-cdcf67-openshift-default-6ff8f5cbc7-975ss\" (UID: \"e5ccd15d-11fd-4f67-9bd5-7a4cfa35d602\") " pod="openshift-ingress/e2e-worker-w3-cdcf67-openshift-default-6ff8f5cbc7-975ss" Nov 04 02:50:47.436013 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:50:47.435839 4405 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e5ccd15d-11fd-4f67-9bd5-7a4cfa35d602-istio-data\") pod \"e2e-worker-w3-cdcf67-openshift-default-6ff8f5cbc7-975ss\" (UID: \"e5ccd15d-11fd-4f67-9bd5-7a4cfa35d602\") " pod="openshift-ingress/e2e-worker-w3-cdcf67-openshift-default-6ff8f5cbc7-975ss" Nov 04 02:50:47.436013 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:50:47.435963 4405 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e5ccd15d-11fd-4f67-9bd5-7a4cfa35d602-istio-envoy\") pod \"e2e-worker-w3-cdcf67-openshift-default-6ff8f5cbc7-975ss\" (UID: \"e5ccd15d-11fd-4f67-9bd5-7a4cfa35d602\") " pod="openshift-ingress/e2e-worker-w3-cdcf67-openshift-default-6ff8f5cbc7-975ss" Nov 04 02:50:47.436013 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:50:47.436008 4405 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e5ccd15d-11fd-4f67-9bd5-7a4cfa35d602-istio-token\") pod \"e2e-worker-w3-cdcf67-openshift-default-6ff8f5cbc7-975ss\" (UID: \"e5ccd15d-11fd-4f67-9bd5-7a4cfa35d602\") " pod="openshift-ingress/e2e-worker-w3-cdcf67-openshift-default-6ff8f5cbc7-975ss" Nov 04 02:50:47.436200 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:50:47.436063 4405 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e5ccd15d-11fd-4f67-9bd5-7a4cfa35d602-istio-podinfo\") pod \"e2e-worker-w3-cdcf67-openshift-default-6ff8f5cbc7-975ss\" (UID: \"e5ccd15d-11fd-4f67-9bd5-7a4cfa35d602\") " pod="openshift-ingress/e2e-worker-w3-cdcf67-openshift-default-6ff8f5cbc7-975ss" Nov 04 02:50:47.436263 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:50:47.436212 4405 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-msfwr\" (UniqueName: \"kubernetes.io/projected/e5ccd15d-11fd-4f67-9bd5-7a4cfa35d602-kube-api-access-msfwr\") pod \"e2e-worker-w3-cdcf67-openshift-default-6ff8f5cbc7-975ss\" (UID: \"e5ccd15d-11fd-4f67-9bd5-7a4cfa35d602\") " pod="openshift-ingress/e2e-worker-w3-cdcf67-openshift-default-6ff8f5cbc7-975ss" Nov 04 02:50:47.436319 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:50:47.436279 4405 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e5ccd15d-11fd-4f67-9bd5-7a4cfa35d602-istiod-ca-cert\") pod \"e2e-worker-w3-cdcf67-openshift-default-6ff8f5cbc7-975ss\" (UID: \"e5ccd15d-11fd-4f67-9bd5-7a4cfa35d602\") " pod="openshift-ingress/e2e-worker-w3-cdcf67-openshift-default-6ff8f5cbc7-975ss" Nov 04 02:50:47.436372 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:50:47.436325 4405 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e5ccd15d-11fd-4f67-9bd5-7a4cfa35d602-workload-certs\") pod \"e2e-worker-w3-cdcf67-openshift-default-6ff8f5cbc7-975ss\" (UID: \"e5ccd15d-11fd-4f67-9bd5-7a4cfa35d602\") " pod="openshift-ingress/e2e-worker-w3-cdcf67-openshift-default-6ff8f5cbc7-975ss" Nov 04 02:50:47.436372 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:50:47.436361 4405 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e5ccd15d-11fd-4f67-9bd5-7a4cfa35d602-credential-socket\") pod \"e2e-worker-w3-cdcf67-openshift-default-6ff8f5cbc7-975ss\" (UID: \"e5ccd15d-11fd-4f67-9bd5-7a4cfa35d602\") " pod="openshift-ingress/e2e-worker-w3-cdcf67-openshift-default-6ff8f5cbc7-975ss" Nov 04 02:50:47.438523 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:50:47.438497 4405 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-cdcf67-openshift-default-6ff8f5cbc7-975ss"] Nov 04 02:50:47.537535 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:50:47.537499 4405 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e5ccd15d-11fd-4f67-9bd5-7a4cfa35d602-istio-data\") pod \"e2e-worker-w3-cdcf67-openshift-default-6ff8f5cbc7-975ss\" (UID: \"e5ccd15d-11fd-4f67-9bd5-7a4cfa35d602\") " pod="openshift-ingress/e2e-worker-w3-cdcf67-openshift-default-6ff8f5cbc7-975ss" Nov 04 02:50:47.537726 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:50:47.537548 4405 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e5ccd15d-11fd-4f67-9bd5-7a4cfa35d602-istio-envoy\") pod \"e2e-worker-w3-cdcf67-openshift-default-6ff8f5cbc7-975ss\" (UID: \"e5ccd15d-11fd-4f67-9bd5-7a4cfa35d602\") " pod="openshift-ingress/e2e-worker-w3-cdcf67-openshift-default-6ff8f5cbc7-975ss" Nov 04 02:50:47.537726 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:50:47.537701 4405 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e5ccd15d-11fd-4f67-9bd5-7a4cfa35d602-istio-token\") pod \"e2e-worker-w3-cdcf67-openshift-default-6ff8f5cbc7-975ss\" (UID: \"e5ccd15d-11fd-4f67-9bd5-7a4cfa35d602\") " pod="openshift-ingress/e2e-worker-w3-cdcf67-openshift-default-6ff8f5cbc7-975ss" Nov 04 02:50:47.537874 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:50:47.537786 4405 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e5ccd15d-11fd-4f67-9bd5-7a4cfa35d602-istio-podinfo\") pod \"e2e-worker-w3-cdcf67-openshift-default-6ff8f5cbc7-975ss\" (UID: \"e5ccd15d-11fd-4f67-9bd5-7a4cfa35d602\") " pod="openshift-ingress/e2e-worker-w3-cdcf67-openshift-default-6ff8f5cbc7-975ss" Nov 04 02:50:47.537874 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:50:47.537850 4405 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-msfwr\" (UniqueName: \"kubernetes.io/projected/e5ccd15d-11fd-4f67-9bd5-7a4cfa35d602-kube-api-access-msfwr\") pod \"e2e-worker-w3-cdcf67-openshift-default-6ff8f5cbc7-975ss\" (UID: \"e5ccd15d-11fd-4f67-9bd5-7a4cfa35d602\") " pod="openshift-ingress/e2e-worker-w3-cdcf67-openshift-default-6ff8f5cbc7-975ss" Nov 04 02:50:47.537991 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:50:47.537909 4405 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e5ccd15d-11fd-4f67-9bd5-7a4cfa35d602-istiod-ca-cert\") pod \"e2e-worker-w3-cdcf67-openshift-default-6ff8f5cbc7-975ss\" (UID: \"e5ccd15d-11fd-4f67-9bd5-7a4cfa35d602\") " pod="openshift-ingress/e2e-worker-w3-cdcf67-openshift-default-6ff8f5cbc7-975ss" Nov 04 02:50:47.537991 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:50:47.537937 4405 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e5ccd15d-11fd-4f67-9bd5-7a4cfa35d602-istio-data\") pod \"e2e-worker-w3-cdcf67-openshift-default-6ff8f5cbc7-975ss\" (UID: \"e5ccd15d-11fd-4f67-9bd5-7a4cfa35d602\") " pod="openshift-ingress/e2e-worker-w3-cdcf67-openshift-default-6ff8f5cbc7-975ss" Nov 04 02:50:47.538141 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:50:47.538120 4405 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e5ccd15d-11fd-4f67-9bd5-7a4cfa35d602-workload-certs\") pod \"e2e-worker-w3-cdcf67-openshift-default-6ff8f5cbc7-975ss\" (UID: \"e5ccd15d-11fd-4f67-9bd5-7a4cfa35d602\") " pod="openshift-ingress/e2e-worker-w3-cdcf67-openshift-default-6ff8f5cbc7-975ss" Nov 04 02:50:47.538213 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:50:47.538168 4405 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e5ccd15d-11fd-4f67-9bd5-7a4cfa35d602-credential-socket\") pod \"e2e-worker-w3-cdcf67-openshift-default-6ff8f5cbc7-975ss\" (UID: \"e5ccd15d-11fd-4f67-9bd5-7a4cfa35d602\") " pod="openshift-ingress/e2e-worker-w3-cdcf67-openshift-default-6ff8f5cbc7-975ss" Nov 04 02:50:47.538292 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:50:47.538276 4405 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e5ccd15d-11fd-4f67-9bd5-7a4cfa35d602-workload-socket\") pod \"e2e-worker-w3-cdcf67-openshift-default-6ff8f5cbc7-975ss\" (UID: \"e5ccd15d-11fd-4f67-9bd5-7a4cfa35d602\") " pod="openshift-ingress/e2e-worker-w3-cdcf67-openshift-default-6ff8f5cbc7-975ss" Nov 04 02:50:47.538528 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:50:47.538501 4405 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e5ccd15d-11fd-4f67-9bd5-7a4cfa35d602-workload-certs\") pod \"e2e-worker-w3-cdcf67-openshift-default-6ff8f5cbc7-975ss\" (UID: \"e5ccd15d-11fd-4f67-9bd5-7a4cfa35d602\") " pod="openshift-ingress/e2e-worker-w3-cdcf67-openshift-default-6ff8f5cbc7-975ss" Nov 04 02:50:47.538631 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:50:47.538606 4405 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e5ccd15d-11fd-4f67-9bd5-7a4cfa35d602-istiod-ca-cert\") pod \"e2e-worker-w3-cdcf67-openshift-default-6ff8f5cbc7-975ss\" (UID: \"e5ccd15d-11fd-4f67-9bd5-7a4cfa35d602\") " pod="openshift-ingress/e2e-worker-w3-cdcf67-openshift-default-6ff8f5cbc7-975ss" Nov 04 02:50:47.538772 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:50:47.538753 4405 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e5ccd15d-11fd-4f67-9bd5-7a4cfa35d602-credential-socket\") pod \"e2e-worker-w3-cdcf67-openshift-default-6ff8f5cbc7-975ss\" (UID: \"e5ccd15d-11fd-4f67-9bd5-7a4cfa35d602\") " pod="openshift-ingress/e2e-worker-w3-cdcf67-openshift-default-6ff8f5cbc7-975ss" Nov 04 02:50:47.538853 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:50:47.538832 4405 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e5ccd15d-11fd-4f67-9bd5-7a4cfa35d602-workload-socket\") pod \"e2e-worker-w3-cdcf67-openshift-default-6ff8f5cbc7-975ss\" (UID: \"e5ccd15d-11fd-4f67-9bd5-7a4cfa35d602\") " pod="openshift-ingress/e2e-worker-w3-cdcf67-openshift-default-6ff8f5cbc7-975ss" Nov 04 02:50:47.540010 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:50:47.539985 4405 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e5ccd15d-11fd-4f67-9bd5-7a4cfa35d602-istio-envoy\") pod \"e2e-worker-w3-cdcf67-openshift-default-6ff8f5cbc7-975ss\" (UID: \"e5ccd15d-11fd-4f67-9bd5-7a4cfa35d602\") " pod="openshift-ingress/e2e-worker-w3-cdcf67-openshift-default-6ff8f5cbc7-975ss" Nov 04 02:50:47.540339 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:50:47.540321 4405 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e5ccd15d-11fd-4f67-9bd5-7a4cfa35d602-istio-podinfo\") pod \"e2e-worker-w3-cdcf67-openshift-default-6ff8f5cbc7-975ss\" (UID: \"e5ccd15d-11fd-4f67-9bd5-7a4cfa35d602\") " pod="openshift-ingress/e2e-worker-w3-cdcf67-openshift-default-6ff8f5cbc7-975ss" Nov 04 02:50:47.599264 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:50:47.599222 4405 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e5ccd15d-11fd-4f67-9bd5-7a4cfa35d602-istio-token\") pod \"e2e-worker-w3-cdcf67-openshift-default-6ff8f5cbc7-975ss\" (UID: \"e5ccd15d-11fd-4f67-9bd5-7a4cfa35d602\") " pod="openshift-ingress/e2e-worker-w3-cdcf67-openshift-default-6ff8f5cbc7-975ss" Nov 04 02:50:47.599416 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:50:47.599337 4405 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-msfwr\" (UniqueName: \"kubernetes.io/projected/e5ccd15d-11fd-4f67-9bd5-7a4cfa35d602-kube-api-access-msfwr\") pod \"e2e-worker-w3-cdcf67-openshift-default-6ff8f5cbc7-975ss\" (UID: \"e5ccd15d-11fd-4f67-9bd5-7a4cfa35d602\") " pod="openshift-ingress/e2e-worker-w3-cdcf67-openshift-default-6ff8f5cbc7-975ss" Nov 04 02:50:47.608255 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:50:47.608229 4405 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w6-ebb7fd/distinct-w6-ebb7fd-kserve-55c47d8865-7z9pz"] Nov 04 02:50:47.613735 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:50:47.613712 4405 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-ebb7fd/distinct-w6-ebb7fd-kserve-55c47d8865-7z9pz" Nov 04 02:50:47.639511 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:50:47.639483 4405 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ccb8e181-5610-46da-bb8b-e0b07fa02ff1-tmp-dir\") pod \"distinct-w6-ebb7fd-kserve-55c47d8865-7z9pz\" (UID: \"ccb8e181-5610-46da-bb8b-e0b07fa02ff1\") " pod="e2e-models-e2e-worker-w6-ebb7fd/distinct-w6-ebb7fd-kserve-55c47d8865-7z9pz" Nov 04 02:50:47.639700 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:50:47.639533 4405 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ccb8e181-5610-46da-bb8b-e0b07fa02ff1-tls-certs\") pod \"distinct-w6-ebb7fd-kserve-55c47d8865-7z9pz\" (UID: \"ccb8e181-5610-46da-bb8b-e0b07fa02ff1\") " pod="e2e-models-e2e-worker-w6-ebb7fd/distinct-w6-ebb7fd-kserve-55c47d8865-7z9pz" Nov 04 02:50:47.639700 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:50:47.639662 4405 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ccb8e181-5610-46da-bb8b-e0b07fa02ff1-home\") pod \"distinct-w6-ebb7fd-kserve-55c47d8865-7z9pz\" (UID: \"ccb8e181-5610-46da-bb8b-e0b07fa02ff1\") " pod="e2e-models-e2e-worker-w6-ebb7fd/distinct-w6-ebb7fd-kserve-55c47d8865-7z9pz" Nov 04 02:50:47.639949 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:50:47.639856 4405 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ccb8e181-5610-46da-bb8b-e0b07fa02ff1-model-cache\") pod \"distinct-w6-ebb7fd-kserve-55c47d8865-7z9pz\" (UID: \"ccb8e181-5610-46da-bb8b-e0b07fa02ff1\") " pod="e2e-models-e2e-worker-w6-ebb7fd/distinct-w6-ebb7fd-kserve-55c47d8865-7z9pz" Nov 04 02:50:47.640026 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:50:47.639958 4405 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ccb8e181-5610-46da-bb8b-e0b07fa02ff1-dshm\") pod \"distinct-w6-ebb7fd-kserve-55c47d8865-7z9pz\" (UID: \"ccb8e181-5610-46da-bb8b-e0b07fa02ff1\") " pod="e2e-models-e2e-worker-w6-ebb7fd/distinct-w6-ebb7fd-kserve-55c47d8865-7z9pz" Nov 04 02:50:47.640026 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:50:47.639992 4405 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-shknm\" (UniqueName: \"kubernetes.io/projected/ccb8e181-5610-46da-bb8b-e0b07fa02ff1-kube-api-access-shknm\") pod \"distinct-w6-ebb7fd-kserve-55c47d8865-7z9pz\" (UID: \"ccb8e181-5610-46da-bb8b-e0b07fa02ff1\") " pod="e2e-models-e2e-worker-w6-ebb7fd/distinct-w6-ebb7fd-kserve-55c47d8865-7z9pz" Nov 04 02:50:47.649188 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:50:47.649162 4405 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-cdcf67-openshift-default-6ff8f5cbc7-975ss" Nov 04 02:50:47.671122 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:50:47.671089 4405 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w6-ebb7fd\"/\"distinct-w6-ebb7fd-kserve-self-signed-certs\"" Nov 04 02:50:47.671485 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:50:47.671470 4405 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w6-ebb7fd\"/\"default-dockercfg-w24cn\"" Nov 04 02:50:47.671740 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:50:47.671661 4405 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w6-ebb7fd\"/\"openshift-service-ca.crt\"" Nov 04 02:50:47.672063 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:50:47.672045 4405 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w6-ebb7fd\"/\"kube-root-ca.crt\"" Nov 04 02:50:47.678038 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:50:47.678012 4405 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w6-ebb7fd/distinct-w6-ebb7fd-kserve-55c47d8865-7z9pz"] Nov 04 02:50:47.741384 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:50:47.741353 4405 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ccb8e181-5610-46da-bb8b-e0b07fa02ff1-dshm\") pod \"distinct-w6-ebb7fd-kserve-55c47d8865-7z9pz\" (UID: \"ccb8e181-5610-46da-bb8b-e0b07fa02ff1\") " pod="e2e-models-e2e-worker-w6-ebb7fd/distinct-w6-ebb7fd-kserve-55c47d8865-7z9pz" Nov 04 02:50:47.741537 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:50:47.741399 4405 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-shknm\" (UniqueName: \"kubernetes.io/projected/ccb8e181-5610-46da-bb8b-e0b07fa02ff1-kube-api-access-shknm\") pod \"distinct-w6-ebb7fd-kserve-55c47d8865-7z9pz\" (UID: \"ccb8e181-5610-46da-bb8b-e0b07fa02ff1\") " pod="e2e-models-e2e-worker-w6-ebb7fd/distinct-w6-ebb7fd-kserve-55c47d8865-7z9pz" Nov 04 02:50:47.741537 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:50:47.741432 4405 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ccb8e181-5610-46da-bb8b-e0b07fa02ff1-tmp-dir\") pod \"distinct-w6-ebb7fd-kserve-55c47d8865-7z9pz\" (UID: \"ccb8e181-5610-46da-bb8b-e0b07fa02ff1\") " pod="e2e-models-e2e-worker-w6-ebb7fd/distinct-w6-ebb7fd-kserve-55c47d8865-7z9pz" Nov 04 02:50:47.741537 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:50:47.741453 4405 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ccb8e181-5610-46da-bb8b-e0b07fa02ff1-tls-certs\") pod \"distinct-w6-ebb7fd-kserve-55c47d8865-7z9pz\" (UID: \"ccb8e181-5610-46da-bb8b-e0b07fa02ff1\") " pod="e2e-models-e2e-worker-w6-ebb7fd/distinct-w6-ebb7fd-kserve-55c47d8865-7z9pz" Nov 04 02:50:47.741537 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:50:47.741482 4405 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ccb8e181-5610-46da-bb8b-e0b07fa02ff1-home\") pod \"distinct-w6-ebb7fd-kserve-55c47d8865-7z9pz\" (UID: \"ccb8e181-5610-46da-bb8b-e0b07fa02ff1\") " pod="e2e-models-e2e-worker-w6-ebb7fd/distinct-w6-ebb7fd-kserve-55c47d8865-7z9pz" Nov 04 02:50:47.741809 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:50:47.741559 4405 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ccb8e181-5610-46da-bb8b-e0b07fa02ff1-model-cache\") pod \"distinct-w6-ebb7fd-kserve-55c47d8865-7z9pz\" (UID: \"ccb8e181-5610-46da-bb8b-e0b07fa02ff1\") " pod="e2e-models-e2e-worker-w6-ebb7fd/distinct-w6-ebb7fd-kserve-55c47d8865-7z9pz" Nov 04 02:50:47.745931 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:50:47.742210 4405 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ccb8e181-5610-46da-bb8b-e0b07fa02ff1-model-cache\") pod \"distinct-w6-ebb7fd-kserve-55c47d8865-7z9pz\" (UID: \"ccb8e181-5610-46da-bb8b-e0b07fa02ff1\") " pod="e2e-models-e2e-worker-w6-ebb7fd/distinct-w6-ebb7fd-kserve-55c47d8865-7z9pz" Nov 04 02:50:47.745931 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:50:47.742485 4405 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ccb8e181-5610-46da-bb8b-e0b07fa02ff1-home\") pod \"distinct-w6-ebb7fd-kserve-55c47d8865-7z9pz\" (UID: \"ccb8e181-5610-46da-bb8b-e0b07fa02ff1\") " pod="e2e-models-e2e-worker-w6-ebb7fd/distinct-w6-ebb7fd-kserve-55c47d8865-7z9pz" Nov 04 02:50:47.745931 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:50:47.745865 4405 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ccb8e181-5610-46da-bb8b-e0b07fa02ff1-tmp-dir\") pod \"distinct-w6-ebb7fd-kserve-55c47d8865-7z9pz\" (UID: \"ccb8e181-5610-46da-bb8b-e0b07fa02ff1\") " pod="e2e-models-e2e-worker-w6-ebb7fd/distinct-w6-ebb7fd-kserve-55c47d8865-7z9pz" Nov 04 02:50:47.751394 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:50:47.750443 4405 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ccb8e181-5610-46da-bb8b-e0b07fa02ff1-dshm\") pod \"distinct-w6-ebb7fd-kserve-55c47d8865-7z9pz\" (UID: \"ccb8e181-5610-46da-bb8b-e0b07fa02ff1\") " pod="e2e-models-e2e-worker-w6-ebb7fd/distinct-w6-ebb7fd-kserve-55c47d8865-7z9pz" Nov 04 02:50:47.760889 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:50:47.760860 4405 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ccb8e181-5610-46da-bb8b-e0b07fa02ff1-tls-certs\") pod \"distinct-w6-ebb7fd-kserve-55c47d8865-7z9pz\" (UID: \"ccb8e181-5610-46da-bb8b-e0b07fa02ff1\") " pod="e2e-models-e2e-worker-w6-ebb7fd/distinct-w6-ebb7fd-kserve-55c47d8865-7z9pz" Nov 04 02:50:47.807187 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:50:47.807145 4405 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-shknm\" (UniqueName: \"kubernetes.io/projected/ccb8e181-5610-46da-bb8b-e0b07fa02ff1-kube-api-access-shknm\") pod \"distinct-w6-ebb7fd-kserve-55c47d8865-7z9pz\" (UID: \"ccb8e181-5610-46da-bb8b-e0b07fa02ff1\") " pod="e2e-models-e2e-worker-w6-ebb7fd/distinct-w6-ebb7fd-kserve-55c47d8865-7z9pz" Nov 04 02:50:47.928110 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:50:47.928030 4405 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-ebb7fd/distinct-w6-ebb7fd-kserve-55c47d8865-7z9pz" Nov 04 02:50:48.092081 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:50:48.092056 4405 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-cdcf67-openshift-default-6ff8f5cbc7-975ss"] Nov 04 02:50:48.094307 ip-10-0-134-125 kubenswrapper[4405]: W1104 02:50:48.094278 4405 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode5ccd15d_11fd_4f67_9bd5_7a4cfa35d602.slice/crio-4bc10598f608376d1ab4f226dce8a2263f36f4ac6e35a53faeb4630d720c9a37 WatchSource:0}: Error finding container 4bc10598f608376d1ab4f226dce8a2263f36f4ac6e35a53faeb4630d720c9a37: Status 404 returned error can't find the container with id 4bc10598f608376d1ab4f226dce8a2263f36f4ac6e35a53faeb4630d720c9a37 Nov 04 02:50:48.097125 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:50:48.097086 4405 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 04 02:50:48.097250 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:50:48.097179 4405 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 04 02:50:48.097250 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:50:48.097226 4405 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 04 02:50:48.339909 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:50:48.339872 4405 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w6-ebb7fd/distinct-w6-ebb7fd-kserve-55c47d8865-7z9pz"] Nov 04 02:50:48.340556 ip-10-0-134-125 kubenswrapper[4405]: W1104 02:50:48.340527 4405 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podccb8e181_5610_46da_bb8b_e0b07fa02ff1.slice/crio-61710beb52c66594a12b82ff262456f27fd09a6e2fa6ada0356aea5c0d572844 WatchSource:0}: Error finding container 61710beb52c66594a12b82ff262456f27fd09a6e2fa6ada0356aea5c0d572844: Status 404 returned error can't find the container with id 61710beb52c66594a12b82ff262456f27fd09a6e2fa6ada0356aea5c0d572844 Nov 04 02:50:49.029839 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:50:49.029801 4405 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-ebb7fd/distinct-w6-ebb7fd-kserve-55c47d8865-7z9pz" event={"ID":"ccb8e181-5610-46da-bb8b-e0b07fa02ff1","Type":"ContainerStarted","Data":"33f579eb8d27949bb4ba12ebe84d9ec19d482af4a3509bf9dffe01fb714f22d3"} Nov 04 02:50:49.029839 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:50:49.029850 4405 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-ebb7fd/distinct-w6-ebb7fd-kserve-55c47d8865-7z9pz" event={"ID":"ccb8e181-5610-46da-bb8b-e0b07fa02ff1","Type":"ContainerStarted","Data":"61710beb52c66594a12b82ff262456f27fd09a6e2fa6ada0356aea5c0d572844"} Nov 04 02:50:49.031635 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:50:49.031608 4405 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-cdcf67-openshift-default-6ff8f5cbc7-975ss" event={"ID":"e5ccd15d-11fd-4f67-9bd5-7a4cfa35d602","Type":"ContainerStarted","Data":"795acf882ba76a67847bc16da20bf2fbd18afd9b410a73878a9e0ea63211deb3"} Nov 04 02:50:49.031803 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:50:49.031652 4405 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-cdcf67-openshift-default-6ff8f5cbc7-975ss" event={"ID":"e5ccd15d-11fd-4f67-9bd5-7a4cfa35d602","Type":"ContainerStarted","Data":"4bc10598f608376d1ab4f226dce8a2263f36f4ac6e35a53faeb4630d720c9a37"} Nov 04 02:50:49.149370 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:50:49.149309 4405 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w6-ebb7fd/distinct-w6-ebb7fd-kserve-55c47d8865-7z9pz" podStartSLOduration=2.149294309 podStartE2EDuration="2.149294309s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 02:50:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 02:50:49.146846422 +0000 UTC m=+1095.577546744" watchObservedRunningTime="2026-11-04 02:50:49.149294309 +0000 UTC m=+1095.579994428" Nov 04 02:50:49.273136 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:50:49.273080 4405 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-cdcf67-openshift-default-6ff8f5cbc7-975ss" podStartSLOduration=2.273063981 podStartE2EDuration="2.273063981s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 02:50:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 02:50:49.267881798 +0000 UTC m=+1095.698581918" watchObservedRunningTime="2026-11-04 02:50:49.273063981 +0000 UTC m=+1095.703764101" Nov 04 02:50:49.649544 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:50:49.649505 4405 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-cdcf67-openshift-default-6ff8f5cbc7-975ss" Nov 04 02:50:49.651247 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:50:49.651220 4405 patch_prober.go:29] interesting pod/e2e-worker-w3-cdcf67-openshift-default-6ff8f5cbc7-975ss container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused" start-of-body= Nov 04 02:50:49.651390 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:50:49.651277 4405 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-cdcf67-openshift-default-6ff8f5cbc7-975ss" podUID="e5ccd15d-11fd-4f67-9bd5-7a4cfa35d602" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused" Nov 04 02:50:49.929274 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:50:49.929186 4405 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-ebb7fd/distinct-w6-ebb7fd-kserve-55c47d8865-7z9pz" Nov 04 02:50:50.649644 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:50:50.649608 4405 patch_prober.go:29] interesting pod/e2e-worker-w3-cdcf67-openshift-default-6ff8f5cbc7-975ss container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused" start-of-body= Nov 04 02:50:50.650073 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:50:50.649686 4405 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-cdcf67-openshift-default-6ff8f5cbc7-975ss" podUID="e5ccd15d-11fd-4f67-9bd5-7a4cfa35d602" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused" Nov 04 02:50:51.650390 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:50:51.650352 4405 patch_prober.go:29] interesting pod/e2e-worker-w3-cdcf67-openshift-default-6ff8f5cbc7-975ss container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused" start-of-body= Nov 04 02:50:51.650947 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:50:51.650417 4405 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-cdcf67-openshift-default-6ff8f5cbc7-975ss" podUID="e5ccd15d-11fd-4f67-9bd5-7a4cfa35d602" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused" Nov 04 02:50:52.649659 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:50:52.649622 4405 patch_prober.go:29] interesting pod/e2e-worker-w3-cdcf67-openshift-default-6ff8f5cbc7-975ss container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused" start-of-body= Nov 04 02:50:52.649897 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:50:52.649701 4405 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-cdcf67-openshift-default-6ff8f5cbc7-975ss" podUID="e5ccd15d-11fd-4f67-9bd5-7a4cfa35d602" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused" Nov 04 02:50:53.649756 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:50:53.649726 4405 patch_prober.go:29] interesting pod/e2e-worker-w3-cdcf67-openshift-default-6ff8f5cbc7-975ss container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused" start-of-body= Nov 04 02:50:53.650229 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:50:53.649779 4405 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-cdcf67-openshift-default-6ff8f5cbc7-975ss" podUID="e5ccd15d-11fd-4f67-9bd5-7a4cfa35d602" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused" Nov 04 02:50:54.650038 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:50:54.649998 4405 patch_prober.go:29] interesting pod/e2e-worker-w3-cdcf67-openshift-default-6ff8f5cbc7-975ss container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused" start-of-body= Nov 04 02:50:54.650457 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:50:54.650058 4405 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-cdcf67-openshift-default-6ff8f5cbc7-975ss" podUID="e5ccd15d-11fd-4f67-9bd5-7a4cfa35d602" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused" Nov 04 02:50:55.654599 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:50:55.654565 4405 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-cdcf67-openshift-default-6ff8f5cbc7-975ss" Nov 04 02:50:55.655175 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:50:55.655039 4405 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-cdcf67-openshift-default-6ff8f5cbc7-975ss" Nov 04 02:50:55.656095 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:50:55.656073 4405 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-cdcf67-openshift-default-6ff8f5cbc7-975ss" Nov 04 02:50:57.912265 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:50:57.912231 4405 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-bl5mg"] Nov 04 02:50:57.929112 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:50:57.929089 4405 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w6-ebb7fd/distinct-w6-ebb7fd-kserve-55c47d8865-7z9pz" Nov 04 02:50:57.942246 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:50:57.942223 4405 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w6-ebb7fd/distinct-w6-ebb7fd-kserve-55c47d8865-7z9pz" Nov 04 02:50:58.096112 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:50:58.096080 4405 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-ebb7fd/distinct-w6-ebb7fd-kserve-55c47d8865-7z9pz" Nov 04 02:51:05.011449 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:51:05.011408 4405 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w1-e516ff/distinct-w1-e516ff-kserve-f4474cb9c-glpzw"] Nov 04 02:51:05.016498 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:51:05.016474 4405 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-e516ff/distinct-w1-e516ff-kserve-f4474cb9c-glpzw" Nov 04 02:51:05.084029 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:51:05.083991 4405 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hwc48\" (UniqueName: \"kubernetes.io/projected/8842ad09-22ff-4492-ba5d-d4cacf4eac75-kube-api-access-hwc48\") pod \"distinct-w1-e516ff-kserve-f4474cb9c-glpzw\" (UID: \"8842ad09-22ff-4492-ba5d-d4cacf4eac75\") " pod="e2e-models-e2e-worker-w1-e516ff/distinct-w1-e516ff-kserve-f4474cb9c-glpzw" Nov 04 02:51:05.084221 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:51:05.084198 4405 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8842ad09-22ff-4492-ba5d-d4cacf4eac75-dshm\") pod \"distinct-w1-e516ff-kserve-f4474cb9c-glpzw\" (UID: \"8842ad09-22ff-4492-ba5d-d4cacf4eac75\") " pod="e2e-models-e2e-worker-w1-e516ff/distinct-w1-e516ff-kserve-f4474cb9c-glpzw" Nov 04 02:51:05.084295 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:51:05.084246 4405 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8842ad09-22ff-4492-ba5d-d4cacf4eac75-home\") pod \"distinct-w1-e516ff-kserve-f4474cb9c-glpzw\" (UID: \"8842ad09-22ff-4492-ba5d-d4cacf4eac75\") " pod="e2e-models-e2e-worker-w1-e516ff/distinct-w1-e516ff-kserve-f4474cb9c-glpzw" Nov 04 02:51:05.084402 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:51:05.084380 4405 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8842ad09-22ff-4492-ba5d-d4cacf4eac75-model-cache\") pod \"distinct-w1-e516ff-kserve-f4474cb9c-glpzw\" (UID: \"8842ad09-22ff-4492-ba5d-d4cacf4eac75\") " pod="e2e-models-e2e-worker-w1-e516ff/distinct-w1-e516ff-kserve-f4474cb9c-glpzw" Nov 04 02:51:05.084474 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:51:05.084420 4405 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8842ad09-22ff-4492-ba5d-d4cacf4eac75-tmp-dir\") pod \"distinct-w1-e516ff-kserve-f4474cb9c-glpzw\" (UID: \"8842ad09-22ff-4492-ba5d-d4cacf4eac75\") " pod="e2e-models-e2e-worker-w1-e516ff/distinct-w1-e516ff-kserve-f4474cb9c-glpzw" Nov 04 02:51:05.084474 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:51:05.084461 4405 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8842ad09-22ff-4492-ba5d-d4cacf4eac75-tls-certs\") pod \"distinct-w1-e516ff-kserve-f4474cb9c-glpzw\" (UID: \"8842ad09-22ff-4492-ba5d-d4cacf4eac75\") " pod="e2e-models-e2e-worker-w1-e516ff/distinct-w1-e516ff-kserve-f4474cb9c-glpzw" Nov 04 02:51:05.119499 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:51:05.119462 4405 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w1-e516ff\"/\"kube-root-ca.crt\"" Nov 04 02:51:05.119702 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:51:05.119658 4405 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w1-e516ff\"/\"openshift-service-ca.crt\"" Nov 04 02:51:05.119876 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:51:05.119792 4405 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w1-e516ff\"/\"default-dockercfg-9d9b9\"" Nov 04 02:51:05.119876 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:51:05.119838 4405 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w1-e516ff\"/\"distinct-w1-e516ff-kserve-self-signed-certs\"" Nov 04 02:51:05.123863 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:51:05.123842 4405 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-e516ff/distinct-w1-e516ff-kserve-f4474cb9c-glpzw"] Nov 04 02:51:05.187970 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:51:05.187938 4405 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8842ad09-22ff-4492-ba5d-d4cacf4eac75-dshm\") pod \"distinct-w1-e516ff-kserve-f4474cb9c-glpzw\" (UID: \"8842ad09-22ff-4492-ba5d-d4cacf4eac75\") " pod="e2e-models-e2e-worker-w1-e516ff/distinct-w1-e516ff-kserve-f4474cb9c-glpzw" Nov 04 02:51:05.187970 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:51:05.187987 4405 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8842ad09-22ff-4492-ba5d-d4cacf4eac75-home\") pod \"distinct-w1-e516ff-kserve-f4474cb9c-glpzw\" (UID: \"8842ad09-22ff-4492-ba5d-d4cacf4eac75\") " pod="e2e-models-e2e-worker-w1-e516ff/distinct-w1-e516ff-kserve-f4474cb9c-glpzw" Nov 04 02:51:05.188256 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:51:05.188012 4405 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8842ad09-22ff-4492-ba5d-d4cacf4eac75-model-cache\") pod \"distinct-w1-e516ff-kserve-f4474cb9c-glpzw\" (UID: \"8842ad09-22ff-4492-ba5d-d4cacf4eac75\") " pod="e2e-models-e2e-worker-w1-e516ff/distinct-w1-e516ff-kserve-f4474cb9c-glpzw" Nov 04 02:51:05.188256 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:51:05.188069 4405 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8842ad09-22ff-4492-ba5d-d4cacf4eac75-tmp-dir\") pod \"distinct-w1-e516ff-kserve-f4474cb9c-glpzw\" (UID: \"8842ad09-22ff-4492-ba5d-d4cacf4eac75\") " pod="e2e-models-e2e-worker-w1-e516ff/distinct-w1-e516ff-kserve-f4474cb9c-glpzw" Nov 04 02:51:05.188256 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:51:05.188132 4405 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8842ad09-22ff-4492-ba5d-d4cacf4eac75-tls-certs\") pod \"distinct-w1-e516ff-kserve-f4474cb9c-glpzw\" (UID: \"8842ad09-22ff-4492-ba5d-d4cacf4eac75\") " pod="e2e-models-e2e-worker-w1-e516ff/distinct-w1-e516ff-kserve-f4474cb9c-glpzw" Nov 04 02:51:05.188362 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:51:05.188294 4405 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hwc48\" (UniqueName: \"kubernetes.io/projected/8842ad09-22ff-4492-ba5d-d4cacf4eac75-kube-api-access-hwc48\") pod \"distinct-w1-e516ff-kserve-f4474cb9c-glpzw\" (UID: \"8842ad09-22ff-4492-ba5d-d4cacf4eac75\") " pod="e2e-models-e2e-worker-w1-e516ff/distinct-w1-e516ff-kserve-f4474cb9c-glpzw" Nov 04 02:51:05.189538 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:51:05.189507 4405 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8842ad09-22ff-4492-ba5d-d4cacf4eac75-home\") pod \"distinct-w1-e516ff-kserve-f4474cb9c-glpzw\" (UID: \"8842ad09-22ff-4492-ba5d-d4cacf4eac75\") " pod="e2e-models-e2e-worker-w1-e516ff/distinct-w1-e516ff-kserve-f4474cb9c-glpzw" Nov 04 02:51:05.189710 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:51:05.189553 4405 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8842ad09-22ff-4492-ba5d-d4cacf4eac75-model-cache\") pod \"distinct-w1-e516ff-kserve-f4474cb9c-glpzw\" (UID: \"8842ad09-22ff-4492-ba5d-d4cacf4eac75\") " pod="e2e-models-e2e-worker-w1-e516ff/distinct-w1-e516ff-kserve-f4474cb9c-glpzw" Nov 04 02:51:05.189710 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:51:05.189593 4405 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8842ad09-22ff-4492-ba5d-d4cacf4eac75-tmp-dir\") pod \"distinct-w1-e516ff-kserve-f4474cb9c-glpzw\" (UID: \"8842ad09-22ff-4492-ba5d-d4cacf4eac75\") " pod="e2e-models-e2e-worker-w1-e516ff/distinct-w1-e516ff-kserve-f4474cb9c-glpzw" Nov 04 02:51:05.191736 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:51:05.191710 4405 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8842ad09-22ff-4492-ba5d-d4cacf4eac75-dshm\") pod \"distinct-w1-e516ff-kserve-f4474cb9c-glpzw\" (UID: \"8842ad09-22ff-4492-ba5d-d4cacf4eac75\") " pod="e2e-models-e2e-worker-w1-e516ff/distinct-w1-e516ff-kserve-f4474cb9c-glpzw" Nov 04 02:51:05.192283 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:51:05.192233 4405 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8842ad09-22ff-4492-ba5d-d4cacf4eac75-tls-certs\") pod \"distinct-w1-e516ff-kserve-f4474cb9c-glpzw\" (UID: \"8842ad09-22ff-4492-ba5d-d4cacf4eac75\") " pod="e2e-models-e2e-worker-w1-e516ff/distinct-w1-e516ff-kserve-f4474cb9c-glpzw" Nov 04 02:51:05.249873 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:51:05.249841 4405 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hwc48\" (UniqueName: \"kubernetes.io/projected/8842ad09-22ff-4492-ba5d-d4cacf4eac75-kube-api-access-hwc48\") pod \"distinct-w1-e516ff-kserve-f4474cb9c-glpzw\" (UID: \"8842ad09-22ff-4492-ba5d-d4cacf4eac75\") " pod="e2e-models-e2e-worker-w1-e516ff/distinct-w1-e516ff-kserve-f4474cb9c-glpzw" Nov 04 02:51:05.329330 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:51:05.329242 4405 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-e516ff/distinct-w1-e516ff-kserve-f4474cb9c-glpzw" Nov 04 02:51:05.707871 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:51:05.707850 4405 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-e516ff/distinct-w1-e516ff-kserve-f4474cb9c-glpzw"] Nov 04 02:51:05.709501 ip-10-0-134-125 kubenswrapper[4405]: W1104 02:51:05.709472 4405 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod8842ad09_22ff_4492_ba5d_d4cacf4eac75.slice/crio-54d72f22a7ddba3dcd0cd9418c8b969d1256e9da25486b1a06437413a075c6a4 WatchSource:0}: Error finding container 54d72f22a7ddba3dcd0cd9418c8b969d1256e9da25486b1a06437413a075c6a4: Status 404 returned error can't find the container with id 54d72f22a7ddba3dcd0cd9418c8b969d1256e9da25486b1a06437413a075c6a4 Nov 04 02:51:06.131907 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:51:06.131866 4405 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-e516ff/distinct-w1-e516ff-kserve-f4474cb9c-glpzw" event={"ID":"8842ad09-22ff-4492-ba5d-d4cacf4eac75","Type":"ContainerStarted","Data":"8466c7b2a5aee2d8317eed1fead0e2d0f8a7976eb7f7a3bb8242a77ff6d01666"} Nov 04 02:51:06.131907 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:51:06.131920 4405 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-e516ff/distinct-w1-e516ff-kserve-f4474cb9c-glpzw" event={"ID":"8842ad09-22ff-4492-ba5d-d4cacf4eac75","Type":"ContainerStarted","Data":"54d72f22a7ddba3dcd0cd9418c8b969d1256e9da25486b1a06437413a075c6a4"} Nov 04 02:51:06.250654 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:51:06.250596 4405 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w1-e516ff/distinct-w1-e516ff-kserve-f4474cb9c-glpzw" podStartSLOduration=2.250581816 podStartE2EDuration="2.250581816s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 02:51:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 02:51:06.247944408 +0000 UTC m=+1112.678644528" watchObservedRunningTime="2026-11-04 02:51:06.250581816 +0000 UTC m=+1112.681281940" Nov 04 02:51:06.330007 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:51:06.329959 4405 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-e516ff/distinct-w1-e516ff-kserve-f4474cb9c-glpzw" Nov 04 02:51:06.890871 ip-10-0-134-125 kubenswrapper[4405]: E1104 02:51:06.890831 4405 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5f2vb" containerName="extractor" Nov 04 02:51:15.346092 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:51:15.345880 4405 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w1-e516ff/distinct-w1-e516ff-kserve-f4474cb9c-glpzw" Nov 04 02:51:15.381992 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:51:15.381962 4405 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w1-e516ff/distinct-w1-e516ff-kserve-f4474cb9c-glpzw" Nov 04 02:51:16.199388 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:51:16.199363 4405 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-e516ff/distinct-w1-e516ff-kserve-f4474cb9c-glpzw" Nov 04 02:51:18.966097 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:51:18.966063 4405 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-76597d6855-dhvd2"] Nov 04 02:51:18.966611 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:51:18.966262 4405 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-dhvd2" podUID="ad17f139-f05a-40fe-8750-b3b5938bfacc" containerName="istio-proxy" containerID="cri-o://abab077e0b3bacb6976169fa82208abeba6c11efc953d43029100ba49ba15ee4" gracePeriod=30 Nov 04 02:51:24.237257 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:51:24.237221 4405 generic.go:356] "Generic (PLEG): container finished" podID="ad17f139-f05a-40fe-8750-b3b5938bfacc" containerID="abab077e0b3bacb6976169fa82208abeba6c11efc953d43029100ba49ba15ee4" exitCode=0 Nov 04 02:51:24.237593 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:51:24.237292 4405 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-dhvd2" event={"ID":"ad17f139-f05a-40fe-8750-b3b5938bfacc","Type":"ContainerDied","Data":"abab077e0b3bacb6976169fa82208abeba6c11efc953d43029100ba49ba15ee4"} Nov 04 02:51:24.237593 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:51:24.237347 4405 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-dhvd2" event={"ID":"ad17f139-f05a-40fe-8750-b3b5938bfacc","Type":"ContainerDied","Data":"a21bf6cf4f256b1c4872b3602aae804f7ebcb2989144a76b188281d769772d31"} Nov 04 02:51:24.237593 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:51:24.237364 4405 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a21bf6cf4f256b1c4872b3602aae804f7ebcb2989144a76b188281d769772d31" Nov 04 02:51:24.246202 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:51:24.246176 4405 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-dhvd2" Nov 04 02:51:24.347682 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:51:24.347646 4405 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ad17f139-f05a-40fe-8750-b3b5938bfacc-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ad17f139-f05a-40fe-8750-b3b5938bfacc-workload-certs\") pod \"ad17f139-f05a-40fe-8750-b3b5938bfacc\" (UID: \"ad17f139-f05a-40fe-8750-b3b5938bfacc\") " Nov 04 02:51:24.347852 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:51:24.347802 4405 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ad17f139-f05a-40fe-8750-b3b5938bfacc-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ad17f139-f05a-40fe-8750-b3b5938bfacc-istio-data\") pod \"ad17f139-f05a-40fe-8750-b3b5938bfacc\" (UID: \"ad17f139-f05a-40fe-8750-b3b5938bfacc\") " Nov 04 02:51:24.347852 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:51:24.347838 4405 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ad17f139-f05a-40fe-8750-b3b5938bfacc-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ad17f139-f05a-40fe-8750-b3b5938bfacc-workload-socket\") pod \"ad17f139-f05a-40fe-8750-b3b5938bfacc\" (UID: \"ad17f139-f05a-40fe-8750-b3b5938bfacc\") " Nov 04 02:51:24.347921 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:51:24.347859 4405 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ad17f139-f05a-40fe-8750-b3b5938bfacc-kube-api-access-4x8fr\" (UniqueName: \"kubernetes.io/projected/ad17f139-f05a-40fe-8750-b3b5938bfacc-kube-api-access-4x8fr\") pod \"ad17f139-f05a-40fe-8750-b3b5938bfacc\" (UID: \"ad17f139-f05a-40fe-8750-b3b5938bfacc\") " Nov 04 02:51:24.347963 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:51:24.347921 4405 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ad17f139-f05a-40fe-8750-b3b5938bfacc-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ad17f139-f05a-40fe-8750-b3b5938bfacc-credential-socket\") pod \"ad17f139-f05a-40fe-8750-b3b5938bfacc\" (UID: \"ad17f139-f05a-40fe-8750-b3b5938bfacc\") " Nov 04 02:51:24.347963 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:51:24.347923 4405 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ad17f139-f05a-40fe-8750-b3b5938bfacc-workload-certs" pod "ad17f139-f05a-40fe-8750-b3b5938bfacc" (UID: "ad17f139-f05a-40fe-8750-b3b5938bfacc"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 02:51:24.348040 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:51:24.347963 4405 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/ad17f139-f05a-40fe-8750-b3b5938bfacc-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ad17f139-f05a-40fe-8750-b3b5938bfacc-istio-podinfo\") pod \"ad17f139-f05a-40fe-8750-b3b5938bfacc\" (UID: \"ad17f139-f05a-40fe-8750-b3b5938bfacc\") " Nov 04 02:51:24.348040 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:51:24.348005 4405 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ad17f139-f05a-40fe-8750-b3b5938bfacc-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ad17f139-f05a-40fe-8750-b3b5938bfacc-istio-envoy\") pod \"ad17f139-f05a-40fe-8750-b3b5938bfacc\" (UID: \"ad17f139-f05a-40fe-8750-b3b5938bfacc\") " Nov 04 02:51:24.348040 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:51:24.348022 4405 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ad17f139-f05a-40fe-8750-b3b5938bfacc-istio-token\" (UniqueName: \"kubernetes.io/projected/ad17f139-f05a-40fe-8750-b3b5938bfacc-istio-token\") pod \"ad17f139-f05a-40fe-8750-b3b5938bfacc\" (UID: \"ad17f139-f05a-40fe-8750-b3b5938bfacc\") " Nov 04 02:51:24.348156 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:51:24.348058 4405 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ad17f139-f05a-40fe-8750-b3b5938bfacc-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ad17f139-f05a-40fe-8750-b3b5938bfacc-istiod-ca-cert\") pod \"ad17f139-f05a-40fe-8750-b3b5938bfacc\" (UID: \"ad17f139-f05a-40fe-8750-b3b5938bfacc\") " Nov 04 02:51:24.348156 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:51:24.348137 4405 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ad17f139-f05a-40fe-8750-b3b5938bfacc-istio-data" pod "ad17f139-f05a-40fe-8750-b3b5938bfacc" (UID: "ad17f139-f05a-40fe-8750-b3b5938bfacc"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 02:51:24.348253 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:51:24.348195 4405 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ad17f139-f05a-40fe-8750-b3b5938bfacc-credential-socket" pod "ad17f139-f05a-40fe-8750-b3b5938bfacc" (UID: "ad17f139-f05a-40fe-8750-b3b5938bfacc"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 02:51:24.348505 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:51:24.348409 4405 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ad17f139-f05a-40fe-8750-b3b5938bfacc-workload-socket" pod "ad17f139-f05a-40fe-8750-b3b5938bfacc" (UID: "ad17f139-f05a-40fe-8750-b3b5938bfacc"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 02:51:24.348658 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:51:24.348578 4405 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ad17f139-f05a-40fe-8750-b3b5938bfacc-istiod-ca-cert" pod "ad17f139-f05a-40fe-8750-b3b5938bfacc" (UID: "ad17f139-f05a-40fe-8750-b3b5938bfacc"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 02:51:24.348658 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:51:24.348602 4405 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ad17f139-f05a-40fe-8750-b3b5938bfacc-istio-data\") on node \"ip-10-0-134-125.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 02:51:24.348658 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:51:24.348621 4405 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ad17f139-f05a-40fe-8750-b3b5938bfacc-workload-socket\") on node \"ip-10-0-134-125.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 02:51:24.348658 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:51:24.348635 4405 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ad17f139-f05a-40fe-8750-b3b5938bfacc-credential-socket\") on node \"ip-10-0-134-125.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 02:51:24.348658 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:51:24.348657 4405 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ad17f139-f05a-40fe-8750-b3b5938bfacc-workload-certs\") on node \"ip-10-0-134-125.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 02:51:24.350420 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:51:24.350392 4405 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ad17f139-f05a-40fe-8750-b3b5938bfacc-istio-token" pod "ad17f139-f05a-40fe-8750-b3b5938bfacc" (UID: "ad17f139-f05a-40fe-8750-b3b5938bfacc"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 02:51:24.350900 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:51:24.350877 4405 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ad17f139-f05a-40fe-8750-b3b5938bfacc-istio-podinfo" pod "ad17f139-f05a-40fe-8750-b3b5938bfacc" (UID: "ad17f139-f05a-40fe-8750-b3b5938bfacc"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 04 02:51:24.350981 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:51:24.350900 4405 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ad17f139-f05a-40fe-8750-b3b5938bfacc-istio-envoy" pod "ad17f139-f05a-40fe-8750-b3b5938bfacc" (UID: "ad17f139-f05a-40fe-8750-b3b5938bfacc"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 02:51:24.350981 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:51:24.350906 4405 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ad17f139-f05a-40fe-8750-b3b5938bfacc-kube-api-access-4x8fr" pod "ad17f139-f05a-40fe-8750-b3b5938bfacc" (UID: "ad17f139-f05a-40fe-8750-b3b5938bfacc"). InnerVolumeSpecName "kube-api-access-4x8fr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 02:51:24.450068 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:51:24.450023 4405 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4x8fr\" (UniqueName: \"kubernetes.io/projected/ad17f139-f05a-40fe-8750-b3b5938bfacc-kube-api-access-4x8fr\") on node \"ip-10-0-134-125.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 02:51:24.450068 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:51:24.450069 4405 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ad17f139-f05a-40fe-8750-b3b5938bfacc-istio-podinfo\") on node \"ip-10-0-134-125.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 02:51:24.450284 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:51:24.450085 4405 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ad17f139-f05a-40fe-8750-b3b5938bfacc-istio-envoy\") on node \"ip-10-0-134-125.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 02:51:24.450284 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:51:24.450099 4405 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ad17f139-f05a-40fe-8750-b3b5938bfacc-istio-token\") on node \"ip-10-0-134-125.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 02:51:24.450284 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:51:24.450113 4405 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ad17f139-f05a-40fe-8750-b3b5938bfacc-istiod-ca-cert\") on node \"ip-10-0-134-125.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 02:51:25.243202 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:51:25.243169 4405 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-dhvd2" Nov 04 02:51:25.465054 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:51:25.459231 4405 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-76597d6855-dhvd2"] Nov 04 02:51:25.507406 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:51:25.507317 4405 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-76597d6855-dhvd2"] Nov 04 02:51:25.701750 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:51:25.701711 4405 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ad17f139-f05a-40fe-8750-b3b5938bfacc" path="/var/lib/kubelet/pods/ad17f139-f05a-40fe-8750-b3b5938bfacc/volumes" Nov 04 02:51:28.565446 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:51:28.565367 4405 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w3-cdcf67/distinct-w3-cdcf67-kserve-667cc7f7d7-5m2lq"] Nov 04 02:51:28.565940 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:51:28.565925 4405 cpu_manager.go:414] "containerMap: removing container" podUID="ad17f139-f05a-40fe-8750-b3b5938bfacc" containerName="istio-proxy" Nov 04 02:51:28.565985 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:51:28.565941 4405 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ad17f139-f05a-40fe-8750-b3b5938bfacc" containerName="istio-proxy" Nov 04 02:51:28.566029 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:51:28.566020 4405 memory_manager.go:365] "RemoveStaleState removing state" podUID="ad17f139-f05a-40fe-8750-b3b5938bfacc" containerName="istio-proxy" Nov 04 02:51:28.570577 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:51:28.570557 4405 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-cdcf67/distinct-w3-cdcf67-kserve-667cc7f7d7-5m2lq" Nov 04 02:51:28.597079 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:51:28.597049 4405 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/52ed22cc-96e4-4ef0-b34d-ed58c89a9183-dshm\") pod \"distinct-w3-cdcf67-kserve-667cc7f7d7-5m2lq\" (UID: \"52ed22cc-96e4-4ef0-b34d-ed58c89a9183\") " pod="e2e-models-e2e-worker-w3-cdcf67/distinct-w3-cdcf67-kserve-667cc7f7d7-5m2lq" Nov 04 02:51:28.597271 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:51:28.597096 4405 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/52ed22cc-96e4-4ef0-b34d-ed58c89a9183-tmp-dir\") pod \"distinct-w3-cdcf67-kserve-667cc7f7d7-5m2lq\" (UID: \"52ed22cc-96e4-4ef0-b34d-ed58c89a9183\") " pod="e2e-models-e2e-worker-w3-cdcf67/distinct-w3-cdcf67-kserve-667cc7f7d7-5m2lq" Nov 04 02:51:28.597271 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:51:28.597225 4405 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/52ed22cc-96e4-4ef0-b34d-ed58c89a9183-home\") pod \"distinct-w3-cdcf67-kserve-667cc7f7d7-5m2lq\" (UID: \"52ed22cc-96e4-4ef0-b34d-ed58c89a9183\") " pod="e2e-models-e2e-worker-w3-cdcf67/distinct-w3-cdcf67-kserve-667cc7f7d7-5m2lq" Nov 04 02:51:28.597271 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:51:28.597258 4405 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/52ed22cc-96e4-4ef0-b34d-ed58c89a9183-tls-certs\") pod \"distinct-w3-cdcf67-kserve-667cc7f7d7-5m2lq\" (UID: \"52ed22cc-96e4-4ef0-b34d-ed58c89a9183\") " pod="e2e-models-e2e-worker-w3-cdcf67/distinct-w3-cdcf67-kserve-667cc7f7d7-5m2lq" Nov 04 02:51:28.597454 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:51:28.597290 4405 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/52ed22cc-96e4-4ef0-b34d-ed58c89a9183-model-cache\") pod \"distinct-w3-cdcf67-kserve-667cc7f7d7-5m2lq\" (UID: \"52ed22cc-96e4-4ef0-b34d-ed58c89a9183\") " pod="e2e-models-e2e-worker-w3-cdcf67/distinct-w3-cdcf67-kserve-667cc7f7d7-5m2lq" Nov 04 02:51:28.597454 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:51:28.597408 4405 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pb5bk\" (UniqueName: \"kubernetes.io/projected/52ed22cc-96e4-4ef0-b34d-ed58c89a9183-kube-api-access-pb5bk\") pod \"distinct-w3-cdcf67-kserve-667cc7f7d7-5m2lq\" (UID: \"52ed22cc-96e4-4ef0-b34d-ed58c89a9183\") " pod="e2e-models-e2e-worker-w3-cdcf67/distinct-w3-cdcf67-kserve-667cc7f7d7-5m2lq" Nov 04 02:51:28.625469 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:51:28.625435 4405 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-cdcf67\"/\"default-dockercfg-dp8tf\"" Nov 04 02:51:28.675347 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:51:28.675308 4405 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-cdcf67\"/\"distinct-w3-cdcf67-kserve-self-signed-certs\"" Nov 04 02:51:28.675546 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:51:28.675357 4405 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w3-cdcf67\"/\"kube-root-ca.crt\"" Nov 04 02:51:28.675546 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:51:28.675398 4405 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w3-cdcf67\"/\"openshift-service-ca.crt\"" Nov 04 02:51:28.680545 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:51:28.680516 4405 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-cdcf67/distinct-w3-cdcf67-kserve-667cc7f7d7-5m2lq"] Nov 04 02:51:28.698543 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:51:28.698516 4405 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/52ed22cc-96e4-4ef0-b34d-ed58c89a9183-dshm\") pod \"distinct-w3-cdcf67-kserve-667cc7f7d7-5m2lq\" (UID: \"52ed22cc-96e4-4ef0-b34d-ed58c89a9183\") " pod="e2e-models-e2e-worker-w3-cdcf67/distinct-w3-cdcf67-kserve-667cc7f7d7-5m2lq" Nov 04 02:51:28.698746 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:51:28.698561 4405 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/52ed22cc-96e4-4ef0-b34d-ed58c89a9183-tmp-dir\") pod \"distinct-w3-cdcf67-kserve-667cc7f7d7-5m2lq\" (UID: \"52ed22cc-96e4-4ef0-b34d-ed58c89a9183\") " pod="e2e-models-e2e-worker-w3-cdcf67/distinct-w3-cdcf67-kserve-667cc7f7d7-5m2lq" Nov 04 02:51:28.698746 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:51:28.698621 4405 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/52ed22cc-96e4-4ef0-b34d-ed58c89a9183-home\") pod \"distinct-w3-cdcf67-kserve-667cc7f7d7-5m2lq\" (UID: \"52ed22cc-96e4-4ef0-b34d-ed58c89a9183\") " pod="e2e-models-e2e-worker-w3-cdcf67/distinct-w3-cdcf67-kserve-667cc7f7d7-5m2lq" Nov 04 02:51:28.698746 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:51:28.698650 4405 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/52ed22cc-96e4-4ef0-b34d-ed58c89a9183-tls-certs\") pod \"distinct-w3-cdcf67-kserve-667cc7f7d7-5m2lq\" (UID: \"52ed22cc-96e4-4ef0-b34d-ed58c89a9183\") " pod="e2e-models-e2e-worker-w3-cdcf67/distinct-w3-cdcf67-kserve-667cc7f7d7-5m2lq" Nov 04 02:51:28.698746 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:51:28.698705 4405 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/52ed22cc-96e4-4ef0-b34d-ed58c89a9183-model-cache\") pod \"distinct-w3-cdcf67-kserve-667cc7f7d7-5m2lq\" (UID: \"52ed22cc-96e4-4ef0-b34d-ed58c89a9183\") " pod="e2e-models-e2e-worker-w3-cdcf67/distinct-w3-cdcf67-kserve-667cc7f7d7-5m2lq" Nov 04 02:51:28.698949 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:51:28.698767 4405 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pb5bk\" (UniqueName: \"kubernetes.io/projected/52ed22cc-96e4-4ef0-b34d-ed58c89a9183-kube-api-access-pb5bk\") pod \"distinct-w3-cdcf67-kserve-667cc7f7d7-5m2lq\" (UID: \"52ed22cc-96e4-4ef0-b34d-ed58c89a9183\") " pod="e2e-models-e2e-worker-w3-cdcf67/distinct-w3-cdcf67-kserve-667cc7f7d7-5m2lq" Nov 04 02:51:28.699120 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:51:28.699085 4405 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/52ed22cc-96e4-4ef0-b34d-ed58c89a9183-home\") pod \"distinct-w3-cdcf67-kserve-667cc7f7d7-5m2lq\" (UID: \"52ed22cc-96e4-4ef0-b34d-ed58c89a9183\") " pod="e2e-models-e2e-worker-w3-cdcf67/distinct-w3-cdcf67-kserve-667cc7f7d7-5m2lq" Nov 04 02:51:28.699264 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:51:28.699138 4405 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/52ed22cc-96e4-4ef0-b34d-ed58c89a9183-tmp-dir\") pod \"distinct-w3-cdcf67-kserve-667cc7f7d7-5m2lq\" (UID: \"52ed22cc-96e4-4ef0-b34d-ed58c89a9183\") " pod="e2e-models-e2e-worker-w3-cdcf67/distinct-w3-cdcf67-kserve-667cc7f7d7-5m2lq" Nov 04 02:51:28.699478 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:51:28.699412 4405 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/52ed22cc-96e4-4ef0-b34d-ed58c89a9183-model-cache\") pod \"distinct-w3-cdcf67-kserve-667cc7f7d7-5m2lq\" (UID: \"52ed22cc-96e4-4ef0-b34d-ed58c89a9183\") " pod="e2e-models-e2e-worker-w3-cdcf67/distinct-w3-cdcf67-kserve-667cc7f7d7-5m2lq" Nov 04 02:51:28.700979 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:51:28.700959 4405 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/52ed22cc-96e4-4ef0-b34d-ed58c89a9183-dshm\") pod \"distinct-w3-cdcf67-kserve-667cc7f7d7-5m2lq\" (UID: \"52ed22cc-96e4-4ef0-b34d-ed58c89a9183\") " pod="e2e-models-e2e-worker-w3-cdcf67/distinct-w3-cdcf67-kserve-667cc7f7d7-5m2lq" Nov 04 02:51:28.701234 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:51:28.701218 4405 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/52ed22cc-96e4-4ef0-b34d-ed58c89a9183-tls-certs\") pod \"distinct-w3-cdcf67-kserve-667cc7f7d7-5m2lq\" (UID: \"52ed22cc-96e4-4ef0-b34d-ed58c89a9183\") " pod="e2e-models-e2e-worker-w3-cdcf67/distinct-w3-cdcf67-kserve-667cc7f7d7-5m2lq" Nov 04 02:51:28.760023 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:51:28.759989 4405 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pb5bk\" (UniqueName: \"kubernetes.io/projected/52ed22cc-96e4-4ef0-b34d-ed58c89a9183-kube-api-access-pb5bk\") pod \"distinct-w3-cdcf67-kserve-667cc7f7d7-5m2lq\" (UID: \"52ed22cc-96e4-4ef0-b34d-ed58c89a9183\") " pod="e2e-models-e2e-worker-w3-cdcf67/distinct-w3-cdcf67-kserve-667cc7f7d7-5m2lq" Nov 04 02:51:28.883377 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:51:28.883337 4405 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-cdcf67/distinct-w3-cdcf67-kserve-667cc7f7d7-5m2lq" Nov 04 02:51:29.289465 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:51:29.289433 4405 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-cdcf67/distinct-w3-cdcf67-kserve-667cc7f7d7-5m2lq"] Nov 04 02:51:29.291561 ip-10-0-134-125 kubenswrapper[4405]: W1104 02:51:29.291527 4405 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod52ed22cc_96e4_4ef0_b34d_ed58c89a9183.slice/crio-18102e2aa5eea61d2882013dfcb2d162e34b669e9d004cab6c90b3c3c8225a2f WatchSource:0}: Error finding container 18102e2aa5eea61d2882013dfcb2d162e34b669e9d004cab6c90b3c3c8225a2f: Status 404 returned error can't find the container with id 18102e2aa5eea61d2882013dfcb2d162e34b669e9d004cab6c90b3c3c8225a2f Nov 04 02:51:30.274772 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:51:30.274726 4405 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-cdcf67/distinct-w3-cdcf67-kserve-667cc7f7d7-5m2lq" event={"ID":"52ed22cc-96e4-4ef0-b34d-ed58c89a9183","Type":"ContainerStarted","Data":"bf8c65e09b476c923b798d5224fc5e9e0b4a06a3dbd06904a953f30cddbc6571"} Nov 04 02:51:30.275195 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:51:30.274783 4405 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-cdcf67/distinct-w3-cdcf67-kserve-667cc7f7d7-5m2lq" event={"ID":"52ed22cc-96e4-4ef0-b34d-ed58c89a9183","Type":"ContainerStarted","Data":"18102e2aa5eea61d2882013dfcb2d162e34b669e9d004cab6c90b3c3c8225a2f"} Nov 04 02:51:30.391566 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:51:30.391511 4405 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w3-cdcf67/distinct-w3-cdcf67-kserve-667cc7f7d7-5m2lq" podStartSLOduration=2.391493288 podStartE2EDuration="2.391493288s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 02:51:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 02:51:30.391028676 +0000 UTC m=+1136.821728798" watchObservedRunningTime="2026-11-04 02:51:30.391493288 +0000 UTC m=+1136.822193468" Nov 04 02:51:30.883888 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:51:30.883845 4405 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-cdcf67/distinct-w3-cdcf67-kserve-667cc7f7d7-5m2lq" Nov 04 02:51:33.829325 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:51:33.829294 4405 scope.go:122] "RemoveContainer" containerID="abab077e0b3bacb6976169fa82208abeba6c11efc953d43029100ba49ba15ee4" Nov 04 02:51:36.884173 ip-10-0-134-125 kubenswrapper[4405]: E1104 02:51:36.884136 4405 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5f2vb" containerName="extractor" Nov 04 02:51:38.883858 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:51:38.883819 4405 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w3-cdcf67/distinct-w3-cdcf67-kserve-667cc7f7d7-5m2lq" Nov 04 02:51:38.899014 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:51:38.898983 4405 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w3-cdcf67/distinct-w3-cdcf67-kserve-667cc7f7d7-5m2lq" Nov 04 02:51:39.343455 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:51:39.343429 4405 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-cdcf67/distinct-w3-cdcf67-kserve-667cc7f7d7-5m2lq" Nov 04 02:51:39.870503 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:51:39.870467 4405 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-bl5mg"] Nov 04 02:51:48.503519 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:51:48.503484 4405 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-e516ff-openshift-default-6f4b957485-xg2tv"] Nov 04 02:51:48.508989 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:51:48.508970 4405 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-e516ff-openshift-default-6f4b957485-xg2tv" Nov 04 02:51:48.632868 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:51:48.632826 4405 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a4bcafa2-ec97-486c-819f-7be10e3110a9-istiod-ca-cert\") pod \"e2e-worker-w1-e516ff-openshift-default-6f4b957485-xg2tv\" (UID: \"a4bcafa2-ec97-486c-819f-7be10e3110a9\") " pod="openshift-ingress/e2e-worker-w1-e516ff-openshift-default-6f4b957485-xg2tv" Nov 04 02:51:48.633067 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:51:48.632960 4405 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a4bcafa2-ec97-486c-819f-7be10e3110a9-istio-envoy\") pod \"e2e-worker-w1-e516ff-openshift-default-6f4b957485-xg2tv\" (UID: \"a4bcafa2-ec97-486c-819f-7be10e3110a9\") " pod="openshift-ingress/e2e-worker-w1-e516ff-openshift-default-6f4b957485-xg2tv" Nov 04 02:51:48.633067 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:51:48.633046 4405 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fcp55\" (UniqueName: \"kubernetes.io/projected/a4bcafa2-ec97-486c-819f-7be10e3110a9-kube-api-access-fcp55\") pod \"e2e-worker-w1-e516ff-openshift-default-6f4b957485-xg2tv\" (UID: \"a4bcafa2-ec97-486c-819f-7be10e3110a9\") " pod="openshift-ingress/e2e-worker-w1-e516ff-openshift-default-6f4b957485-xg2tv" Nov 04 02:51:48.633190 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:51:48.633113 4405 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a4bcafa2-ec97-486c-819f-7be10e3110a9-istio-podinfo\") pod \"e2e-worker-w1-e516ff-openshift-default-6f4b957485-xg2tv\" (UID: \"a4bcafa2-ec97-486c-819f-7be10e3110a9\") " pod="openshift-ingress/e2e-worker-w1-e516ff-openshift-default-6f4b957485-xg2tv" Nov 04 02:51:48.633254 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:51:48.633237 4405 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a4bcafa2-ec97-486c-819f-7be10e3110a9-istio-data\") pod \"e2e-worker-w1-e516ff-openshift-default-6f4b957485-xg2tv\" (UID: \"a4bcafa2-ec97-486c-819f-7be10e3110a9\") " pod="openshift-ingress/e2e-worker-w1-e516ff-openshift-default-6f4b957485-xg2tv" Nov 04 02:51:48.633331 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:51:48.633290 4405 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a4bcafa2-ec97-486c-819f-7be10e3110a9-credential-socket\") pod \"e2e-worker-w1-e516ff-openshift-default-6f4b957485-xg2tv\" (UID: \"a4bcafa2-ec97-486c-819f-7be10e3110a9\") " pod="openshift-ingress/e2e-worker-w1-e516ff-openshift-default-6f4b957485-xg2tv" Nov 04 02:51:48.633413 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:51:48.633398 4405 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a4bcafa2-ec97-486c-819f-7be10e3110a9-istio-token\") pod \"e2e-worker-w1-e516ff-openshift-default-6f4b957485-xg2tv\" (UID: \"a4bcafa2-ec97-486c-819f-7be10e3110a9\") " pod="openshift-ingress/e2e-worker-w1-e516ff-openshift-default-6f4b957485-xg2tv" Nov 04 02:51:48.633478 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:51:48.633463 4405 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a4bcafa2-ec97-486c-819f-7be10e3110a9-workload-certs\") pod \"e2e-worker-w1-e516ff-openshift-default-6f4b957485-xg2tv\" (UID: \"a4bcafa2-ec97-486c-819f-7be10e3110a9\") " pod="openshift-ingress/e2e-worker-w1-e516ff-openshift-default-6f4b957485-xg2tv" Nov 04 02:51:48.633562 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:51:48.633541 4405 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a4bcafa2-ec97-486c-819f-7be10e3110a9-workload-socket\") pod \"e2e-worker-w1-e516ff-openshift-default-6f4b957485-xg2tv\" (UID: \"a4bcafa2-ec97-486c-819f-7be10e3110a9\") " pod="openshift-ingress/e2e-worker-w1-e516ff-openshift-default-6f4b957485-xg2tv" Nov 04 02:51:48.726507 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:51:48.726474 4405 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-e516ff-openshift-default-6f4b957485-xg2tv"] Nov 04 02:51:48.731101 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:51:48.731071 4405 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-203c35-openshift-default-56497cdcc-z59vn"] Nov 04 02:51:48.734901 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:51:48.734809 4405 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a4bcafa2-ec97-486c-819f-7be10e3110a9-istio-data\") pod \"e2e-worker-w1-e516ff-openshift-default-6f4b957485-xg2tv\" (UID: \"a4bcafa2-ec97-486c-819f-7be10e3110a9\") " pod="openshift-ingress/e2e-worker-w1-e516ff-openshift-default-6f4b957485-xg2tv" Nov 04 02:51:48.734901 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:51:48.734877 4405 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a4bcafa2-ec97-486c-819f-7be10e3110a9-credential-socket\") pod \"e2e-worker-w1-e516ff-openshift-default-6f4b957485-xg2tv\" (UID: \"a4bcafa2-ec97-486c-819f-7be10e3110a9\") " pod="openshift-ingress/e2e-worker-w1-e516ff-openshift-default-6f4b957485-xg2tv" Nov 04 02:51:48.735119 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:51:48.734922 4405 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a4bcafa2-ec97-486c-819f-7be10e3110a9-istio-token\") pod \"e2e-worker-w1-e516ff-openshift-default-6f4b957485-xg2tv\" (UID: \"a4bcafa2-ec97-486c-819f-7be10e3110a9\") " pod="openshift-ingress/e2e-worker-w1-e516ff-openshift-default-6f4b957485-xg2tv" Nov 04 02:51:48.735119 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:51:48.734984 4405 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a4bcafa2-ec97-486c-819f-7be10e3110a9-workload-certs\") pod \"e2e-worker-w1-e516ff-openshift-default-6f4b957485-xg2tv\" (UID: \"a4bcafa2-ec97-486c-819f-7be10e3110a9\") " pod="openshift-ingress/e2e-worker-w1-e516ff-openshift-default-6f4b957485-xg2tv" Nov 04 02:51:48.735119 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:51:48.735024 4405 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a4bcafa2-ec97-486c-819f-7be10e3110a9-workload-socket\") pod \"e2e-worker-w1-e516ff-openshift-default-6f4b957485-xg2tv\" (UID: \"a4bcafa2-ec97-486c-819f-7be10e3110a9\") " pod="openshift-ingress/e2e-worker-w1-e516ff-openshift-default-6f4b957485-xg2tv" Nov 04 02:51:48.735119 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:51:48.735053 4405 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a4bcafa2-ec97-486c-819f-7be10e3110a9-istiod-ca-cert\") pod \"e2e-worker-w1-e516ff-openshift-default-6f4b957485-xg2tv\" (UID: \"a4bcafa2-ec97-486c-819f-7be10e3110a9\") " pod="openshift-ingress/e2e-worker-w1-e516ff-openshift-default-6f4b957485-xg2tv" Nov 04 02:51:48.735119 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:51:48.735116 4405 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a4bcafa2-ec97-486c-819f-7be10e3110a9-istio-envoy\") pod \"e2e-worker-w1-e516ff-openshift-default-6f4b957485-xg2tv\" (UID: \"a4bcafa2-ec97-486c-819f-7be10e3110a9\") " pod="openshift-ingress/e2e-worker-w1-e516ff-openshift-default-6f4b957485-xg2tv" Nov 04 02:51:48.735361 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:51:48.735178 4405 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fcp55\" (UniqueName: \"kubernetes.io/projected/a4bcafa2-ec97-486c-819f-7be10e3110a9-kube-api-access-fcp55\") pod \"e2e-worker-w1-e516ff-openshift-default-6f4b957485-xg2tv\" (UID: \"a4bcafa2-ec97-486c-819f-7be10e3110a9\") " pod="openshift-ingress/e2e-worker-w1-e516ff-openshift-default-6f4b957485-xg2tv" Nov 04 02:51:48.735361 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:51:48.735206 4405 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a4bcafa2-ec97-486c-819f-7be10e3110a9-istio-podinfo\") pod \"e2e-worker-w1-e516ff-openshift-default-6f4b957485-xg2tv\" (UID: \"a4bcafa2-ec97-486c-819f-7be10e3110a9\") " pod="openshift-ingress/e2e-worker-w1-e516ff-openshift-default-6f4b957485-xg2tv" Nov 04 02:51:48.735361 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:51:48.735229 4405 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a4bcafa2-ec97-486c-819f-7be10e3110a9-istio-data\") pod \"e2e-worker-w1-e516ff-openshift-default-6f4b957485-xg2tv\" (UID: \"a4bcafa2-ec97-486c-819f-7be10e3110a9\") " pod="openshift-ingress/e2e-worker-w1-e516ff-openshift-default-6f4b957485-xg2tv" Nov 04 02:51:48.735626 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:51:48.735596 4405 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a4bcafa2-ec97-486c-819f-7be10e3110a9-workload-socket\") pod \"e2e-worker-w1-e516ff-openshift-default-6f4b957485-xg2tv\" (UID: \"a4bcafa2-ec97-486c-819f-7be10e3110a9\") " pod="openshift-ingress/e2e-worker-w1-e516ff-openshift-default-6f4b957485-xg2tv" Nov 04 02:51:48.735879 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:51:48.735858 4405 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a4bcafa2-ec97-486c-819f-7be10e3110a9-credential-socket\") pod \"e2e-worker-w1-e516ff-openshift-default-6f4b957485-xg2tv\" (UID: \"a4bcafa2-ec97-486c-819f-7be10e3110a9\") " pod="openshift-ingress/e2e-worker-w1-e516ff-openshift-default-6f4b957485-xg2tv" Nov 04 02:51:48.736572 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:51:48.736517 4405 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a4bcafa2-ec97-486c-819f-7be10e3110a9-workload-certs\") pod \"e2e-worker-w1-e516ff-openshift-default-6f4b957485-xg2tv\" (UID: \"a4bcafa2-ec97-486c-819f-7be10e3110a9\") " pod="openshift-ingress/e2e-worker-w1-e516ff-openshift-default-6f4b957485-xg2tv" Nov 04 02:51:48.737020 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:51:48.736999 4405 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a4bcafa2-ec97-486c-819f-7be10e3110a9-istiod-ca-cert\") pod \"e2e-worker-w1-e516ff-openshift-default-6f4b957485-xg2tv\" (UID: \"a4bcafa2-ec97-486c-819f-7be10e3110a9\") " pod="openshift-ingress/e2e-worker-w1-e516ff-openshift-default-6f4b957485-xg2tv" Nov 04 02:51:48.737299 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:51:48.737284 4405 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-203c35-openshift-default-56497cdcc-z59vn" Nov 04 02:51:48.738598 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:51:48.738573 4405 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a4bcafa2-ec97-486c-819f-7be10e3110a9-istio-envoy\") pod \"e2e-worker-w1-e516ff-openshift-default-6f4b957485-xg2tv\" (UID: \"a4bcafa2-ec97-486c-819f-7be10e3110a9\") " pod="openshift-ingress/e2e-worker-w1-e516ff-openshift-default-6f4b957485-xg2tv" Nov 04 02:51:48.738917 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:51:48.738898 4405 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a4bcafa2-ec97-486c-819f-7be10e3110a9-istio-podinfo\") pod \"e2e-worker-w1-e516ff-openshift-default-6f4b957485-xg2tv\" (UID: \"a4bcafa2-ec97-486c-819f-7be10e3110a9\") " pod="openshift-ingress/e2e-worker-w1-e516ff-openshift-default-6f4b957485-xg2tv" Nov 04 02:51:48.797028 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:51:48.796992 4405 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a4bcafa2-ec97-486c-819f-7be10e3110a9-istio-token\") pod \"e2e-worker-w1-e516ff-openshift-default-6f4b957485-xg2tv\" (UID: \"a4bcafa2-ec97-486c-819f-7be10e3110a9\") " pod="openshift-ingress/e2e-worker-w1-e516ff-openshift-default-6f4b957485-xg2tv" Nov 04 02:51:48.797182 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:51:48.797084 4405 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fcp55\" (UniqueName: \"kubernetes.io/projected/a4bcafa2-ec97-486c-819f-7be10e3110a9-kube-api-access-fcp55\") pod \"e2e-worker-w1-e516ff-openshift-default-6f4b957485-xg2tv\" (UID: \"a4bcafa2-ec97-486c-819f-7be10e3110a9\") " pod="openshift-ingress/e2e-worker-w1-e516ff-openshift-default-6f4b957485-xg2tv" Nov 04 02:51:48.825113 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:51:48.825076 4405 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-203c35-openshift-default-56497cdcc-z59vn"] Nov 04 02:51:48.829934 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:51:48.829904 4405 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-e516ff-openshift-default-6f4b957485-xg2tv" Nov 04 02:51:48.838203 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:51:48.836773 4405 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5ebc4fc9-8882-43c6-8713-1ee995a8ee00-istio-envoy\") pod \"e2e-worker-w4-203c35-openshift-default-56497cdcc-z59vn\" (UID: \"5ebc4fc9-8882-43c6-8713-1ee995a8ee00\") " pod="openshift-ingress/e2e-worker-w4-203c35-openshift-default-56497cdcc-z59vn" Nov 04 02:51:48.838203 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:51:48.836859 4405 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5ebc4fc9-8882-43c6-8713-1ee995a8ee00-workload-certs\") pod \"e2e-worker-w4-203c35-openshift-default-56497cdcc-z59vn\" (UID: \"5ebc4fc9-8882-43c6-8713-1ee995a8ee00\") " pod="openshift-ingress/e2e-worker-w4-203c35-openshift-default-56497cdcc-z59vn" Nov 04 02:51:48.838203 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:51:48.837032 4405 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5ebc4fc9-8882-43c6-8713-1ee995a8ee00-workload-socket\") pod \"e2e-worker-w4-203c35-openshift-default-56497cdcc-z59vn\" (UID: \"5ebc4fc9-8882-43c6-8713-1ee995a8ee00\") " pod="openshift-ingress/e2e-worker-w4-203c35-openshift-default-56497cdcc-z59vn" Nov 04 02:51:48.838203 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:51:48.837067 4405 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5ebc4fc9-8882-43c6-8713-1ee995a8ee00-istio-podinfo\") pod \"e2e-worker-w4-203c35-openshift-default-56497cdcc-z59vn\" (UID: \"5ebc4fc9-8882-43c6-8713-1ee995a8ee00\") " pod="openshift-ingress/e2e-worker-w4-203c35-openshift-default-56497cdcc-z59vn" Nov 04 02:51:48.838203 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:51:48.837192 4405 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5ebc4fc9-8882-43c6-8713-1ee995a8ee00-istio-data\") pod \"e2e-worker-w4-203c35-openshift-default-56497cdcc-z59vn\" (UID: \"5ebc4fc9-8882-43c6-8713-1ee995a8ee00\") " pod="openshift-ingress/e2e-worker-w4-203c35-openshift-default-56497cdcc-z59vn" Nov 04 02:51:48.838203 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:51:48.837250 4405 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5ebc4fc9-8882-43c6-8713-1ee995a8ee00-istiod-ca-cert\") pod \"e2e-worker-w4-203c35-openshift-default-56497cdcc-z59vn\" (UID: \"5ebc4fc9-8882-43c6-8713-1ee995a8ee00\") " pod="openshift-ingress/e2e-worker-w4-203c35-openshift-default-56497cdcc-z59vn" Nov 04 02:51:48.838203 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:51:48.837301 4405 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-br6lx\" (UniqueName: \"kubernetes.io/projected/5ebc4fc9-8882-43c6-8713-1ee995a8ee00-kube-api-access-br6lx\") pod \"e2e-worker-w4-203c35-openshift-default-56497cdcc-z59vn\" (UID: \"5ebc4fc9-8882-43c6-8713-1ee995a8ee00\") " pod="openshift-ingress/e2e-worker-w4-203c35-openshift-default-56497cdcc-z59vn" Nov 04 02:51:48.838203 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:51:48.837402 4405 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5ebc4fc9-8882-43c6-8713-1ee995a8ee00-istio-token\") pod \"e2e-worker-w4-203c35-openshift-default-56497cdcc-z59vn\" (UID: \"5ebc4fc9-8882-43c6-8713-1ee995a8ee00\") " pod="openshift-ingress/e2e-worker-w4-203c35-openshift-default-56497cdcc-z59vn" Nov 04 02:51:48.838203 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:51:48.837437 4405 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5ebc4fc9-8882-43c6-8713-1ee995a8ee00-credential-socket\") pod \"e2e-worker-w4-203c35-openshift-default-56497cdcc-z59vn\" (UID: \"5ebc4fc9-8882-43c6-8713-1ee995a8ee00\") " pod="openshift-ingress/e2e-worker-w4-203c35-openshift-default-56497cdcc-z59vn" Nov 04 02:51:48.938446 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:51:48.938405 4405 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5ebc4fc9-8882-43c6-8713-1ee995a8ee00-workload-socket\") pod \"e2e-worker-w4-203c35-openshift-default-56497cdcc-z59vn\" (UID: \"5ebc4fc9-8882-43c6-8713-1ee995a8ee00\") " pod="openshift-ingress/e2e-worker-w4-203c35-openshift-default-56497cdcc-z59vn" Nov 04 02:51:48.938623 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:51:48.938454 4405 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5ebc4fc9-8882-43c6-8713-1ee995a8ee00-istio-podinfo\") pod \"e2e-worker-w4-203c35-openshift-default-56497cdcc-z59vn\" (UID: \"5ebc4fc9-8882-43c6-8713-1ee995a8ee00\") " pod="openshift-ingress/e2e-worker-w4-203c35-openshift-default-56497cdcc-z59vn" Nov 04 02:51:48.938623 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:51:48.938488 4405 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5ebc4fc9-8882-43c6-8713-1ee995a8ee00-istio-data\") pod \"e2e-worker-w4-203c35-openshift-default-56497cdcc-z59vn\" (UID: \"5ebc4fc9-8882-43c6-8713-1ee995a8ee00\") " pod="openshift-ingress/e2e-worker-w4-203c35-openshift-default-56497cdcc-z59vn" Nov 04 02:51:48.938623 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:51:48.938517 4405 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5ebc4fc9-8882-43c6-8713-1ee995a8ee00-istiod-ca-cert\") pod \"e2e-worker-w4-203c35-openshift-default-56497cdcc-z59vn\" (UID: \"5ebc4fc9-8882-43c6-8713-1ee995a8ee00\") " pod="openshift-ingress/e2e-worker-w4-203c35-openshift-default-56497cdcc-z59vn" Nov 04 02:51:48.938623 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:51:48.938563 4405 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-br6lx\" (UniqueName: \"kubernetes.io/projected/5ebc4fc9-8882-43c6-8713-1ee995a8ee00-kube-api-access-br6lx\") pod \"e2e-worker-w4-203c35-openshift-default-56497cdcc-z59vn\" (UID: \"5ebc4fc9-8882-43c6-8713-1ee995a8ee00\") " pod="openshift-ingress/e2e-worker-w4-203c35-openshift-default-56497cdcc-z59vn" Nov 04 02:51:48.938875 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:51:48.938628 4405 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5ebc4fc9-8882-43c6-8713-1ee995a8ee00-istio-token\") pod \"e2e-worker-w4-203c35-openshift-default-56497cdcc-z59vn\" (UID: \"5ebc4fc9-8882-43c6-8713-1ee995a8ee00\") " pod="openshift-ingress/e2e-worker-w4-203c35-openshift-default-56497cdcc-z59vn" Nov 04 02:51:48.938875 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:51:48.938689 4405 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5ebc4fc9-8882-43c6-8713-1ee995a8ee00-credential-socket\") pod \"e2e-worker-w4-203c35-openshift-default-56497cdcc-z59vn\" (UID: \"5ebc4fc9-8882-43c6-8713-1ee995a8ee00\") " pod="openshift-ingress/e2e-worker-w4-203c35-openshift-default-56497cdcc-z59vn" Nov 04 02:51:48.938875 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:51:48.938758 4405 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5ebc4fc9-8882-43c6-8713-1ee995a8ee00-istio-envoy\") pod \"e2e-worker-w4-203c35-openshift-default-56497cdcc-z59vn\" (UID: \"5ebc4fc9-8882-43c6-8713-1ee995a8ee00\") " pod="openshift-ingress/e2e-worker-w4-203c35-openshift-default-56497cdcc-z59vn" Nov 04 02:51:48.938875 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:51:48.938831 4405 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5ebc4fc9-8882-43c6-8713-1ee995a8ee00-workload-certs\") pod \"e2e-worker-w4-203c35-openshift-default-56497cdcc-z59vn\" (UID: \"5ebc4fc9-8882-43c6-8713-1ee995a8ee00\") " pod="openshift-ingress/e2e-worker-w4-203c35-openshift-default-56497cdcc-z59vn" Nov 04 02:51:48.939876 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:51:48.939327 4405 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5ebc4fc9-8882-43c6-8713-1ee995a8ee00-workload-certs\") pod \"e2e-worker-w4-203c35-openshift-default-56497cdcc-z59vn\" (UID: \"5ebc4fc9-8882-43c6-8713-1ee995a8ee00\") " pod="openshift-ingress/e2e-worker-w4-203c35-openshift-default-56497cdcc-z59vn" Nov 04 02:51:48.939876 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:51:48.939516 4405 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5ebc4fc9-8882-43c6-8713-1ee995a8ee00-credential-socket\") pod \"e2e-worker-w4-203c35-openshift-default-56497cdcc-z59vn\" (UID: \"5ebc4fc9-8882-43c6-8713-1ee995a8ee00\") " pod="openshift-ingress/e2e-worker-w4-203c35-openshift-default-56497cdcc-z59vn" Nov 04 02:51:48.939876 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:51:48.939820 4405 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5ebc4fc9-8882-43c6-8713-1ee995a8ee00-workload-socket\") pod \"e2e-worker-w4-203c35-openshift-default-56497cdcc-z59vn\" (UID: \"5ebc4fc9-8882-43c6-8713-1ee995a8ee00\") " pod="openshift-ingress/e2e-worker-w4-203c35-openshift-default-56497cdcc-z59vn" Nov 04 02:51:48.940078 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:51:48.939894 4405 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5ebc4fc9-8882-43c6-8713-1ee995a8ee00-istio-data\") pod \"e2e-worker-w4-203c35-openshift-default-56497cdcc-z59vn\" (UID: \"5ebc4fc9-8882-43c6-8713-1ee995a8ee00\") " pod="openshift-ingress/e2e-worker-w4-203c35-openshift-default-56497cdcc-z59vn" Nov 04 02:51:48.940363 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:51:48.940167 4405 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5ebc4fc9-8882-43c6-8713-1ee995a8ee00-istiod-ca-cert\") pod \"e2e-worker-w4-203c35-openshift-default-56497cdcc-z59vn\" (UID: \"5ebc4fc9-8882-43c6-8713-1ee995a8ee00\") " pod="openshift-ingress/e2e-worker-w4-203c35-openshift-default-56497cdcc-z59vn" Nov 04 02:51:48.942155 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:51:48.942135 4405 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5ebc4fc9-8882-43c6-8713-1ee995a8ee00-istio-envoy\") pod \"e2e-worker-w4-203c35-openshift-default-56497cdcc-z59vn\" (UID: \"5ebc4fc9-8882-43c6-8713-1ee995a8ee00\") " pod="openshift-ingress/e2e-worker-w4-203c35-openshift-default-56497cdcc-z59vn" Nov 04 02:51:48.942987 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:51:48.942970 4405 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5ebc4fc9-8882-43c6-8713-1ee995a8ee00-istio-podinfo\") pod \"e2e-worker-w4-203c35-openshift-default-56497cdcc-z59vn\" (UID: \"5ebc4fc9-8882-43c6-8713-1ee995a8ee00\") " pod="openshift-ingress/e2e-worker-w4-203c35-openshift-default-56497cdcc-z59vn" Nov 04 02:51:49.000234 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:51:49.000190 4405 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5ebc4fc9-8882-43c6-8713-1ee995a8ee00-istio-token\") pod \"e2e-worker-w4-203c35-openshift-default-56497cdcc-z59vn\" (UID: \"5ebc4fc9-8882-43c6-8713-1ee995a8ee00\") " pod="openshift-ingress/e2e-worker-w4-203c35-openshift-default-56497cdcc-z59vn" Nov 04 02:51:49.000401 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:51:49.000289 4405 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-br6lx\" (UniqueName: \"kubernetes.io/projected/5ebc4fc9-8882-43c6-8713-1ee995a8ee00-kube-api-access-br6lx\") pod \"e2e-worker-w4-203c35-openshift-default-56497cdcc-z59vn\" (UID: \"5ebc4fc9-8882-43c6-8713-1ee995a8ee00\") " pod="openshift-ingress/e2e-worker-w4-203c35-openshift-default-56497cdcc-z59vn" Nov 04 02:51:49.063478 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:51:49.063390 4405 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-203c35-openshift-default-56497cdcc-z59vn" Nov 04 02:51:49.267077 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:51:49.267050 4405 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-e516ff-openshift-default-6f4b957485-xg2tv"] Nov 04 02:51:49.268275 ip-10-0-134-125 kubenswrapper[4405]: W1104 02:51:49.268249 4405 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda4bcafa2_ec97_486c_819f_7be10e3110a9.slice/crio-7a3008f5730c1ff77eeecdc2011d555af6c751923a88f499e69b2bcfef0b9751 WatchSource:0}: Error finding container 7a3008f5730c1ff77eeecdc2011d555af6c751923a88f499e69b2bcfef0b9751: Status 404 returned error can't find the container with id 7a3008f5730c1ff77eeecdc2011d555af6c751923a88f499e69b2bcfef0b9751 Nov 04 02:51:49.270605 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:51:49.270575 4405 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 04 02:51:49.270698 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:51:49.270639 4405 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 04 02:51:49.270698 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:51:49.270685 4405 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 04 02:51:49.437970 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:51:49.437927 4405 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-e516ff-openshift-default-6f4b957485-xg2tv" event={"ID":"a4bcafa2-ec97-486c-819f-7be10e3110a9","Type":"ContainerStarted","Data":"5ffa0fcee1b9de3f3ef3bb807b85ed515c57d72a9041a34d52c80bf8ce43b404"} Nov 04 02:51:49.437970 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:51:49.437975 4405 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-e516ff-openshift-default-6f4b957485-xg2tv" event={"ID":"a4bcafa2-ec97-486c-819f-7be10e3110a9","Type":"ContainerStarted","Data":"7a3008f5730c1ff77eeecdc2011d555af6c751923a88f499e69b2bcfef0b9751"} Nov 04 02:51:49.445148 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:51:49.445119 4405 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-203c35-openshift-default-56497cdcc-z59vn"] Nov 04 02:51:49.446846 ip-10-0-134-125 kubenswrapper[4405]: W1104 02:51:49.446819 4405 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5ebc4fc9_8882_43c6_8713_1ee995a8ee00.slice/crio-2b0362fac9ac89e44165fc0ad4de04a0f5171d85b3479a0e693e3c2a202cb580 WatchSource:0}: Error finding container 2b0362fac9ac89e44165fc0ad4de04a0f5171d85b3479a0e693e3c2a202cb580: Status 404 returned error can't find the container with id 2b0362fac9ac89e44165fc0ad4de04a0f5171d85b3479a0e693e3c2a202cb580 Nov 04 02:51:49.449293 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:51:49.449259 4405 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 04 02:51:49.449402 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:51:49.449338 4405 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 04 02:51:49.449465 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:51:49.449401 4405 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 04 02:51:49.558426 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:51:49.558373 4405 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-e516ff-openshift-default-6f4b957485-xg2tv" podStartSLOduration=1.558354115 podStartE2EDuration="1.558354115s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 02:51:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 02:51:49.557524471 +0000 UTC m=+1155.988224589" watchObservedRunningTime="2026-11-04 02:51:49.558354115 +0000 UTC m=+1155.989054236" Nov 04 02:51:49.830472 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:51:49.830438 4405 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-e516ff-openshift-default-6f4b957485-xg2tv" Nov 04 02:51:50.455736 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:51:50.455702 4405 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-203c35-openshift-default-56497cdcc-z59vn" event={"ID":"5ebc4fc9-8882-43c6-8713-1ee995a8ee00","Type":"ContainerStarted","Data":"ffba5bc76b8a20d2d74d2d5516d95428cd0e26f7e0bfddb238c77058347ceb52"} Nov 04 02:51:50.455916 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:51:50.455759 4405 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-203c35-openshift-default-56497cdcc-z59vn" event={"ID":"5ebc4fc9-8882-43c6-8713-1ee995a8ee00","Type":"ContainerStarted","Data":"2b0362fac9ac89e44165fc0ad4de04a0f5171d85b3479a0e693e3c2a202cb580"} Nov 04 02:51:50.633694 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:51:50.626559 4405 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-203c35-openshift-default-56497cdcc-z59vn" podStartSLOduration=2.626537977 podStartE2EDuration="2.626537977s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 02:51:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 02:51:50.625240048 +0000 UTC m=+1157.055940170" watchObservedRunningTime="2026-11-04 02:51:50.626537977 +0000 UTC m=+1157.057238098" Nov 04 02:51:50.842011 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:51:50.841978 4405 patch_prober.go:29] interesting pod/e2e-worker-w1-e516ff-openshift-default-6f4b957485-xg2tv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.60:15021/healthz/ready\": dial tcp 10.134.0.60:15021: connect: connection refused" start-of-body= Nov 04 02:51:50.842278 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:51:50.842253 4405 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-e516ff-openshift-default-6f4b957485-xg2tv" podUID="a4bcafa2-ec97-486c-819f-7be10e3110a9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.60:15021/healthz/ready\": dial tcp 10.134.0.60:15021: connect: connection refused" Nov 04 02:51:51.064052 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:51:51.064014 4405 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-203c35-openshift-default-56497cdcc-z59vn" Nov 04 02:51:51.065859 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:51:51.065833 4405 patch_prober.go:29] interesting pod/e2e-worker-w4-203c35-openshift-default-56497cdcc-z59vn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused" start-of-body= Nov 04 02:51:51.065997 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:51:51.065878 4405 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-203c35-openshift-default-56497cdcc-z59vn" podUID="5ebc4fc9-8882-43c6-8713-1ee995a8ee00" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused" Nov 04 02:51:51.846075 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:51:51.846039 4405 patch_prober.go:29] interesting pod/e2e-worker-w1-e516ff-openshift-default-6f4b957485-xg2tv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.60:15021/healthz/ready\": dial tcp 10.134.0.60:15021: connect: connection refused" start-of-body= Nov 04 02:51:51.846711 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:51:51.846650 4405 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-e516ff-openshift-default-6f4b957485-xg2tv" podUID="a4bcafa2-ec97-486c-819f-7be10e3110a9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.60:15021/healthz/ready\": dial tcp 10.134.0.60:15021: connect: connection refused" Nov 04 02:51:52.063977 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:51:52.063936 4405 patch_prober.go:29] interesting pod/e2e-worker-w4-203c35-openshift-default-56497cdcc-z59vn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused" start-of-body= Nov 04 02:51:52.064177 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:51:52.063997 4405 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-203c35-openshift-default-56497cdcc-z59vn" podUID="5ebc4fc9-8882-43c6-8713-1ee995a8ee00" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused" Nov 04 02:51:52.836694 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:51:52.831047 4405 patch_prober.go:29] interesting pod/e2e-worker-w1-e516ff-openshift-default-6f4b957485-xg2tv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.60:15021/healthz/ready\": dial tcp 10.134.0.60:15021: connect: connection refused" start-of-body= Nov 04 02:51:52.836694 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:51:52.831110 4405 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-e516ff-openshift-default-6f4b957485-xg2tv" podUID="a4bcafa2-ec97-486c-819f-7be10e3110a9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.60:15021/healthz/ready\": dial tcp 10.134.0.60:15021: connect: connection refused" Nov 04 02:51:53.064329 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:51:53.064283 4405 patch_prober.go:29] interesting pod/e2e-worker-w4-203c35-openshift-default-56497cdcc-z59vn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused" start-of-body= Nov 04 02:51:53.064714 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:51:53.064358 4405 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-203c35-openshift-default-56497cdcc-z59vn" podUID="5ebc4fc9-8882-43c6-8713-1ee995a8ee00" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused" Nov 04 02:51:53.831695 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:51:53.831101 4405 patch_prober.go:29] interesting pod/e2e-worker-w1-e516ff-openshift-default-6f4b957485-xg2tv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.60:15021/healthz/ready\": dial tcp 10.134.0.60:15021: connect: connection refused" start-of-body= Nov 04 02:51:53.831695 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:51:53.831179 4405 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-e516ff-openshift-default-6f4b957485-xg2tv" podUID="a4bcafa2-ec97-486c-819f-7be10e3110a9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.60:15021/healthz/ready\": dial tcp 10.134.0.60:15021: connect: connection refused" Nov 04 02:51:54.064377 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:51:54.064342 4405 patch_prober.go:29] interesting pod/e2e-worker-w4-203c35-openshift-default-56497cdcc-z59vn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused" start-of-body= Nov 04 02:51:54.064791 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:51:54.064415 4405 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-203c35-openshift-default-56497cdcc-z59vn" podUID="5ebc4fc9-8882-43c6-8713-1ee995a8ee00" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused" Nov 04 02:51:54.831192 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:51:54.831142 4405 patch_prober.go:29] interesting pod/e2e-worker-w1-e516ff-openshift-default-6f4b957485-xg2tv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.60:15021/healthz/ready\": dial tcp 10.134.0.60:15021: connect: connection refused" start-of-body= Nov 04 02:51:54.831564 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:51:54.831528 4405 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-e516ff-openshift-default-6f4b957485-xg2tv" podUID="a4bcafa2-ec97-486c-819f-7be10e3110a9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.60:15021/healthz/ready\": dial tcp 10.134.0.60:15021: connect: connection refused" Nov 04 02:51:55.064820 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:51:55.064431 4405 patch_prober.go:29] interesting pod/e2e-worker-w4-203c35-openshift-default-56497cdcc-z59vn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused" start-of-body= Nov 04 02:51:55.064820 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:51:55.064524 4405 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-203c35-openshift-default-56497cdcc-z59vn" podUID="5ebc4fc9-8882-43c6-8713-1ee995a8ee00" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused" Nov 04 02:51:55.830963 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:51:55.830917 4405 patch_prober.go:29] interesting pod/e2e-worker-w1-e516ff-openshift-default-6f4b957485-xg2tv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.60:15021/healthz/ready\": dial tcp 10.134.0.60:15021: connect: connection refused" start-of-body= Nov 04 02:51:55.831144 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:51:55.830990 4405 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-e516ff-openshift-default-6f4b957485-xg2tv" podUID="a4bcafa2-ec97-486c-819f-7be10e3110a9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.60:15021/healthz/ready\": dial tcp 10.134.0.60:15021: connect: connection refused" Nov 04 02:51:56.064466 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:51:56.064415 4405 patch_prober.go:29] interesting pod/e2e-worker-w4-203c35-openshift-default-56497cdcc-z59vn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused" start-of-body= Nov 04 02:51:56.064709 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:51:56.064488 4405 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-203c35-openshift-default-56497cdcc-z59vn" podUID="5ebc4fc9-8882-43c6-8713-1ee995a8ee00" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused" Nov 04 02:51:56.836522 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:51:56.836487 4405 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-e516ff-openshift-default-6f4b957485-xg2tv" Nov 04 02:51:56.837107 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:51:56.836947 4405 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-e516ff-openshift-default-6f4b957485-xg2tv" Nov 04 02:51:56.838226 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:51:56.838206 4405 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-e516ff-openshift-default-6f4b957485-xg2tv" Nov 04 02:51:57.151921 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:51:57.151834 4405 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-203c35-openshift-default-56497cdcc-z59vn" Nov 04 02:51:57.152791 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:51:57.152653 4405 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-203c35-openshift-default-56497cdcc-z59vn" Nov 04 02:51:57.154294 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:51:57.154275 4405 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-203c35-openshift-default-56497cdcc-z59vn" Nov 04 02:52:03.221434 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:52:03.221393 4405 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-fe4e0c-openshift-default-c7f4fc84c-pnfhp"] Nov 04 02:52:03.225523 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:52:03.225502 4405 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-fe4e0c-openshift-default-c7f4fc84c-pnfhp" Nov 04 02:52:03.270125 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:52:03.270092 4405 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-fe4e0c-openshift-default-c7f4fc84c-mmngc"] Nov 04 02:52:03.273344 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:52:03.273315 4405 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/86eff01f-03b3-4d4c-a2a3-f6798ffe5a38-istio-data\") pod \"e2e-worker-w0-fe4e0c-openshift-default-c7f4fc84c-pnfhp\" (UID: \"86eff01f-03b3-4d4c-a2a3-f6798ffe5a38\") " pod="openshift-ingress/e2e-worker-w0-fe4e0c-openshift-default-c7f4fc84c-pnfhp" Nov 04 02:52:03.273489 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:52:03.273469 4405 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/86eff01f-03b3-4d4c-a2a3-f6798ffe5a38-credential-socket\") pod \"e2e-worker-w0-fe4e0c-openshift-default-c7f4fc84c-pnfhp\" (UID: \"86eff01f-03b3-4d4c-a2a3-f6798ffe5a38\") " pod="openshift-ingress/e2e-worker-w0-fe4e0c-openshift-default-c7f4fc84c-pnfhp" Nov 04 02:52:03.273547 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:52:03.273499 4405 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/86eff01f-03b3-4d4c-a2a3-f6798ffe5a38-istio-podinfo\") pod \"e2e-worker-w0-fe4e0c-openshift-default-c7f4fc84c-pnfhp\" (UID: \"86eff01f-03b3-4d4c-a2a3-f6798ffe5a38\") " pod="openshift-ingress/e2e-worker-w0-fe4e0c-openshift-default-c7f4fc84c-pnfhp" Nov 04 02:52:03.273547 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:52:03.273518 4405 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tjvdt\" (UniqueName: \"kubernetes.io/projected/86eff01f-03b3-4d4c-a2a3-f6798ffe5a38-kube-api-access-tjvdt\") pod \"e2e-worker-w0-fe4e0c-openshift-default-c7f4fc84c-pnfhp\" (UID: \"86eff01f-03b3-4d4c-a2a3-f6798ffe5a38\") " pod="openshift-ingress/e2e-worker-w0-fe4e0c-openshift-default-c7f4fc84c-pnfhp" Nov 04 02:52:03.273547 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:52:03.273539 4405 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/86eff01f-03b3-4d4c-a2a3-f6798ffe5a38-workload-certs\") pod \"e2e-worker-w0-fe4e0c-openshift-default-c7f4fc84c-pnfhp\" (UID: \"86eff01f-03b3-4d4c-a2a3-f6798ffe5a38\") " pod="openshift-ingress/e2e-worker-w0-fe4e0c-openshift-default-c7f4fc84c-pnfhp" Nov 04 02:52:03.273687 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:52:03.273554 4405 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/86eff01f-03b3-4d4c-a2a3-f6798ffe5a38-istio-token\") pod \"e2e-worker-w0-fe4e0c-openshift-default-c7f4fc84c-pnfhp\" (UID: \"86eff01f-03b3-4d4c-a2a3-f6798ffe5a38\") " pod="openshift-ingress/e2e-worker-w0-fe4e0c-openshift-default-c7f4fc84c-pnfhp" Nov 04 02:52:03.273742 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:52:03.273723 4405 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/86eff01f-03b3-4d4c-a2a3-f6798ffe5a38-istiod-ca-cert\") pod \"e2e-worker-w0-fe4e0c-openshift-default-c7f4fc84c-pnfhp\" (UID: \"86eff01f-03b3-4d4c-a2a3-f6798ffe5a38\") " pod="openshift-ingress/e2e-worker-w0-fe4e0c-openshift-default-c7f4fc84c-pnfhp" Nov 04 02:52:03.273840 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:52:03.273827 4405 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/86eff01f-03b3-4d4c-a2a3-f6798ffe5a38-istio-envoy\") pod \"e2e-worker-w0-fe4e0c-openshift-default-c7f4fc84c-pnfhp\" (UID: \"86eff01f-03b3-4d4c-a2a3-f6798ffe5a38\") " pod="openshift-ingress/e2e-worker-w0-fe4e0c-openshift-default-c7f4fc84c-pnfhp" Nov 04 02:52:03.273892 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:52:03.273859 4405 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/86eff01f-03b3-4d4c-a2a3-f6798ffe5a38-workload-socket\") pod \"e2e-worker-w0-fe4e0c-openshift-default-c7f4fc84c-pnfhp\" (UID: \"86eff01f-03b3-4d4c-a2a3-f6798ffe5a38\") " pod="openshift-ingress/e2e-worker-w0-fe4e0c-openshift-default-c7f4fc84c-pnfhp" Nov 04 02:52:03.274139 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:52:03.274125 4405 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-fe4e0c-openshift-default-c7f4fc84c-mmngc" Nov 04 02:52:03.322606 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:52:03.322572 4405 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-fe4e0c-openshift-default-c7f4fc84c-pnfhp"] Nov 04 02:52:03.333473 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:52:03.333443 4405 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-fe4e0c-openshift-default-c7f4fc84c-mmngc"] Nov 04 02:52:03.375387 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:52:03.375351 4405 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8d37c0a6-e58b-4f0b-a043-6184b537a719-credential-socket\") pod \"e2e-worker-w0-fe4e0c-openshift-default-c7f4fc84c-mmngc\" (UID: \"8d37c0a6-e58b-4f0b-a043-6184b537a719\") " pod="openshift-ingress/e2e-worker-w0-fe4e0c-openshift-default-c7f4fc84c-mmngc" Nov 04 02:52:03.375548 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:52:03.375398 4405 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/86eff01f-03b3-4d4c-a2a3-f6798ffe5a38-workload-socket\") pod \"e2e-worker-w0-fe4e0c-openshift-default-c7f4fc84c-pnfhp\" (UID: \"86eff01f-03b3-4d4c-a2a3-f6798ffe5a38\") " pod="openshift-ingress/e2e-worker-w0-fe4e0c-openshift-default-c7f4fc84c-pnfhp" Nov 04 02:52:03.375548 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:52:03.375511 4405 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/86eff01f-03b3-4d4c-a2a3-f6798ffe5a38-istio-data\") pod \"e2e-worker-w0-fe4e0c-openshift-default-c7f4fc84c-pnfhp\" (UID: \"86eff01f-03b3-4d4c-a2a3-f6798ffe5a38\") " pod="openshift-ingress/e2e-worker-w0-fe4e0c-openshift-default-c7f4fc84c-pnfhp" Nov 04 02:52:03.375637 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:52:03.375586 4405 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8d37c0a6-e58b-4f0b-a043-6184b537a719-istio-token\") pod \"e2e-worker-w0-fe4e0c-openshift-default-c7f4fc84c-mmngc\" (UID: \"8d37c0a6-e58b-4f0b-a043-6184b537a719\") " pod="openshift-ingress/e2e-worker-w0-fe4e0c-openshift-default-c7f4fc84c-mmngc" Nov 04 02:52:03.375637 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:52:03.375614 4405 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8d37c0a6-e58b-4f0b-a043-6184b537a719-istio-envoy\") pod \"e2e-worker-w0-fe4e0c-openshift-default-c7f4fc84c-mmngc\" (UID: \"8d37c0a6-e58b-4f0b-a043-6184b537a719\") " pod="openshift-ingress/e2e-worker-w0-fe4e0c-openshift-default-c7f4fc84c-mmngc" Nov 04 02:52:03.375781 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:52:03.375762 4405 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/86eff01f-03b3-4d4c-a2a3-f6798ffe5a38-workload-socket\") pod \"e2e-worker-w0-fe4e0c-openshift-default-c7f4fc84c-pnfhp\" (UID: \"86eff01f-03b3-4d4c-a2a3-f6798ffe5a38\") " pod="openshift-ingress/e2e-worker-w0-fe4e0c-openshift-default-c7f4fc84c-pnfhp" Nov 04 02:52:03.375833 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:52:03.375790 4405 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/86eff01f-03b3-4d4c-a2a3-f6798ffe5a38-credential-socket\") pod \"e2e-worker-w0-fe4e0c-openshift-default-c7f4fc84c-pnfhp\" (UID: \"86eff01f-03b3-4d4c-a2a3-f6798ffe5a38\") " pod="openshift-ingress/e2e-worker-w0-fe4e0c-openshift-default-c7f4fc84c-pnfhp" Nov 04 02:52:03.375833 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:52:03.375822 4405 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/86eff01f-03b3-4d4c-a2a3-f6798ffe5a38-istio-data\") pod \"e2e-worker-w0-fe4e0c-openshift-default-c7f4fc84c-pnfhp\" (UID: \"86eff01f-03b3-4d4c-a2a3-f6798ffe5a38\") " pod="openshift-ingress/e2e-worker-w0-fe4e0c-openshift-default-c7f4fc84c-pnfhp" Nov 04 02:52:03.375934 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:52:03.375828 4405 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/86eff01f-03b3-4d4c-a2a3-f6798ffe5a38-istio-podinfo\") pod \"e2e-worker-w0-fe4e0c-openshift-default-c7f4fc84c-pnfhp\" (UID: \"86eff01f-03b3-4d4c-a2a3-f6798ffe5a38\") " pod="openshift-ingress/e2e-worker-w0-fe4e0c-openshift-default-c7f4fc84c-pnfhp" Nov 04 02:52:03.375934 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:52:03.375870 4405 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tjvdt\" (UniqueName: \"kubernetes.io/projected/86eff01f-03b3-4d4c-a2a3-f6798ffe5a38-kube-api-access-tjvdt\") pod \"e2e-worker-w0-fe4e0c-openshift-default-c7f4fc84c-pnfhp\" (UID: \"86eff01f-03b3-4d4c-a2a3-f6798ffe5a38\") " pod="openshift-ingress/e2e-worker-w0-fe4e0c-openshift-default-c7f4fc84c-pnfhp" Nov 04 02:52:03.375934 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:52:03.375896 4405 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/86eff01f-03b3-4d4c-a2a3-f6798ffe5a38-workload-certs\") pod \"e2e-worker-w0-fe4e0c-openshift-default-c7f4fc84c-pnfhp\" (UID: \"86eff01f-03b3-4d4c-a2a3-f6798ffe5a38\") " pod="openshift-ingress/e2e-worker-w0-fe4e0c-openshift-default-c7f4fc84c-pnfhp" Nov 04 02:52:03.375934 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:52:03.375923 4405 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/86eff01f-03b3-4d4c-a2a3-f6798ffe5a38-istio-token\") pod \"e2e-worker-w0-fe4e0c-openshift-default-c7f4fc84c-pnfhp\" (UID: \"86eff01f-03b3-4d4c-a2a3-f6798ffe5a38\") " pod="openshift-ingress/e2e-worker-w0-fe4e0c-openshift-default-c7f4fc84c-pnfhp" Nov 04 02:52:03.376125 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:52:03.375942 4405 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8d37c0a6-e58b-4f0b-a043-6184b537a719-istio-podinfo\") pod \"e2e-worker-w0-fe4e0c-openshift-default-c7f4fc84c-mmngc\" (UID: \"8d37c0a6-e58b-4f0b-a043-6184b537a719\") " pod="openshift-ingress/e2e-worker-w0-fe4e0c-openshift-default-c7f4fc84c-mmngc" Nov 04 02:52:03.376125 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:52:03.375967 4405 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8d37c0a6-e58b-4f0b-a043-6184b537a719-istio-data\") pod \"e2e-worker-w0-fe4e0c-openshift-default-c7f4fc84c-mmngc\" (UID: \"8d37c0a6-e58b-4f0b-a043-6184b537a719\") " pod="openshift-ingress/e2e-worker-w0-fe4e0c-openshift-default-c7f4fc84c-mmngc" Nov 04 02:52:03.376125 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:52:03.376007 4405 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8d37c0a6-e58b-4f0b-a043-6184b537a719-istiod-ca-cert\") pod \"e2e-worker-w0-fe4e0c-openshift-default-c7f4fc84c-mmngc\" (UID: \"8d37c0a6-e58b-4f0b-a043-6184b537a719\") " pod="openshift-ingress/e2e-worker-w0-fe4e0c-openshift-default-c7f4fc84c-mmngc" Nov 04 02:52:03.376125 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:52:03.376061 4405 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/86eff01f-03b3-4d4c-a2a3-f6798ffe5a38-credential-socket\") pod \"e2e-worker-w0-fe4e0c-openshift-default-c7f4fc84c-pnfhp\" (UID: \"86eff01f-03b3-4d4c-a2a3-f6798ffe5a38\") " pod="openshift-ingress/e2e-worker-w0-fe4e0c-openshift-default-c7f4fc84c-pnfhp" Nov 04 02:52:03.376331 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:52:03.376145 4405 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/86eff01f-03b3-4d4c-a2a3-f6798ffe5a38-istiod-ca-cert\") pod \"e2e-worker-w0-fe4e0c-openshift-default-c7f4fc84c-pnfhp\" (UID: \"86eff01f-03b3-4d4c-a2a3-f6798ffe5a38\") " pod="openshift-ingress/e2e-worker-w0-fe4e0c-openshift-default-c7f4fc84c-pnfhp" Nov 04 02:52:03.376331 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:52:03.376176 4405 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/86eff01f-03b3-4d4c-a2a3-f6798ffe5a38-workload-certs\") pod \"e2e-worker-w0-fe4e0c-openshift-default-c7f4fc84c-pnfhp\" (UID: \"86eff01f-03b3-4d4c-a2a3-f6798ffe5a38\") " pod="openshift-ingress/e2e-worker-w0-fe4e0c-openshift-default-c7f4fc84c-pnfhp" Nov 04 02:52:03.376331 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:52:03.376205 4405 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8d37c0a6-e58b-4f0b-a043-6184b537a719-workload-certs\") pod \"e2e-worker-w0-fe4e0c-openshift-default-c7f4fc84c-mmngc\" (UID: \"8d37c0a6-e58b-4f0b-a043-6184b537a719\") " pod="openshift-ingress/e2e-worker-w0-fe4e0c-openshift-default-c7f4fc84c-mmngc" Nov 04 02:52:03.376331 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:52:03.376240 4405 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8d37c0a6-e58b-4f0b-a043-6184b537a719-workload-socket\") pod \"e2e-worker-w0-fe4e0c-openshift-default-c7f4fc84c-mmngc\" (UID: \"8d37c0a6-e58b-4f0b-a043-6184b537a719\") " pod="openshift-ingress/e2e-worker-w0-fe4e0c-openshift-default-c7f4fc84c-mmngc" Nov 04 02:52:03.376495 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:52:03.376336 4405 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7vvw8\" (UniqueName: \"kubernetes.io/projected/8d37c0a6-e58b-4f0b-a043-6184b537a719-kube-api-access-7vvw8\") pod \"e2e-worker-w0-fe4e0c-openshift-default-c7f4fc84c-mmngc\" (UID: \"8d37c0a6-e58b-4f0b-a043-6184b537a719\") " pod="openshift-ingress/e2e-worker-w0-fe4e0c-openshift-default-c7f4fc84c-mmngc" Nov 04 02:52:03.376495 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:52:03.376429 4405 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/86eff01f-03b3-4d4c-a2a3-f6798ffe5a38-istio-envoy\") pod \"e2e-worker-w0-fe4e0c-openshift-default-c7f4fc84c-pnfhp\" (UID: \"86eff01f-03b3-4d4c-a2a3-f6798ffe5a38\") " pod="openshift-ingress/e2e-worker-w0-fe4e0c-openshift-default-c7f4fc84c-pnfhp" Nov 04 02:52:03.376852 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:52:03.376826 4405 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/86eff01f-03b3-4d4c-a2a3-f6798ffe5a38-istiod-ca-cert\") pod \"e2e-worker-w0-fe4e0c-openshift-default-c7f4fc84c-pnfhp\" (UID: \"86eff01f-03b3-4d4c-a2a3-f6798ffe5a38\") " pod="openshift-ingress/e2e-worker-w0-fe4e0c-openshift-default-c7f4fc84c-pnfhp" Nov 04 02:52:03.378707 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:52:03.378655 4405 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/86eff01f-03b3-4d4c-a2a3-f6798ffe5a38-istio-envoy\") pod \"e2e-worker-w0-fe4e0c-openshift-default-c7f4fc84c-pnfhp\" (UID: \"86eff01f-03b3-4d4c-a2a3-f6798ffe5a38\") " pod="openshift-ingress/e2e-worker-w0-fe4e0c-openshift-default-c7f4fc84c-pnfhp" Nov 04 02:52:03.378778 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:52:03.378659 4405 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/86eff01f-03b3-4d4c-a2a3-f6798ffe5a38-istio-podinfo\") pod \"e2e-worker-w0-fe4e0c-openshift-default-c7f4fc84c-pnfhp\" (UID: \"86eff01f-03b3-4d4c-a2a3-f6798ffe5a38\") " pod="openshift-ingress/e2e-worker-w0-fe4e0c-openshift-default-c7f4fc84c-pnfhp" Nov 04 02:52:03.437108 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:52:03.437071 4405 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/86eff01f-03b3-4d4c-a2a3-f6798ffe5a38-istio-token\") pod \"e2e-worker-w0-fe4e0c-openshift-default-c7f4fc84c-pnfhp\" (UID: \"86eff01f-03b3-4d4c-a2a3-f6798ffe5a38\") " pod="openshift-ingress/e2e-worker-w0-fe4e0c-openshift-default-c7f4fc84c-pnfhp" Nov 04 02:52:03.437265 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:52:03.437153 4405 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tjvdt\" (UniqueName: \"kubernetes.io/projected/86eff01f-03b3-4d4c-a2a3-f6798ffe5a38-kube-api-access-tjvdt\") pod \"e2e-worker-w0-fe4e0c-openshift-default-c7f4fc84c-pnfhp\" (UID: \"86eff01f-03b3-4d4c-a2a3-f6798ffe5a38\") " pod="openshift-ingress/e2e-worker-w0-fe4e0c-openshift-default-c7f4fc84c-pnfhp" Nov 04 02:52:03.478182 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:52:03.478105 4405 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8d37c0a6-e58b-4f0b-a043-6184b537a719-istio-token\") pod \"e2e-worker-w0-fe4e0c-openshift-default-c7f4fc84c-mmngc\" (UID: \"8d37c0a6-e58b-4f0b-a043-6184b537a719\") " pod="openshift-ingress/e2e-worker-w0-fe4e0c-openshift-default-c7f4fc84c-mmngc" Nov 04 02:52:03.478182 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:52:03.478146 4405 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8d37c0a6-e58b-4f0b-a043-6184b537a719-istio-envoy\") pod \"e2e-worker-w0-fe4e0c-openshift-default-c7f4fc84c-mmngc\" (UID: \"8d37c0a6-e58b-4f0b-a043-6184b537a719\") " pod="openshift-ingress/e2e-worker-w0-fe4e0c-openshift-default-c7f4fc84c-mmngc" Nov 04 02:52:03.478403 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:52:03.478218 4405 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8d37c0a6-e58b-4f0b-a043-6184b537a719-istio-podinfo\") pod \"e2e-worker-w0-fe4e0c-openshift-default-c7f4fc84c-mmngc\" (UID: \"8d37c0a6-e58b-4f0b-a043-6184b537a719\") " pod="openshift-ingress/e2e-worker-w0-fe4e0c-openshift-default-c7f4fc84c-mmngc" Nov 04 02:52:03.478403 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:52:03.478241 4405 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8d37c0a6-e58b-4f0b-a043-6184b537a719-istio-data\") pod \"e2e-worker-w0-fe4e0c-openshift-default-c7f4fc84c-mmngc\" (UID: \"8d37c0a6-e58b-4f0b-a043-6184b537a719\") " pod="openshift-ingress/e2e-worker-w0-fe4e0c-openshift-default-c7f4fc84c-mmngc" Nov 04 02:52:03.478403 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:52:03.478258 4405 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8d37c0a6-e58b-4f0b-a043-6184b537a719-istiod-ca-cert\") pod \"e2e-worker-w0-fe4e0c-openshift-default-c7f4fc84c-mmngc\" (UID: \"8d37c0a6-e58b-4f0b-a043-6184b537a719\") " pod="openshift-ingress/e2e-worker-w0-fe4e0c-openshift-default-c7f4fc84c-mmngc" Nov 04 02:52:03.478403 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:52:03.478307 4405 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8d37c0a6-e58b-4f0b-a043-6184b537a719-workload-certs\") pod \"e2e-worker-w0-fe4e0c-openshift-default-c7f4fc84c-mmngc\" (UID: \"8d37c0a6-e58b-4f0b-a043-6184b537a719\") " pod="openshift-ingress/e2e-worker-w0-fe4e0c-openshift-default-c7f4fc84c-mmngc" Nov 04 02:52:03.478403 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:52:03.478326 4405 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8d37c0a6-e58b-4f0b-a043-6184b537a719-workload-socket\") pod \"e2e-worker-w0-fe4e0c-openshift-default-c7f4fc84c-mmngc\" (UID: \"8d37c0a6-e58b-4f0b-a043-6184b537a719\") " pod="openshift-ingress/e2e-worker-w0-fe4e0c-openshift-default-c7f4fc84c-mmngc" Nov 04 02:52:03.478403 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:52:03.478378 4405 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7vvw8\" (UniqueName: \"kubernetes.io/projected/8d37c0a6-e58b-4f0b-a043-6184b537a719-kube-api-access-7vvw8\") pod \"e2e-worker-w0-fe4e0c-openshift-default-c7f4fc84c-mmngc\" (UID: \"8d37c0a6-e58b-4f0b-a043-6184b537a719\") " pod="openshift-ingress/e2e-worker-w0-fe4e0c-openshift-default-c7f4fc84c-mmngc" Nov 04 02:52:03.478717 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:52:03.478462 4405 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8d37c0a6-e58b-4f0b-a043-6184b537a719-credential-socket\") pod \"e2e-worker-w0-fe4e0c-openshift-default-c7f4fc84c-mmngc\" (UID: \"8d37c0a6-e58b-4f0b-a043-6184b537a719\") " pod="openshift-ingress/e2e-worker-w0-fe4e0c-openshift-default-c7f4fc84c-mmngc" Nov 04 02:52:03.478787 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:52:03.478740 4405 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8d37c0a6-e58b-4f0b-a043-6184b537a719-istio-data\") pod \"e2e-worker-w0-fe4e0c-openshift-default-c7f4fc84c-mmngc\" (UID: \"8d37c0a6-e58b-4f0b-a043-6184b537a719\") " pod="openshift-ingress/e2e-worker-w0-fe4e0c-openshift-default-c7f4fc84c-mmngc" Nov 04 02:52:03.478877 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:52:03.478845 4405 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8d37c0a6-e58b-4f0b-a043-6184b537a719-workload-certs\") pod \"e2e-worker-w0-fe4e0c-openshift-default-c7f4fc84c-mmngc\" (UID: \"8d37c0a6-e58b-4f0b-a043-6184b537a719\") " pod="openshift-ingress/e2e-worker-w0-fe4e0c-openshift-default-c7f4fc84c-mmngc" Nov 04 02:52:03.478969 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:52:03.478943 4405 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8d37c0a6-e58b-4f0b-a043-6184b537a719-workload-socket\") pod \"e2e-worker-w0-fe4e0c-openshift-default-c7f4fc84c-mmngc\" (UID: \"8d37c0a6-e58b-4f0b-a043-6184b537a719\") " pod="openshift-ingress/e2e-worker-w0-fe4e0c-openshift-default-c7f4fc84c-mmngc" Nov 04 02:52:03.479077 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:52:03.479002 4405 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8d37c0a6-e58b-4f0b-a043-6184b537a719-credential-socket\") pod \"e2e-worker-w0-fe4e0c-openshift-default-c7f4fc84c-mmngc\" (UID: \"8d37c0a6-e58b-4f0b-a043-6184b537a719\") " pod="openshift-ingress/e2e-worker-w0-fe4e0c-openshift-default-c7f4fc84c-mmngc" Nov 04 02:52:03.479323 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:52:03.479299 4405 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8d37c0a6-e58b-4f0b-a043-6184b537a719-istiod-ca-cert\") pod \"e2e-worker-w0-fe4e0c-openshift-default-c7f4fc84c-mmngc\" (UID: \"8d37c0a6-e58b-4f0b-a043-6184b537a719\") " pod="openshift-ingress/e2e-worker-w0-fe4e0c-openshift-default-c7f4fc84c-mmngc" Nov 04 02:52:03.480662 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:52:03.480632 4405 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8d37c0a6-e58b-4f0b-a043-6184b537a719-istio-envoy\") pod \"e2e-worker-w0-fe4e0c-openshift-default-c7f4fc84c-mmngc\" (UID: \"8d37c0a6-e58b-4f0b-a043-6184b537a719\") " pod="openshift-ingress/e2e-worker-w0-fe4e0c-openshift-default-c7f4fc84c-mmngc" Nov 04 02:52:03.480979 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:52:03.480963 4405 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8d37c0a6-e58b-4f0b-a043-6184b537a719-istio-podinfo\") pod \"e2e-worker-w0-fe4e0c-openshift-default-c7f4fc84c-mmngc\" (UID: \"8d37c0a6-e58b-4f0b-a043-6184b537a719\") " pod="openshift-ingress/e2e-worker-w0-fe4e0c-openshift-default-c7f4fc84c-mmngc" Nov 04 02:52:03.539182 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:52:03.539151 4405 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8d37c0a6-e58b-4f0b-a043-6184b537a719-istio-token\") pod \"e2e-worker-w0-fe4e0c-openshift-default-c7f4fc84c-mmngc\" (UID: \"8d37c0a6-e58b-4f0b-a043-6184b537a719\") " pod="openshift-ingress/e2e-worker-w0-fe4e0c-openshift-default-c7f4fc84c-mmngc" Nov 04 02:52:03.539464 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:52:03.539434 4405 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7vvw8\" (UniqueName: \"kubernetes.io/projected/8d37c0a6-e58b-4f0b-a043-6184b537a719-kube-api-access-7vvw8\") pod \"e2e-worker-w0-fe4e0c-openshift-default-c7f4fc84c-mmngc\" (UID: \"8d37c0a6-e58b-4f0b-a043-6184b537a719\") " pod="openshift-ingress/e2e-worker-w0-fe4e0c-openshift-default-c7f4fc84c-mmngc" Nov 04 02:52:03.545167 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:52:03.545143 4405 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-fe4e0c-openshift-default-c7f4fc84c-pnfhp" Nov 04 02:52:03.588869 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:52:03.588837 4405 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-fe4e0c-openshift-default-c7f4fc84c-mmngc" Nov 04 02:52:03.978940 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:52:03.978910 4405 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-fe4e0c-openshift-default-c7f4fc84c-pnfhp"] Nov 04 02:52:03.979648 ip-10-0-134-125 kubenswrapper[4405]: W1104 02:52:03.979624 4405 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod86eff01f_03b3_4d4c_a2a3_f6798ffe5a38.slice/crio-7f9c19f1d0ddb3115bcfc74139c976dc70b286dbe4944bc99d157a8bb83032b2 WatchSource:0}: Error finding container 7f9c19f1d0ddb3115bcfc74139c976dc70b286dbe4944bc99d157a8bb83032b2: Status 404 returned error can't find the container with id 7f9c19f1d0ddb3115bcfc74139c976dc70b286dbe4944bc99d157a8bb83032b2 Nov 04 02:52:03.982362 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:52:03.982328 4405 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 04 02:52:03.982495 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:52:03.982408 4405 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 04 02:52:03.982495 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:52:03.982460 4405 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 04 02:52:04.033531 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:52:04.032733 4405 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-fe4e0c-openshift-default-c7f4fc84c-mmngc"] Nov 04 02:52:04.038050 ip-10-0-134-125 kubenswrapper[4405]: W1104 02:52:04.038022 4405 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8d37c0a6_e58b_4f0b_a043_6184b537a719.slice/crio-d6a30fdd7902fe148fa82f2dc6f8840e9196daa265d3c66605c491b6ccd87885 WatchSource:0}: Error finding container d6a30fdd7902fe148fa82f2dc6f8840e9196daa265d3c66605c491b6ccd87885: Status 404 returned error can't find the container with id d6a30fdd7902fe148fa82f2dc6f8840e9196daa265d3c66605c491b6ccd87885 Nov 04 02:52:04.040604 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:52:04.040571 4405 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 04 02:52:04.040727 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:52:04.040695 4405 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 04 02:52:04.040805 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:52:04.040743 4405 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 04 02:52:04.545948 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:52:04.545911 4405 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-fe4e0c-openshift-default-c7f4fc84c-mmngc" event={"ID":"8d37c0a6-e58b-4f0b-a043-6184b537a719","Type":"ContainerStarted","Data":"9d1aa8164135250664e9340506c6a3ecc0d38105e5bb63b6ebb9d6e8de2d0a91"} Nov 04 02:52:04.546464 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:52:04.545976 4405 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-fe4e0c-openshift-default-c7f4fc84c-mmngc" event={"ID":"8d37c0a6-e58b-4f0b-a043-6184b537a719","Type":"ContainerStarted","Data":"d6a30fdd7902fe148fa82f2dc6f8840e9196daa265d3c66605c491b6ccd87885"} Nov 04 02:52:04.547619 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:52:04.547591 4405 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-fe4e0c-openshift-default-c7f4fc84c-pnfhp" event={"ID":"86eff01f-03b3-4d4c-a2a3-f6798ffe5a38","Type":"ContainerStarted","Data":"6f33a3f08deda349abe224c91001482797196beb9470673511e1ac1b3a5db9da"} Nov 04 02:52:04.547766 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:52:04.547633 4405 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-fe4e0c-openshift-default-c7f4fc84c-pnfhp" event={"ID":"86eff01f-03b3-4d4c-a2a3-f6798ffe5a38","Type":"ContainerStarted","Data":"7f9c19f1d0ddb3115bcfc74139c976dc70b286dbe4944bc99d157a8bb83032b2"} Nov 04 02:52:04.589141 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:52:04.589106 4405 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-fe4e0c-openshift-default-c7f4fc84c-mmngc" Nov 04 02:52:04.669540 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:52:04.669488 4405 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-fe4e0c-openshift-default-c7f4fc84c-mmngc" podStartSLOduration=1.669469707 podStartE2EDuration="1.669469707s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 02:52:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 02:52:04.665914279 +0000 UTC m=+1171.096614400" watchObservedRunningTime="2026-11-04 02:52:04.669469707 +0000 UTC m=+1171.100169830" Nov 04 02:52:04.788126 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:52:04.788076 4405 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-fe4e0c-openshift-default-c7f4fc84c-pnfhp" podStartSLOduration=1.788061592 podStartE2EDuration="1.788061592s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 02:52:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 02:52:04.785940255 +0000 UTC m=+1171.216640375" watchObservedRunningTime="2026-11-04 02:52:04.788061592 +0000 UTC m=+1171.218761711" Nov 04 02:52:05.548693 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:52:05.547978 4405 patch_prober.go:29] interesting pod/e2e-worker-w0-fe4e0c-openshift-default-c7f4fc84c-pnfhp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused" start-of-body= Nov 04 02:52:05.548693 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:52:05.548047 4405 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-fe4e0c-openshift-default-c7f4fc84c-pnfhp" podUID="86eff01f-03b3-4d4c-a2a3-f6798ffe5a38" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused" Nov 04 02:52:05.548693 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:52:05.548349 4405 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-fe4e0c-openshift-default-c7f4fc84c-pnfhp" Nov 04 02:52:05.591281 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:52:05.591247 4405 patch_prober.go:29] interesting pod/e2e-worker-w0-fe4e0c-openshift-default-c7f4fc84c-mmngc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.63:15021/healthz/ready\": dial tcp 10.134.0.63:15021: connect: connection refused" start-of-body= Nov 04 02:52:05.591466 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:52:05.591308 4405 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-fe4e0c-openshift-default-c7f4fc84c-mmngc" podUID="8d37c0a6-e58b-4f0b-a043-6184b537a719" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.63:15021/healthz/ready\": dial tcp 10.134.0.63:15021: connect: connection refused" Nov 04 02:52:06.548143 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:52:06.548102 4405 patch_prober.go:29] interesting pod/e2e-worker-w0-fe4e0c-openshift-default-c7f4fc84c-pnfhp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused" start-of-body= Nov 04 02:52:06.548434 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:52:06.548404 4405 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-fe4e0c-openshift-default-c7f4fc84c-pnfhp" podUID="86eff01f-03b3-4d4c-a2a3-f6798ffe5a38" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused" Nov 04 02:52:06.590063 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:52:06.590030 4405 patch_prober.go:29] interesting pod/e2e-worker-w0-fe4e0c-openshift-default-c7f4fc84c-mmngc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.63:15021/healthz/ready\": dial tcp 10.134.0.63:15021: connect: connection refused" start-of-body= Nov 04 02:52:06.590564 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:52:06.590536 4405 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-fe4e0c-openshift-default-c7f4fc84c-mmngc" podUID="8d37c0a6-e58b-4f0b-a043-6184b537a719" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.63:15021/healthz/ready\": dial tcp 10.134.0.63:15021: connect: connection refused" Nov 04 02:52:06.885242 ip-10-0-134-125 kubenswrapper[4405]: E1104 02:52:06.885208 4405 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5f2vb" containerName="extractor" Nov 04 02:52:07.554750 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:52:07.554207 4405 patch_prober.go:29] interesting pod/e2e-worker-w0-fe4e0c-openshift-default-c7f4fc84c-pnfhp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused" start-of-body= Nov 04 02:52:07.554750 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:52:07.554284 4405 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-fe4e0c-openshift-default-c7f4fc84c-pnfhp" podUID="86eff01f-03b3-4d4c-a2a3-f6798ffe5a38" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused" Nov 04 02:52:07.589652 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:52:07.589620 4405 patch_prober.go:29] interesting pod/e2e-worker-w0-fe4e0c-openshift-default-c7f4fc84c-mmngc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.63:15021/healthz/ready\": dial tcp 10.134.0.63:15021: connect: connection refused" start-of-body= Nov 04 02:52:07.589848 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:52:07.589717 4405 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-fe4e0c-openshift-default-c7f4fc84c-mmngc" podUID="8d37c0a6-e58b-4f0b-a043-6184b537a719" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.63:15021/healthz/ready\": dial tcp 10.134.0.63:15021: connect: connection refused" Nov 04 02:52:08.546082 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:52:08.546041 4405 patch_prober.go:29] interesting pod/e2e-worker-w0-fe4e0c-openshift-default-c7f4fc84c-pnfhp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused" start-of-body= Nov 04 02:52:08.546727 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:52:08.546694 4405 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-fe4e0c-openshift-default-c7f4fc84c-pnfhp" podUID="86eff01f-03b3-4d4c-a2a3-f6798ffe5a38" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused" Nov 04 02:52:08.589720 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:52:08.589656 4405 patch_prober.go:29] interesting pod/e2e-worker-w0-fe4e0c-openshift-default-c7f4fc84c-mmngc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.63:15021/healthz/ready\": dial tcp 10.134.0.63:15021: connect: connection refused" start-of-body= Nov 04 02:52:08.589896 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:52:08.589750 4405 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-fe4e0c-openshift-default-c7f4fc84c-mmngc" podUID="8d37c0a6-e58b-4f0b-a043-6184b537a719" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.63:15021/healthz/ready\": dial tcp 10.134.0.63:15021: connect: connection refused" Nov 04 02:52:09.546540 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:52:09.546332 4405 patch_prober.go:29] interesting pod/e2e-worker-w0-fe4e0c-openshift-default-c7f4fc84c-pnfhp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused" start-of-body= Nov 04 02:52:09.546540 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:52:09.546428 4405 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-fe4e0c-openshift-default-c7f4fc84c-pnfhp" podUID="86eff01f-03b3-4d4c-a2a3-f6798ffe5a38" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused" Nov 04 02:52:09.589885 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:52:09.589845 4405 patch_prober.go:29] interesting pod/e2e-worker-w0-fe4e0c-openshift-default-c7f4fc84c-mmngc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.63:15021/healthz/ready\": dial tcp 10.134.0.63:15021: connect: connection refused" start-of-body= Nov 04 02:52:09.590052 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:52:09.589907 4405 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-fe4e0c-openshift-default-c7f4fc84c-mmngc" podUID="8d37c0a6-e58b-4f0b-a043-6184b537a719" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.63:15021/healthz/ready\": dial tcp 10.134.0.63:15021: connect: connection refused" Nov 04 02:52:10.546343 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:52:10.546302 4405 patch_prober.go:29] interesting pod/e2e-worker-w0-fe4e0c-openshift-default-c7f4fc84c-pnfhp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused" start-of-body= Nov 04 02:52:10.546580 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:52:10.546376 4405 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-fe4e0c-openshift-default-c7f4fc84c-pnfhp" podUID="86eff01f-03b3-4d4c-a2a3-f6798ffe5a38" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused" Nov 04 02:52:10.589599 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:52:10.589563 4405 patch_prober.go:29] interesting pod/e2e-worker-w0-fe4e0c-openshift-default-c7f4fc84c-mmngc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.63:15021/healthz/ready\": dial tcp 10.134.0.63:15021: connect: connection refused" start-of-body= Nov 04 02:52:10.589818 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:52:10.589623 4405 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-fe4e0c-openshift-default-c7f4fc84c-mmngc" podUID="8d37c0a6-e58b-4f0b-a043-6184b537a719" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.63:15021/healthz/ready\": dial tcp 10.134.0.63:15021: connect: connection refused" Nov 04 02:52:11.719632 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:52:11.719523 4405 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-fe4e0c-openshift-default-c7f4fc84c-mmngc" Nov 04 02:52:11.722203 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:52:11.722119 4405 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-fe4e0c-openshift-default-c7f4fc84c-mmngc" Nov 04 02:52:11.723314 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:52:11.723292 4405 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-fe4e0c-openshift-default-c7f4fc84c-mmngc" Nov 04 02:52:12.027210 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:52:12.026710 4405 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-fe4e0c-openshift-default-c7f4fc84c-pnfhp" Nov 04 02:52:12.027210 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:52:12.027086 4405 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-fe4e0c-openshift-default-c7f4fc84c-pnfhp" Nov 04 02:52:12.028205 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:52:12.028184 4405 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-fe4e0c-openshift-default-c7f4fc84c-pnfhp" Nov 04 02:52:17.787727 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:52:17.787691 4405 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-a0bcbc-openshift-default-65dc9d9b6c-g52rd"] Nov 04 02:52:17.792141 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:52:17.792122 4405 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-a0bcbc-openshift-default-65dc9d9b6c-g52rd" Nov 04 02:52:17.870951 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:52:17.870918 4405 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e5b1b60a-7429-43fe-afd0-1338430750e7-workload-certs\") pod \"e2e-worker-w2-a0bcbc-openshift-default-65dc9d9b6c-g52rd\" (UID: \"e5b1b60a-7429-43fe-afd0-1338430750e7\") " pod="openshift-ingress/e2e-worker-w2-a0bcbc-openshift-default-65dc9d9b6c-g52rd" Nov 04 02:52:17.871150 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:52:17.870983 4405 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e5b1b60a-7429-43fe-afd0-1338430750e7-istio-podinfo\") pod \"e2e-worker-w2-a0bcbc-openshift-default-65dc9d9b6c-g52rd\" (UID: \"e5b1b60a-7429-43fe-afd0-1338430750e7\") " pod="openshift-ingress/e2e-worker-w2-a0bcbc-openshift-default-65dc9d9b6c-g52rd" Nov 04 02:52:17.871150 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:52:17.871056 4405 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sll9s\" (UniqueName: \"kubernetes.io/projected/e5b1b60a-7429-43fe-afd0-1338430750e7-kube-api-access-sll9s\") pod \"e2e-worker-w2-a0bcbc-openshift-default-65dc9d9b6c-g52rd\" (UID: \"e5b1b60a-7429-43fe-afd0-1338430750e7\") " pod="openshift-ingress/e2e-worker-w2-a0bcbc-openshift-default-65dc9d9b6c-g52rd" Nov 04 02:52:17.871150 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:52:17.871126 4405 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e5b1b60a-7429-43fe-afd0-1338430750e7-istio-envoy\") pod \"e2e-worker-w2-a0bcbc-openshift-default-65dc9d9b6c-g52rd\" (UID: \"e5b1b60a-7429-43fe-afd0-1338430750e7\") " pod="openshift-ingress/e2e-worker-w2-a0bcbc-openshift-default-65dc9d9b6c-g52rd" Nov 04 02:52:17.871287 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:52:17.871190 4405 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e5b1b60a-7429-43fe-afd0-1338430750e7-istio-data\") pod \"e2e-worker-w2-a0bcbc-openshift-default-65dc9d9b6c-g52rd\" (UID: \"e5b1b60a-7429-43fe-afd0-1338430750e7\") " pod="openshift-ingress/e2e-worker-w2-a0bcbc-openshift-default-65dc9d9b6c-g52rd" Nov 04 02:52:17.871326 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:52:17.871286 4405 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e5b1b60a-7429-43fe-afd0-1338430750e7-istiod-ca-cert\") pod \"e2e-worker-w2-a0bcbc-openshift-default-65dc9d9b6c-g52rd\" (UID: \"e5b1b60a-7429-43fe-afd0-1338430750e7\") " pod="openshift-ingress/e2e-worker-w2-a0bcbc-openshift-default-65dc9d9b6c-g52rd" Nov 04 02:52:17.871326 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:52:17.871313 4405 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e5b1b60a-7429-43fe-afd0-1338430750e7-workload-socket\") pod \"e2e-worker-w2-a0bcbc-openshift-default-65dc9d9b6c-g52rd\" (UID: \"e5b1b60a-7429-43fe-afd0-1338430750e7\") " pod="openshift-ingress/e2e-worker-w2-a0bcbc-openshift-default-65dc9d9b6c-g52rd" Nov 04 02:52:17.871410 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:52:17.871330 4405 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e5b1b60a-7429-43fe-afd0-1338430750e7-istio-token\") pod \"e2e-worker-w2-a0bcbc-openshift-default-65dc9d9b6c-g52rd\" (UID: \"e5b1b60a-7429-43fe-afd0-1338430750e7\") " pod="openshift-ingress/e2e-worker-w2-a0bcbc-openshift-default-65dc9d9b6c-g52rd" Nov 04 02:52:17.871445 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:52:17.871427 4405 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e5b1b60a-7429-43fe-afd0-1338430750e7-credential-socket\") pod \"e2e-worker-w2-a0bcbc-openshift-default-65dc9d9b6c-g52rd\" (UID: \"e5b1b60a-7429-43fe-afd0-1338430750e7\") " pod="openshift-ingress/e2e-worker-w2-a0bcbc-openshift-default-65dc9d9b6c-g52rd" Nov 04 02:52:17.972427 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:52:17.972388 4405 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e5b1b60a-7429-43fe-afd0-1338430750e7-credential-socket\") pod \"e2e-worker-w2-a0bcbc-openshift-default-65dc9d9b6c-g52rd\" (UID: \"e5b1b60a-7429-43fe-afd0-1338430750e7\") " pod="openshift-ingress/e2e-worker-w2-a0bcbc-openshift-default-65dc9d9b6c-g52rd" Nov 04 02:52:17.972626 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:52:17.972492 4405 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e5b1b60a-7429-43fe-afd0-1338430750e7-workload-certs\") pod \"e2e-worker-w2-a0bcbc-openshift-default-65dc9d9b6c-g52rd\" (UID: \"e5b1b60a-7429-43fe-afd0-1338430750e7\") " pod="openshift-ingress/e2e-worker-w2-a0bcbc-openshift-default-65dc9d9b6c-g52rd" Nov 04 02:52:17.972626 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:52:17.972538 4405 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e5b1b60a-7429-43fe-afd0-1338430750e7-istio-podinfo\") pod \"e2e-worker-w2-a0bcbc-openshift-default-65dc9d9b6c-g52rd\" (UID: \"e5b1b60a-7429-43fe-afd0-1338430750e7\") " pod="openshift-ingress/e2e-worker-w2-a0bcbc-openshift-default-65dc9d9b6c-g52rd" Nov 04 02:52:17.972626 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:52:17.972555 4405 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sll9s\" (UniqueName: \"kubernetes.io/projected/e5b1b60a-7429-43fe-afd0-1338430750e7-kube-api-access-sll9s\") pod \"e2e-worker-w2-a0bcbc-openshift-default-65dc9d9b6c-g52rd\" (UID: \"e5b1b60a-7429-43fe-afd0-1338430750e7\") " pod="openshift-ingress/e2e-worker-w2-a0bcbc-openshift-default-65dc9d9b6c-g52rd" Nov 04 02:52:17.972626 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:52:17.972605 4405 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e5b1b60a-7429-43fe-afd0-1338430750e7-istio-envoy\") pod \"e2e-worker-w2-a0bcbc-openshift-default-65dc9d9b6c-g52rd\" (UID: \"e5b1b60a-7429-43fe-afd0-1338430750e7\") " pod="openshift-ingress/e2e-worker-w2-a0bcbc-openshift-default-65dc9d9b6c-g52rd" Nov 04 02:52:17.972626 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:52:17.972620 4405 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e5b1b60a-7429-43fe-afd0-1338430750e7-istio-data\") pod \"e2e-worker-w2-a0bcbc-openshift-default-65dc9d9b6c-g52rd\" (UID: \"e5b1b60a-7429-43fe-afd0-1338430750e7\") " pod="openshift-ingress/e2e-worker-w2-a0bcbc-openshift-default-65dc9d9b6c-g52rd" Nov 04 02:52:17.973410 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:52:17.972736 4405 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e5b1b60a-7429-43fe-afd0-1338430750e7-istiod-ca-cert\") pod \"e2e-worker-w2-a0bcbc-openshift-default-65dc9d9b6c-g52rd\" (UID: \"e5b1b60a-7429-43fe-afd0-1338430750e7\") " pod="openshift-ingress/e2e-worker-w2-a0bcbc-openshift-default-65dc9d9b6c-g52rd" Nov 04 02:52:17.973410 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:52:17.972773 4405 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e5b1b60a-7429-43fe-afd0-1338430750e7-workload-socket\") pod \"e2e-worker-w2-a0bcbc-openshift-default-65dc9d9b6c-g52rd\" (UID: \"e5b1b60a-7429-43fe-afd0-1338430750e7\") " pod="openshift-ingress/e2e-worker-w2-a0bcbc-openshift-default-65dc9d9b6c-g52rd" Nov 04 02:52:17.973410 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:52:17.972799 4405 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e5b1b60a-7429-43fe-afd0-1338430750e7-istio-token\") pod \"e2e-worker-w2-a0bcbc-openshift-default-65dc9d9b6c-g52rd\" (UID: \"e5b1b60a-7429-43fe-afd0-1338430750e7\") " pod="openshift-ingress/e2e-worker-w2-a0bcbc-openshift-default-65dc9d9b6c-g52rd" Nov 04 02:52:17.973410 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:52:17.972876 4405 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e5b1b60a-7429-43fe-afd0-1338430750e7-credential-socket\") pod \"e2e-worker-w2-a0bcbc-openshift-default-65dc9d9b6c-g52rd\" (UID: \"e5b1b60a-7429-43fe-afd0-1338430750e7\") " pod="openshift-ingress/e2e-worker-w2-a0bcbc-openshift-default-65dc9d9b6c-g52rd" Nov 04 02:52:17.973410 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:52:17.972924 4405 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e5b1b60a-7429-43fe-afd0-1338430750e7-workload-certs\") pod \"e2e-worker-w2-a0bcbc-openshift-default-65dc9d9b6c-g52rd\" (UID: \"e5b1b60a-7429-43fe-afd0-1338430750e7\") " pod="openshift-ingress/e2e-worker-w2-a0bcbc-openshift-default-65dc9d9b6c-g52rd" Nov 04 02:52:17.973410 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:52:17.973039 4405 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e5b1b60a-7429-43fe-afd0-1338430750e7-istio-data\") pod \"e2e-worker-w2-a0bcbc-openshift-default-65dc9d9b6c-g52rd\" (UID: \"e5b1b60a-7429-43fe-afd0-1338430750e7\") " pod="openshift-ingress/e2e-worker-w2-a0bcbc-openshift-default-65dc9d9b6c-g52rd" Nov 04 02:52:17.973410 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:52:17.973251 4405 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e5b1b60a-7429-43fe-afd0-1338430750e7-workload-socket\") pod \"e2e-worker-w2-a0bcbc-openshift-default-65dc9d9b6c-g52rd\" (UID: \"e5b1b60a-7429-43fe-afd0-1338430750e7\") " pod="openshift-ingress/e2e-worker-w2-a0bcbc-openshift-default-65dc9d9b6c-g52rd" Nov 04 02:52:17.973980 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:52:17.973563 4405 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e5b1b60a-7429-43fe-afd0-1338430750e7-istiod-ca-cert\") pod \"e2e-worker-w2-a0bcbc-openshift-default-65dc9d9b6c-g52rd\" (UID: \"e5b1b60a-7429-43fe-afd0-1338430750e7\") " pod="openshift-ingress/e2e-worker-w2-a0bcbc-openshift-default-65dc9d9b6c-g52rd" Nov 04 02:52:17.975263 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:52:17.975236 4405 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e5b1b60a-7429-43fe-afd0-1338430750e7-istio-envoy\") pod \"e2e-worker-w2-a0bcbc-openshift-default-65dc9d9b6c-g52rd\" (UID: \"e5b1b60a-7429-43fe-afd0-1338430750e7\") " pod="openshift-ingress/e2e-worker-w2-a0bcbc-openshift-default-65dc9d9b6c-g52rd" Nov 04 02:52:17.975361 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:52:17.975332 4405 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e5b1b60a-7429-43fe-afd0-1338430750e7-istio-podinfo\") pod \"e2e-worker-w2-a0bcbc-openshift-default-65dc9d9b6c-g52rd\" (UID: \"e5b1b60a-7429-43fe-afd0-1338430750e7\") " pod="openshift-ingress/e2e-worker-w2-a0bcbc-openshift-default-65dc9d9b6c-g52rd" Nov 04 02:52:17.991151 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:52:17.991114 4405 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-a0bcbc-openshift-default-65dc9d9b6c-g52rd"] Nov 04 02:52:18.034640 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:52:18.034609 4405 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sll9s\" (UniqueName: \"kubernetes.io/projected/e5b1b60a-7429-43fe-afd0-1338430750e7-kube-api-access-sll9s\") pod \"e2e-worker-w2-a0bcbc-openshift-default-65dc9d9b6c-g52rd\" (UID: \"e5b1b60a-7429-43fe-afd0-1338430750e7\") " pod="openshift-ingress/e2e-worker-w2-a0bcbc-openshift-default-65dc9d9b6c-g52rd" Nov 04 02:52:18.036028 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:52:18.036005 4405 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e5b1b60a-7429-43fe-afd0-1338430750e7-istio-token\") pod \"e2e-worker-w2-a0bcbc-openshift-default-65dc9d9b6c-g52rd\" (UID: \"e5b1b60a-7429-43fe-afd0-1338430750e7\") " pod="openshift-ingress/e2e-worker-w2-a0bcbc-openshift-default-65dc9d9b6c-g52rd" Nov 04 02:52:18.105963 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:52:18.105934 4405 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-a0bcbc-openshift-default-65dc9d9b6c-g52rd" Nov 04 02:52:18.537390 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:52:18.537354 4405 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-a0bcbc-openshift-default-65dc9d9b6c-g52rd"] Nov 04 02:52:18.538083 ip-10-0-134-125 kubenswrapper[4405]: W1104 02:52:18.538056 4405 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode5b1b60a_7429_43fe_afd0_1338430750e7.slice/crio-43d13a31145ea726c9b6daa7ad9155ad60ed8d52fd7165845d2ef934cd1c75f2 WatchSource:0}: Error finding container 43d13a31145ea726c9b6daa7ad9155ad60ed8d52fd7165845d2ef934cd1c75f2: Status 404 returned error can't find the container with id 43d13a31145ea726c9b6daa7ad9155ad60ed8d52fd7165845d2ef934cd1c75f2 Nov 04 02:52:18.541009 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:52:18.540976 4405 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 04 02:52:18.541091 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:52:18.541043 4405 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 04 02:52:18.541091 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:52:18.541074 4405 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 04 02:52:18.651690 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:52:18.651258 4405 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-a0bcbc-openshift-default-65dc9d9b6c-g52rd" event={"ID":"e5b1b60a-7429-43fe-afd0-1338430750e7","Type":"ContainerStarted","Data":"f63f0a7ec7a5d0d7027b4dadfe46a71d035ff39a7a92442bf67b255cb6cceed9"} Nov 04 02:52:18.651690 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:52:18.651323 4405 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-a0bcbc-openshift-default-65dc9d9b6c-g52rd" event={"ID":"e5b1b60a-7429-43fe-afd0-1338430750e7","Type":"ContainerStarted","Data":"43d13a31145ea726c9b6daa7ad9155ad60ed8d52fd7165845d2ef934cd1c75f2"} Nov 04 02:52:18.777958 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:52:18.777890 4405 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-a0bcbc-openshift-default-65dc9d9b6c-g52rd" podStartSLOduration=1.7778697970000001 podStartE2EDuration="1.777869797s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 02:52:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 02:52:18.772913372 +0000 UTC m=+1185.203613529" watchObservedRunningTime="2026-11-04 02:52:18.777869797 +0000 UTC m=+1185.208569919" Nov 04 02:52:19.114689 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:52:19.108598 4405 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-a0bcbc-openshift-default-65dc9d9b6c-g52rd" Nov 04 02:52:19.114689 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:52:19.108794 4405 patch_prober.go:29] interesting pod/e2e-worker-w2-a0bcbc-openshift-default-65dc9d9b6c-g52rd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused" start-of-body= Nov 04 02:52:19.114689 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:52:19.108846 4405 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-a0bcbc-openshift-default-65dc9d9b6c-g52rd" podUID="e5b1b60a-7429-43fe-afd0-1338430750e7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused" Nov 04 02:52:20.110695 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:52:20.109577 4405 patch_prober.go:29] interesting pod/e2e-worker-w2-a0bcbc-openshift-default-65dc9d9b6c-g52rd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused" start-of-body= Nov 04 02:52:20.110695 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:52:20.109638 4405 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-a0bcbc-openshift-default-65dc9d9b6c-g52rd" podUID="e5b1b60a-7429-43fe-afd0-1338430750e7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused" Nov 04 02:52:21.114693 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:52:21.113697 4405 patch_prober.go:29] interesting pod/e2e-worker-w2-a0bcbc-openshift-default-65dc9d9b6c-g52rd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused" start-of-body= Nov 04 02:52:21.114693 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:52:21.113758 4405 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-a0bcbc-openshift-default-65dc9d9b6c-g52rd" podUID="e5b1b60a-7429-43fe-afd0-1338430750e7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused" Nov 04 02:52:22.125690 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:52:22.113296 4405 patch_prober.go:29] interesting pod/e2e-worker-w2-a0bcbc-openshift-default-65dc9d9b6c-g52rd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused" start-of-body= Nov 04 02:52:22.125690 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:52:22.113371 4405 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-a0bcbc-openshift-default-65dc9d9b6c-g52rd" podUID="e5b1b60a-7429-43fe-afd0-1338430750e7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused" Nov 04 02:52:23.111061 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:52:23.111025 4405 patch_prober.go:29] interesting pod/e2e-worker-w2-a0bcbc-openshift-default-65dc9d9b6c-g52rd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused" start-of-body= Nov 04 02:52:23.111364 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:52:23.111336 4405 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-a0bcbc-openshift-default-65dc9d9b6c-g52rd" podUID="e5b1b60a-7429-43fe-afd0-1338430750e7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused" Nov 04 02:52:24.121693 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:52:24.119370 4405 patch_prober.go:29] interesting pod/e2e-worker-w2-a0bcbc-openshift-default-65dc9d9b6c-g52rd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused" start-of-body= Nov 04 02:52:24.121693 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:52:24.119432 4405 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-a0bcbc-openshift-default-65dc9d9b6c-g52rd" podUID="e5b1b60a-7429-43fe-afd0-1338430750e7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused" Nov 04 02:52:25.106276 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:52:25.106237 4405 patch_prober.go:29] interesting pod/e2e-worker-w2-a0bcbc-openshift-default-65dc9d9b6c-g52rd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused" start-of-body= Nov 04 02:52:25.106469 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:52:25.106323 4405 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-a0bcbc-openshift-default-65dc9d9b6c-g52rd" podUID="e5b1b60a-7429-43fe-afd0-1338430750e7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused" Nov 04 02:52:25.646037 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:52:25.646002 4405 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-bl5mg"] Nov 04 02:52:26.311482 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:52:26.311447 4405 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-a0bcbc-openshift-default-65dc9d9b6c-g52rd" Nov 04 02:52:26.311909 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:52:26.311881 4405 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-a0bcbc-openshift-default-65dc9d9b6c-g52rd" Nov 04 02:52:26.312616 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:52:26.312595 4405 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-a0bcbc-openshift-default-65dc9d9b6c-g52rd" Nov 04 02:52:30.280753 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:52:30.280705 4405 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w6-ebb7fd/distinct-2-w6-ebb7fd-kserve-6f68c5db86-zrpjs"] Nov 04 02:52:30.285012 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:52:30.284994 4405 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-ebb7fd/distinct-2-w6-ebb7fd-kserve-6f68c5db86-zrpjs" Nov 04 02:52:30.310015 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:52:30.309981 4405 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b7b786e0-9c94-4126-9949-cd354ebbfb2a-model-cache\") pod \"distinct-2-w6-ebb7fd-kserve-6f68c5db86-zrpjs\" (UID: \"b7b786e0-9c94-4126-9949-cd354ebbfb2a\") " pod="e2e-models-e2e-worker-w6-ebb7fd/distinct-2-w6-ebb7fd-kserve-6f68c5db86-zrpjs" Nov 04 02:52:30.310155 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:52:30.310050 4405 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-krcb7\" (UniqueName: \"kubernetes.io/projected/b7b786e0-9c94-4126-9949-cd354ebbfb2a-kube-api-access-krcb7\") pod \"distinct-2-w6-ebb7fd-kserve-6f68c5db86-zrpjs\" (UID: \"b7b786e0-9c94-4126-9949-cd354ebbfb2a\") " pod="e2e-models-e2e-worker-w6-ebb7fd/distinct-2-w6-ebb7fd-kserve-6f68c5db86-zrpjs" Nov 04 02:52:30.310208 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:52:30.310178 4405 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b7b786e0-9c94-4126-9949-cd354ebbfb2a-home\") pod \"distinct-2-w6-ebb7fd-kserve-6f68c5db86-zrpjs\" (UID: \"b7b786e0-9c94-4126-9949-cd354ebbfb2a\") " pod="e2e-models-e2e-worker-w6-ebb7fd/distinct-2-w6-ebb7fd-kserve-6f68c5db86-zrpjs" Nov 04 02:52:30.310256 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:52:30.310222 4405 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b7b786e0-9c94-4126-9949-cd354ebbfb2a-dshm\") pod \"distinct-2-w6-ebb7fd-kserve-6f68c5db86-zrpjs\" (UID: \"b7b786e0-9c94-4126-9949-cd354ebbfb2a\") " pod="e2e-models-e2e-worker-w6-ebb7fd/distinct-2-w6-ebb7fd-kserve-6f68c5db86-zrpjs" Nov 04 02:52:30.310311 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:52:30.310297 4405 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b7b786e0-9c94-4126-9949-cd354ebbfb2a-tls-certs\") pod \"distinct-2-w6-ebb7fd-kserve-6f68c5db86-zrpjs\" (UID: \"b7b786e0-9c94-4126-9949-cd354ebbfb2a\") " pod="e2e-models-e2e-worker-w6-ebb7fd/distinct-2-w6-ebb7fd-kserve-6f68c5db86-zrpjs" Nov 04 02:52:30.310356 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:52:30.310334 4405 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b7b786e0-9c94-4126-9949-cd354ebbfb2a-tmp-dir\") pod \"distinct-2-w6-ebb7fd-kserve-6f68c5db86-zrpjs\" (UID: \"b7b786e0-9c94-4126-9949-cd354ebbfb2a\") " pod="e2e-models-e2e-worker-w6-ebb7fd/distinct-2-w6-ebb7fd-kserve-6f68c5db86-zrpjs" Nov 04 02:52:30.357686 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:52:30.357637 4405 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w6-ebb7fd/distinct-2-w6-ebb7fd-kserve-6f68c5db86-zrpjs"] Nov 04 02:52:30.403539 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:52:30.403502 4405 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w6-ebb7fd\"/\"distinct-2-w6-ebb7fd-kserve-self-signed-certs\"" Nov 04 02:52:30.414438 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:52:30.414409 4405 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b7b786e0-9c94-4126-9949-cd354ebbfb2a-home\") pod \"distinct-2-w6-ebb7fd-kserve-6f68c5db86-zrpjs\" (UID: \"b7b786e0-9c94-4126-9949-cd354ebbfb2a\") " pod="e2e-models-e2e-worker-w6-ebb7fd/distinct-2-w6-ebb7fd-kserve-6f68c5db86-zrpjs" Nov 04 02:52:30.414627 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:52:30.414458 4405 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b7b786e0-9c94-4126-9949-cd354ebbfb2a-dshm\") pod \"distinct-2-w6-ebb7fd-kserve-6f68c5db86-zrpjs\" (UID: \"b7b786e0-9c94-4126-9949-cd354ebbfb2a\") " pod="e2e-models-e2e-worker-w6-ebb7fd/distinct-2-w6-ebb7fd-kserve-6f68c5db86-zrpjs" Nov 04 02:52:30.414627 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:52:30.414513 4405 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b7b786e0-9c94-4126-9949-cd354ebbfb2a-tls-certs\") pod \"distinct-2-w6-ebb7fd-kserve-6f68c5db86-zrpjs\" (UID: \"b7b786e0-9c94-4126-9949-cd354ebbfb2a\") " pod="e2e-models-e2e-worker-w6-ebb7fd/distinct-2-w6-ebb7fd-kserve-6f68c5db86-zrpjs" Nov 04 02:52:30.414627 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:52:30.414536 4405 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b7b786e0-9c94-4126-9949-cd354ebbfb2a-tmp-dir\") pod \"distinct-2-w6-ebb7fd-kserve-6f68c5db86-zrpjs\" (UID: \"b7b786e0-9c94-4126-9949-cd354ebbfb2a\") " pod="e2e-models-e2e-worker-w6-ebb7fd/distinct-2-w6-ebb7fd-kserve-6f68c5db86-zrpjs" Nov 04 02:52:30.414921 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:52:30.414639 4405 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b7b786e0-9c94-4126-9949-cd354ebbfb2a-model-cache\") pod \"distinct-2-w6-ebb7fd-kserve-6f68c5db86-zrpjs\" (UID: \"b7b786e0-9c94-4126-9949-cd354ebbfb2a\") " pod="e2e-models-e2e-worker-w6-ebb7fd/distinct-2-w6-ebb7fd-kserve-6f68c5db86-zrpjs" Nov 04 02:52:30.414921 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:52:30.414755 4405 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-krcb7\" (UniqueName: \"kubernetes.io/projected/b7b786e0-9c94-4126-9949-cd354ebbfb2a-kube-api-access-krcb7\") pod \"distinct-2-w6-ebb7fd-kserve-6f68c5db86-zrpjs\" (UID: \"b7b786e0-9c94-4126-9949-cd354ebbfb2a\") " pod="e2e-models-e2e-worker-w6-ebb7fd/distinct-2-w6-ebb7fd-kserve-6f68c5db86-zrpjs" Nov 04 02:52:30.415026 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:52:30.414961 4405 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b7b786e0-9c94-4126-9949-cd354ebbfb2a-home\") pod \"distinct-2-w6-ebb7fd-kserve-6f68c5db86-zrpjs\" (UID: \"b7b786e0-9c94-4126-9949-cd354ebbfb2a\") " pod="e2e-models-e2e-worker-w6-ebb7fd/distinct-2-w6-ebb7fd-kserve-6f68c5db86-zrpjs" Nov 04 02:52:30.415207 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:52:30.415184 4405 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b7b786e0-9c94-4126-9949-cd354ebbfb2a-tmp-dir\") pod \"distinct-2-w6-ebb7fd-kserve-6f68c5db86-zrpjs\" (UID: \"b7b786e0-9c94-4126-9949-cd354ebbfb2a\") " pod="e2e-models-e2e-worker-w6-ebb7fd/distinct-2-w6-ebb7fd-kserve-6f68c5db86-zrpjs" Nov 04 02:52:30.415325 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:52:30.415287 4405 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b7b786e0-9c94-4126-9949-cd354ebbfb2a-model-cache\") pod \"distinct-2-w6-ebb7fd-kserve-6f68c5db86-zrpjs\" (UID: \"b7b786e0-9c94-4126-9949-cd354ebbfb2a\") " pod="e2e-models-e2e-worker-w6-ebb7fd/distinct-2-w6-ebb7fd-kserve-6f68c5db86-zrpjs" Nov 04 02:52:30.417084 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:52:30.417058 4405 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b7b786e0-9c94-4126-9949-cd354ebbfb2a-dshm\") pod \"distinct-2-w6-ebb7fd-kserve-6f68c5db86-zrpjs\" (UID: \"b7b786e0-9c94-4126-9949-cd354ebbfb2a\") " pod="e2e-models-e2e-worker-w6-ebb7fd/distinct-2-w6-ebb7fd-kserve-6f68c5db86-zrpjs" Nov 04 02:52:30.417386 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:52:30.417367 4405 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b7b786e0-9c94-4126-9949-cd354ebbfb2a-tls-certs\") pod \"distinct-2-w6-ebb7fd-kserve-6f68c5db86-zrpjs\" (UID: \"b7b786e0-9c94-4126-9949-cd354ebbfb2a\") " pod="e2e-models-e2e-worker-w6-ebb7fd/distinct-2-w6-ebb7fd-kserve-6f68c5db86-zrpjs" Nov 04 02:52:30.476798 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:52:30.476768 4405 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-krcb7\" (UniqueName: \"kubernetes.io/projected/b7b786e0-9c94-4126-9949-cd354ebbfb2a-kube-api-access-krcb7\") pod \"distinct-2-w6-ebb7fd-kserve-6f68c5db86-zrpjs\" (UID: \"b7b786e0-9c94-4126-9949-cd354ebbfb2a\") " pod="e2e-models-e2e-worker-w6-ebb7fd/distinct-2-w6-ebb7fd-kserve-6f68c5db86-zrpjs" Nov 04 02:52:30.599053 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:52:30.599021 4405 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-ebb7fd/distinct-2-w6-ebb7fd-kserve-6f68c5db86-zrpjs" Nov 04 02:52:30.976323 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:52:30.976294 4405 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w6-ebb7fd/distinct-2-w6-ebb7fd-kserve-6f68c5db86-zrpjs"] Nov 04 02:52:30.977938 ip-10-0-134-125 kubenswrapper[4405]: W1104 02:52:30.977911 4405 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podb7b786e0_9c94_4126_9949_cd354ebbfb2a.slice/crio-5213d798ca18714e389301b7ab15565fa0279322501a0ee360aca5f464cde233 WatchSource:0}: Error finding container 5213d798ca18714e389301b7ab15565fa0279322501a0ee360aca5f464cde233: Status 404 returned error can't find the container with id 5213d798ca18714e389301b7ab15565fa0279322501a0ee360aca5f464cde233 Nov 04 02:52:31.794025 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:52:31.793991 4405 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-ebb7fd/distinct-2-w6-ebb7fd-kserve-6f68c5db86-zrpjs" event={"ID":"b7b786e0-9c94-4126-9949-cd354ebbfb2a","Type":"ContainerStarted","Data":"e2f3c926f88fc03a1997095f2ab36a48350313e4f58c734bfc2571d1daa8689d"} Nov 04 02:52:31.794525 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:52:31.794036 4405 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-ebb7fd/distinct-2-w6-ebb7fd-kserve-6f68c5db86-zrpjs" event={"ID":"b7b786e0-9c94-4126-9949-cd354ebbfb2a","Type":"ContainerStarted","Data":"5213d798ca18714e389301b7ab15565fa0279322501a0ee360aca5f464cde233"} Nov 04 02:52:31.911933 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:52:31.911879 4405 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w6-ebb7fd/distinct-2-w6-ebb7fd-kserve-6f68c5db86-zrpjs" podStartSLOduration=1.911858361 podStartE2EDuration="1.911858361s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 02:52:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 02:52:31.910050541 +0000 UTC m=+1198.340750671" watchObservedRunningTime="2026-11-04 02:52:31.911858361 +0000 UTC m=+1198.342558480" Nov 04 02:52:32.599724 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:52:32.599680 4405 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-ebb7fd/distinct-2-w6-ebb7fd-kserve-6f68c5db86-zrpjs" Nov 04 02:52:33.809719 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:52:33.809690 4405 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-vwcx8_ffbb2d83-5a93-4a5b-bce3-e40afb8f58ea/ovn-acl-logging/0.log" Nov 04 02:52:33.811920 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:52:33.811898 4405 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-vwcx8_ffbb2d83-5a93-4a5b-bce3-e40afb8f58ea/ovn-acl-logging/0.log" Nov 04 02:52:36.884540 ip-10-0-134-125 kubenswrapper[4405]: E1104 02:52:36.884498 4405 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5f2vb" containerName="extractor" Nov 04 02:52:40.600274 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:52:40.600213 4405 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w6-ebb7fd/distinct-2-w6-ebb7fd-kserve-6f68c5db86-zrpjs" Nov 04 02:52:40.614262 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:52:40.614235 4405 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w6-ebb7fd/distinct-2-w6-ebb7fd-kserve-6f68c5db86-zrpjs" Nov 04 02:52:40.874999 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:52:40.874907 4405 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-ebb7fd/distinct-2-w6-ebb7fd-kserve-6f68c5db86-zrpjs" Nov 04 02:53:03.138023 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:53:03.137942 4405 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-a0bcbc-openshift-default-65dc9d9b6c-fzwkp"] Nov 04 02:53:03.143174 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:53:03.143121 4405 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-a0bcbc-openshift-default-65dc9d9b6c-fzwkp" Nov 04 02:53:03.249076 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:53:03.249041 4405 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-a0bcbc-openshift-default-65dc9d9b6c-fzwkp"] Nov 04 02:53:03.294313 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:53:03.294272 4405 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/700d7662-7d5b-40b0-afea-e55dc01922e6-workload-certs\") pod \"e2e-worker-w2-a0bcbc-openshift-default-65dc9d9b6c-fzwkp\" (UID: \"700d7662-7d5b-40b0-afea-e55dc01922e6\") " pod="openshift-ingress/e2e-worker-w2-a0bcbc-openshift-default-65dc9d9b6c-fzwkp" Nov 04 02:53:03.294531 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:53:03.294320 4405 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/700d7662-7d5b-40b0-afea-e55dc01922e6-istio-envoy\") pod \"e2e-worker-w2-a0bcbc-openshift-default-65dc9d9b6c-fzwkp\" (UID: \"700d7662-7d5b-40b0-afea-e55dc01922e6\") " pod="openshift-ingress/e2e-worker-w2-a0bcbc-openshift-default-65dc9d9b6c-fzwkp" Nov 04 02:53:03.294531 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:53:03.294391 4405 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/700d7662-7d5b-40b0-afea-e55dc01922e6-istiod-ca-cert\") pod \"e2e-worker-w2-a0bcbc-openshift-default-65dc9d9b6c-fzwkp\" (UID: \"700d7662-7d5b-40b0-afea-e55dc01922e6\") " pod="openshift-ingress/e2e-worker-w2-a0bcbc-openshift-default-65dc9d9b6c-fzwkp" Nov 04 02:53:03.294659 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:53:03.294545 4405 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/700d7662-7d5b-40b0-afea-e55dc01922e6-istio-token\") pod \"e2e-worker-w2-a0bcbc-openshift-default-65dc9d9b6c-fzwkp\" (UID: \"700d7662-7d5b-40b0-afea-e55dc01922e6\") " pod="openshift-ingress/e2e-worker-w2-a0bcbc-openshift-default-65dc9d9b6c-fzwkp" Nov 04 02:53:03.294727 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:53:03.294692 4405 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/700d7662-7d5b-40b0-afea-e55dc01922e6-istio-podinfo\") pod \"e2e-worker-w2-a0bcbc-openshift-default-65dc9d9b6c-fzwkp\" (UID: \"700d7662-7d5b-40b0-afea-e55dc01922e6\") " pod="openshift-ingress/e2e-worker-w2-a0bcbc-openshift-default-65dc9d9b6c-fzwkp" Nov 04 02:53:03.294789 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:53:03.294771 4405 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vvn59\" (UniqueName: \"kubernetes.io/projected/700d7662-7d5b-40b0-afea-e55dc01922e6-kube-api-access-vvn59\") pod \"e2e-worker-w2-a0bcbc-openshift-default-65dc9d9b6c-fzwkp\" (UID: \"700d7662-7d5b-40b0-afea-e55dc01922e6\") " pod="openshift-ingress/e2e-worker-w2-a0bcbc-openshift-default-65dc9d9b6c-fzwkp" Nov 04 02:53:03.294869 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:53:03.294855 4405 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/700d7662-7d5b-40b0-afea-e55dc01922e6-workload-socket\") pod \"e2e-worker-w2-a0bcbc-openshift-default-65dc9d9b6c-fzwkp\" (UID: \"700d7662-7d5b-40b0-afea-e55dc01922e6\") " pod="openshift-ingress/e2e-worker-w2-a0bcbc-openshift-default-65dc9d9b6c-fzwkp" Nov 04 02:53:03.294906 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:53:03.294885 4405 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/700d7662-7d5b-40b0-afea-e55dc01922e6-credential-socket\") pod \"e2e-worker-w2-a0bcbc-openshift-default-65dc9d9b6c-fzwkp\" (UID: \"700d7662-7d5b-40b0-afea-e55dc01922e6\") " pod="openshift-ingress/e2e-worker-w2-a0bcbc-openshift-default-65dc9d9b6c-fzwkp" Nov 04 02:53:03.294944 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:53:03.294912 4405 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/700d7662-7d5b-40b0-afea-e55dc01922e6-istio-data\") pod \"e2e-worker-w2-a0bcbc-openshift-default-65dc9d9b6c-fzwkp\" (UID: \"700d7662-7d5b-40b0-afea-e55dc01922e6\") " pod="openshift-ingress/e2e-worker-w2-a0bcbc-openshift-default-65dc9d9b6c-fzwkp" Nov 04 02:53:03.396695 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:53:03.396567 4405 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/700d7662-7d5b-40b0-afea-e55dc01922e6-istio-token\") pod \"e2e-worker-w2-a0bcbc-openshift-default-65dc9d9b6c-fzwkp\" (UID: \"700d7662-7d5b-40b0-afea-e55dc01922e6\") " pod="openshift-ingress/e2e-worker-w2-a0bcbc-openshift-default-65dc9d9b6c-fzwkp" Nov 04 02:53:03.396880 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:53:03.396713 4405 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/700d7662-7d5b-40b0-afea-e55dc01922e6-istio-podinfo\") pod \"e2e-worker-w2-a0bcbc-openshift-default-65dc9d9b6c-fzwkp\" (UID: \"700d7662-7d5b-40b0-afea-e55dc01922e6\") " pod="openshift-ingress/e2e-worker-w2-a0bcbc-openshift-default-65dc9d9b6c-fzwkp" Nov 04 02:53:03.396880 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:53:03.396777 4405 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vvn59\" (UniqueName: \"kubernetes.io/projected/700d7662-7d5b-40b0-afea-e55dc01922e6-kube-api-access-vvn59\") pod \"e2e-worker-w2-a0bcbc-openshift-default-65dc9d9b6c-fzwkp\" (UID: \"700d7662-7d5b-40b0-afea-e55dc01922e6\") " pod="openshift-ingress/e2e-worker-w2-a0bcbc-openshift-default-65dc9d9b6c-fzwkp" Nov 04 02:53:03.396880 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:53:03.396810 4405 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/700d7662-7d5b-40b0-afea-e55dc01922e6-workload-socket\") pod \"e2e-worker-w2-a0bcbc-openshift-default-65dc9d9b6c-fzwkp\" (UID: \"700d7662-7d5b-40b0-afea-e55dc01922e6\") " pod="openshift-ingress/e2e-worker-w2-a0bcbc-openshift-default-65dc9d9b6c-fzwkp" Nov 04 02:53:03.396880 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:53:03.396834 4405 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/700d7662-7d5b-40b0-afea-e55dc01922e6-credential-socket\") pod \"e2e-worker-w2-a0bcbc-openshift-default-65dc9d9b6c-fzwkp\" (UID: \"700d7662-7d5b-40b0-afea-e55dc01922e6\") " pod="openshift-ingress/e2e-worker-w2-a0bcbc-openshift-default-65dc9d9b6c-fzwkp" Nov 04 02:53:03.396880 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:53:03.396860 4405 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/700d7662-7d5b-40b0-afea-e55dc01922e6-istio-data\") pod \"e2e-worker-w2-a0bcbc-openshift-default-65dc9d9b6c-fzwkp\" (UID: \"700d7662-7d5b-40b0-afea-e55dc01922e6\") " pod="openshift-ingress/e2e-worker-w2-a0bcbc-openshift-default-65dc9d9b6c-fzwkp" Nov 04 02:53:03.397145 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:53:03.397013 4405 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/700d7662-7d5b-40b0-afea-e55dc01922e6-workload-certs\") pod \"e2e-worker-w2-a0bcbc-openshift-default-65dc9d9b6c-fzwkp\" (UID: \"700d7662-7d5b-40b0-afea-e55dc01922e6\") " pod="openshift-ingress/e2e-worker-w2-a0bcbc-openshift-default-65dc9d9b6c-fzwkp" Nov 04 02:53:03.397145 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:53:03.397036 4405 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/700d7662-7d5b-40b0-afea-e55dc01922e6-istio-envoy\") pod \"e2e-worker-w2-a0bcbc-openshift-default-65dc9d9b6c-fzwkp\" (UID: \"700d7662-7d5b-40b0-afea-e55dc01922e6\") " pod="openshift-ingress/e2e-worker-w2-a0bcbc-openshift-default-65dc9d9b6c-fzwkp" Nov 04 02:53:03.397145 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:53:03.397062 4405 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/700d7662-7d5b-40b0-afea-e55dc01922e6-istiod-ca-cert\") pod \"e2e-worker-w2-a0bcbc-openshift-default-65dc9d9b6c-fzwkp\" (UID: \"700d7662-7d5b-40b0-afea-e55dc01922e6\") " pod="openshift-ingress/e2e-worker-w2-a0bcbc-openshift-default-65dc9d9b6c-fzwkp" Nov 04 02:53:03.397493 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:53:03.397461 4405 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/700d7662-7d5b-40b0-afea-e55dc01922e6-workload-certs\") pod \"e2e-worker-w2-a0bcbc-openshift-default-65dc9d9b6c-fzwkp\" (UID: \"700d7662-7d5b-40b0-afea-e55dc01922e6\") " pod="openshift-ingress/e2e-worker-w2-a0bcbc-openshift-default-65dc9d9b6c-fzwkp" Nov 04 02:53:03.397646 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:53:03.397460 4405 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/700d7662-7d5b-40b0-afea-e55dc01922e6-istio-data\") pod \"e2e-worker-w2-a0bcbc-openshift-default-65dc9d9b6c-fzwkp\" (UID: \"700d7662-7d5b-40b0-afea-e55dc01922e6\") " pod="openshift-ingress/e2e-worker-w2-a0bcbc-openshift-default-65dc9d9b6c-fzwkp" Nov 04 02:53:03.397886 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:53:03.397696 4405 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/700d7662-7d5b-40b0-afea-e55dc01922e6-workload-socket\") pod \"e2e-worker-w2-a0bcbc-openshift-default-65dc9d9b6c-fzwkp\" (UID: \"700d7662-7d5b-40b0-afea-e55dc01922e6\") " pod="openshift-ingress/e2e-worker-w2-a0bcbc-openshift-default-65dc9d9b6c-fzwkp" Nov 04 02:53:03.397886 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:53:03.397899 4405 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/700d7662-7d5b-40b0-afea-e55dc01922e6-istiod-ca-cert\") pod \"e2e-worker-w2-a0bcbc-openshift-default-65dc9d9b6c-fzwkp\" (UID: \"700d7662-7d5b-40b0-afea-e55dc01922e6\") " pod="openshift-ingress/e2e-worker-w2-a0bcbc-openshift-default-65dc9d9b6c-fzwkp" Nov 04 02:53:03.397886 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:53:03.397927 4405 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/700d7662-7d5b-40b0-afea-e55dc01922e6-credential-socket\") pod \"e2e-worker-w2-a0bcbc-openshift-default-65dc9d9b6c-fzwkp\" (UID: \"700d7662-7d5b-40b0-afea-e55dc01922e6\") " pod="openshift-ingress/e2e-worker-w2-a0bcbc-openshift-default-65dc9d9b6c-fzwkp" Nov 04 02:53:03.399548 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:53:03.399513 4405 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/700d7662-7d5b-40b0-afea-e55dc01922e6-istio-podinfo\") pod \"e2e-worker-w2-a0bcbc-openshift-default-65dc9d9b6c-fzwkp\" (UID: \"700d7662-7d5b-40b0-afea-e55dc01922e6\") " pod="openshift-ingress/e2e-worker-w2-a0bcbc-openshift-default-65dc9d9b6c-fzwkp" Nov 04 02:53:03.399548 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:53:03.399535 4405 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/700d7662-7d5b-40b0-afea-e55dc01922e6-istio-envoy\") pod \"e2e-worker-w2-a0bcbc-openshift-default-65dc9d9b6c-fzwkp\" (UID: \"700d7662-7d5b-40b0-afea-e55dc01922e6\") " pod="openshift-ingress/e2e-worker-w2-a0bcbc-openshift-default-65dc9d9b6c-fzwkp" Nov 04 02:53:03.459801 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:53:03.459765 4405 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/700d7662-7d5b-40b0-afea-e55dc01922e6-istio-token\") pod \"e2e-worker-w2-a0bcbc-openshift-default-65dc9d9b6c-fzwkp\" (UID: \"700d7662-7d5b-40b0-afea-e55dc01922e6\") " pod="openshift-ingress/e2e-worker-w2-a0bcbc-openshift-default-65dc9d9b6c-fzwkp" Nov 04 02:53:03.460020 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:53:03.459987 4405 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vvn59\" (UniqueName: \"kubernetes.io/projected/700d7662-7d5b-40b0-afea-e55dc01922e6-kube-api-access-vvn59\") pod \"e2e-worker-w2-a0bcbc-openshift-default-65dc9d9b6c-fzwkp\" (UID: \"700d7662-7d5b-40b0-afea-e55dc01922e6\") " pod="openshift-ingress/e2e-worker-w2-a0bcbc-openshift-default-65dc9d9b6c-fzwkp" Nov 04 02:53:03.757872 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:53:03.757786 4405 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-a0bcbc-openshift-default-65dc9d9b6c-fzwkp" Nov 04 02:53:04.193186 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:53:04.193141 4405 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-a0bcbc-openshift-default-65dc9d9b6c-fzwkp"] Nov 04 02:53:04.195068 ip-10-0-134-125 kubenswrapper[4405]: W1104 02:53:04.195035 4405 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod700d7662_7d5b_40b0_afea_e55dc01922e6.slice/crio-8dbab7567bfc10c8268f7166e7294c13a0c339f85203ccf7e1226e0aca4b9120 WatchSource:0}: Error finding container 8dbab7567bfc10c8268f7166e7294c13a0c339f85203ccf7e1226e0aca4b9120: Status 404 returned error can't find the container with id 8dbab7567bfc10c8268f7166e7294c13a0c339f85203ccf7e1226e0aca4b9120 Nov 04 02:53:04.198350 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:53:04.198306 4405 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 04 02:53:04.198454 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:53:04.198402 4405 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 04 02:53:04.198454 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:53:04.198445 4405 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 04 02:53:05.030913 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:53:05.030879 4405 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-a0bcbc-openshift-default-65dc9d9b6c-fzwkp" event={"ID":"700d7662-7d5b-40b0-afea-e55dc01922e6","Type":"ContainerStarted","Data":"78dd8506b9d9043226ee6d6914645f7efbd422a5e486742873dbb05dbe7b5758"} Nov 04 02:53:05.031102 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:53:05.030939 4405 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-a0bcbc-openshift-default-65dc9d9b6c-fzwkp" event={"ID":"700d7662-7d5b-40b0-afea-e55dc01922e6","Type":"ContainerStarted","Data":"8dbab7567bfc10c8268f7166e7294c13a0c339f85203ccf7e1226e0aca4b9120"} Nov 04 02:53:05.758572 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:53:05.758533 4405 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-a0bcbc-openshift-default-65dc9d9b6c-fzwkp" Nov 04 02:53:05.759970 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:53:05.759947 4405 patch_prober.go:29] interesting pod/e2e-worker-w2-a0bcbc-openshift-default-65dc9d9b6c-fzwkp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused" start-of-body= Nov 04 02:53:05.760037 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:53:05.759999 4405 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-a0bcbc-openshift-default-65dc9d9b6c-fzwkp" podUID="700d7662-7d5b-40b0-afea-e55dc01922e6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused" Nov 04 02:53:06.085526 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:53:06.085478 4405 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-a0bcbc-openshift-default-65dc9d9b6c-fzwkp" podStartSLOduration=3.085464166 podStartE2EDuration="3.085464166s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 02:53:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 02:53:05.225052581 +0000 UTC m=+1231.655752702" watchObservedRunningTime="2026-11-04 02:53:06.085464166 +0000 UTC m=+1232.516164286" Nov 04 02:53:06.086367 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:53:06.086346 4405 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-bl5mg"] Nov 04 02:53:06.758690 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:53:06.758611 4405 patch_prober.go:29] interesting pod/e2e-worker-w2-a0bcbc-openshift-default-65dc9d9b6c-fzwkp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused" start-of-body= Nov 04 02:53:06.758690 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:53:06.758661 4405 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-a0bcbc-openshift-default-65dc9d9b6c-fzwkp" podUID="700d7662-7d5b-40b0-afea-e55dc01922e6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused" Nov 04 02:53:07.050870 ip-10-0-134-125 kubenswrapper[4405]: E1104 02:53:07.049442 4405 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5f2vb" containerName="extractor" Nov 04 02:53:07.777022 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:53:07.776973 4405 patch_prober.go:29] interesting pod/e2e-worker-w2-a0bcbc-openshift-default-65dc9d9b6c-fzwkp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused" start-of-body= Nov 04 02:53:07.777806 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:53:07.777763 4405 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-a0bcbc-openshift-default-65dc9d9b6c-fzwkp" podUID="700d7662-7d5b-40b0-afea-e55dc01922e6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused" Nov 04 02:53:08.761260 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:53:08.761222 4405 patch_prober.go:29] interesting pod/e2e-worker-w2-a0bcbc-openshift-default-65dc9d9b6c-fzwkp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused" start-of-body= Nov 04 02:53:08.761736 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:53:08.761704 4405 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-a0bcbc-openshift-default-65dc9d9b6c-fzwkp" podUID="700d7662-7d5b-40b0-afea-e55dc01922e6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused" Nov 04 02:53:09.758418 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:53:09.758382 4405 patch_prober.go:29] interesting pod/e2e-worker-w2-a0bcbc-openshift-default-65dc9d9b6c-fzwkp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused" start-of-body= Nov 04 02:53:09.758877 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:53:09.758453 4405 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-a0bcbc-openshift-default-65dc9d9b6c-fzwkp" podUID="700d7662-7d5b-40b0-afea-e55dc01922e6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused" Nov 04 02:53:10.759089 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:53:10.759044 4405 patch_prober.go:29] interesting pod/e2e-worker-w2-a0bcbc-openshift-default-65dc9d9b6c-fzwkp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused" start-of-body= Nov 04 02:53:10.759654 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:53:10.759134 4405 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-a0bcbc-openshift-default-65dc9d9b6c-fzwkp" podUID="700d7662-7d5b-40b0-afea-e55dc01922e6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused" Nov 04 02:53:12.759940 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:53:12.759898 4405 patch_prober.go:29] interesting pod/e2e-worker-w2-a0bcbc-openshift-default-65dc9d9b6c-fzwkp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.66:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 04 02:53:12.760583 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:53:12.760549 4405 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-a0bcbc-openshift-default-65dc9d9b6c-fzwkp" podUID="700d7662-7d5b-40b0-afea-e55dc01922e6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.66:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 04 02:53:13.642048 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:53:13.642017 4405 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-a0bcbc-openshift-default-65dc9d9b6c-fzwkp" Nov 04 02:53:13.642377 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:53:13.642353 4405 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-a0bcbc-openshift-default-65dc9d9b6c-fzwkp" Nov 04 02:53:13.643335 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:53:13.643317 4405 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-a0bcbc-openshift-default-65dc9d9b6c-fzwkp" Nov 04 02:53:20.088183 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:53:20.088139 4405 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w6-ebb7fd-openshift-default-544f5768f-mngkl"] Nov 04 02:53:20.093504 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:53:20.093478 4405 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-ebb7fd-openshift-default-544f5768f-mngkl" Nov 04 02:53:20.152939 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:53:20.152899 4405 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-ebb7fd-openshift-default-544f5768f-mngkl"] Nov 04 02:53:20.201543 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:53:20.201510 4405 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f55884d8-38e1-416b-853d-9c09082444bf-workload-certs\") pod \"e2e-worker-w6-ebb7fd-openshift-default-544f5768f-mngkl\" (UID: \"f55884d8-38e1-416b-853d-9c09082444bf\") " pod="openshift-ingress/e2e-worker-w6-ebb7fd-openshift-default-544f5768f-mngkl" Nov 04 02:53:20.201773 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:53:20.201551 4405 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sjjlx\" (UniqueName: \"kubernetes.io/projected/f55884d8-38e1-416b-853d-9c09082444bf-kube-api-access-sjjlx\") pod \"e2e-worker-w6-ebb7fd-openshift-default-544f5768f-mngkl\" (UID: \"f55884d8-38e1-416b-853d-9c09082444bf\") " pod="openshift-ingress/e2e-worker-w6-ebb7fd-openshift-default-544f5768f-mngkl" Nov 04 02:53:20.201773 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:53:20.201697 4405 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f55884d8-38e1-416b-853d-9c09082444bf-istiod-ca-cert\") pod \"e2e-worker-w6-ebb7fd-openshift-default-544f5768f-mngkl\" (UID: \"f55884d8-38e1-416b-853d-9c09082444bf\") " pod="openshift-ingress/e2e-worker-w6-ebb7fd-openshift-default-544f5768f-mngkl" Nov 04 02:53:20.201875 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:53:20.201771 4405 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f55884d8-38e1-416b-853d-9c09082444bf-istio-envoy\") pod \"e2e-worker-w6-ebb7fd-openshift-default-544f5768f-mngkl\" (UID: \"f55884d8-38e1-416b-853d-9c09082444bf\") " pod="openshift-ingress/e2e-worker-w6-ebb7fd-openshift-default-544f5768f-mngkl" Nov 04 02:53:20.201875 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:53:20.201822 4405 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f55884d8-38e1-416b-853d-9c09082444bf-istio-token\") pod \"e2e-worker-w6-ebb7fd-openshift-default-544f5768f-mngkl\" (UID: \"f55884d8-38e1-416b-853d-9c09082444bf\") " pod="openshift-ingress/e2e-worker-w6-ebb7fd-openshift-default-544f5768f-mngkl" Nov 04 02:53:20.201875 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:53:20.201854 4405 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f55884d8-38e1-416b-853d-9c09082444bf-credential-socket\") pod \"e2e-worker-w6-ebb7fd-openshift-default-544f5768f-mngkl\" (UID: \"f55884d8-38e1-416b-853d-9c09082444bf\") " pod="openshift-ingress/e2e-worker-w6-ebb7fd-openshift-default-544f5768f-mngkl" Nov 04 02:53:20.202075 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:53:20.202054 4405 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f55884d8-38e1-416b-853d-9c09082444bf-workload-socket\") pod \"e2e-worker-w6-ebb7fd-openshift-default-544f5768f-mngkl\" (UID: \"f55884d8-38e1-416b-853d-9c09082444bf\") " pod="openshift-ingress/e2e-worker-w6-ebb7fd-openshift-default-544f5768f-mngkl" Nov 04 02:53:20.202115 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:53:20.202103 4405 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f55884d8-38e1-416b-853d-9c09082444bf-istio-data\") pod \"e2e-worker-w6-ebb7fd-openshift-default-544f5768f-mngkl\" (UID: \"f55884d8-38e1-416b-853d-9c09082444bf\") " pod="openshift-ingress/e2e-worker-w6-ebb7fd-openshift-default-544f5768f-mngkl" Nov 04 02:53:20.202171 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:53:20.202154 4405 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f55884d8-38e1-416b-853d-9c09082444bf-istio-podinfo\") pod \"e2e-worker-w6-ebb7fd-openshift-default-544f5768f-mngkl\" (UID: \"f55884d8-38e1-416b-853d-9c09082444bf\") " pod="openshift-ingress/e2e-worker-w6-ebb7fd-openshift-default-544f5768f-mngkl" Nov 04 02:53:20.303339 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:53:20.303298 4405 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f55884d8-38e1-416b-853d-9c09082444bf-workload-socket\") pod \"e2e-worker-w6-ebb7fd-openshift-default-544f5768f-mngkl\" (UID: \"f55884d8-38e1-416b-853d-9c09082444bf\") " pod="openshift-ingress/e2e-worker-w6-ebb7fd-openshift-default-544f5768f-mngkl" Nov 04 02:53:20.303339 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:53:20.303345 4405 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f55884d8-38e1-416b-853d-9c09082444bf-istio-data\") pod \"e2e-worker-w6-ebb7fd-openshift-default-544f5768f-mngkl\" (UID: \"f55884d8-38e1-416b-853d-9c09082444bf\") " pod="openshift-ingress/e2e-worker-w6-ebb7fd-openshift-default-544f5768f-mngkl" Nov 04 02:53:20.303604 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:53:20.303379 4405 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f55884d8-38e1-416b-853d-9c09082444bf-istio-podinfo\") pod \"e2e-worker-w6-ebb7fd-openshift-default-544f5768f-mngkl\" (UID: \"f55884d8-38e1-416b-853d-9c09082444bf\") " pod="openshift-ingress/e2e-worker-w6-ebb7fd-openshift-default-544f5768f-mngkl" Nov 04 02:53:20.303604 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:53:20.303439 4405 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f55884d8-38e1-416b-853d-9c09082444bf-workload-certs\") pod \"e2e-worker-w6-ebb7fd-openshift-default-544f5768f-mngkl\" (UID: \"f55884d8-38e1-416b-853d-9c09082444bf\") " pod="openshift-ingress/e2e-worker-w6-ebb7fd-openshift-default-544f5768f-mngkl" Nov 04 02:53:20.303604 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:53:20.303462 4405 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sjjlx\" (UniqueName: \"kubernetes.io/projected/f55884d8-38e1-416b-853d-9c09082444bf-kube-api-access-sjjlx\") pod \"e2e-worker-w6-ebb7fd-openshift-default-544f5768f-mngkl\" (UID: \"f55884d8-38e1-416b-853d-9c09082444bf\") " pod="openshift-ingress/e2e-worker-w6-ebb7fd-openshift-default-544f5768f-mngkl" Nov 04 02:53:20.303604 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:53:20.303501 4405 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f55884d8-38e1-416b-853d-9c09082444bf-istiod-ca-cert\") pod \"e2e-worker-w6-ebb7fd-openshift-default-544f5768f-mngkl\" (UID: \"f55884d8-38e1-416b-853d-9c09082444bf\") " pod="openshift-ingress/e2e-worker-w6-ebb7fd-openshift-default-544f5768f-mngkl" Nov 04 02:53:20.303604 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:53:20.303516 4405 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f55884d8-38e1-416b-853d-9c09082444bf-istio-envoy\") pod \"e2e-worker-w6-ebb7fd-openshift-default-544f5768f-mngkl\" (UID: \"f55884d8-38e1-416b-853d-9c09082444bf\") " pod="openshift-ingress/e2e-worker-w6-ebb7fd-openshift-default-544f5768f-mngkl" Nov 04 02:53:20.303604 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:53:20.303535 4405 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f55884d8-38e1-416b-853d-9c09082444bf-istio-token\") pod \"e2e-worker-w6-ebb7fd-openshift-default-544f5768f-mngkl\" (UID: \"f55884d8-38e1-416b-853d-9c09082444bf\") " pod="openshift-ingress/e2e-worker-w6-ebb7fd-openshift-default-544f5768f-mngkl" Nov 04 02:53:20.303604 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:53:20.303551 4405 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f55884d8-38e1-416b-853d-9c09082444bf-credential-socket\") pod \"e2e-worker-w6-ebb7fd-openshift-default-544f5768f-mngkl\" (UID: \"f55884d8-38e1-416b-853d-9c09082444bf\") " pod="openshift-ingress/e2e-worker-w6-ebb7fd-openshift-default-544f5768f-mngkl" Nov 04 02:53:20.304012 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:53:20.303836 4405 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f55884d8-38e1-416b-853d-9c09082444bf-workload-socket\") pod \"e2e-worker-w6-ebb7fd-openshift-default-544f5768f-mngkl\" (UID: \"f55884d8-38e1-416b-853d-9c09082444bf\") " pod="openshift-ingress/e2e-worker-w6-ebb7fd-openshift-default-544f5768f-mngkl" Nov 04 02:53:20.304012 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:53:20.303926 4405 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f55884d8-38e1-416b-853d-9c09082444bf-workload-certs\") pod \"e2e-worker-w6-ebb7fd-openshift-default-544f5768f-mngkl\" (UID: \"f55884d8-38e1-416b-853d-9c09082444bf\") " pod="openshift-ingress/e2e-worker-w6-ebb7fd-openshift-default-544f5768f-mngkl" Nov 04 02:53:20.304012 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:53:20.303951 4405 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f55884d8-38e1-416b-853d-9c09082444bf-istio-data\") pod \"e2e-worker-w6-ebb7fd-openshift-default-544f5768f-mngkl\" (UID: \"f55884d8-38e1-416b-853d-9c09082444bf\") " pod="openshift-ingress/e2e-worker-w6-ebb7fd-openshift-default-544f5768f-mngkl" Nov 04 02:53:20.304428 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:53:20.304399 4405 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f55884d8-38e1-416b-853d-9c09082444bf-istiod-ca-cert\") pod \"e2e-worker-w6-ebb7fd-openshift-default-544f5768f-mngkl\" (UID: \"f55884d8-38e1-416b-853d-9c09082444bf\") " pod="openshift-ingress/e2e-worker-w6-ebb7fd-openshift-default-544f5768f-mngkl" Nov 04 02:53:20.304552 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:53:20.304459 4405 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f55884d8-38e1-416b-853d-9c09082444bf-credential-socket\") pod \"e2e-worker-w6-ebb7fd-openshift-default-544f5768f-mngkl\" (UID: \"f55884d8-38e1-416b-853d-9c09082444bf\") " pod="openshift-ingress/e2e-worker-w6-ebb7fd-openshift-default-544f5768f-mngkl" Nov 04 02:53:20.306418 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:53:20.306394 4405 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f55884d8-38e1-416b-853d-9c09082444bf-istio-envoy\") pod \"e2e-worker-w6-ebb7fd-openshift-default-544f5768f-mngkl\" (UID: \"f55884d8-38e1-416b-853d-9c09082444bf\") " pod="openshift-ingress/e2e-worker-w6-ebb7fd-openshift-default-544f5768f-mngkl" Nov 04 02:53:20.307016 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:53:20.306992 4405 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f55884d8-38e1-416b-853d-9c09082444bf-istio-podinfo\") pod \"e2e-worker-w6-ebb7fd-openshift-default-544f5768f-mngkl\" (UID: \"f55884d8-38e1-416b-853d-9c09082444bf\") " pod="openshift-ingress/e2e-worker-w6-ebb7fd-openshift-default-544f5768f-mngkl" Nov 04 02:53:20.365320 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:53:20.365237 4405 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f55884d8-38e1-416b-853d-9c09082444bf-istio-token\") pod \"e2e-worker-w6-ebb7fd-openshift-default-544f5768f-mngkl\" (UID: \"f55884d8-38e1-416b-853d-9c09082444bf\") " pod="openshift-ingress/e2e-worker-w6-ebb7fd-openshift-default-544f5768f-mngkl" Nov 04 02:53:20.365506 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:53:20.365476 4405 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sjjlx\" (UniqueName: \"kubernetes.io/projected/f55884d8-38e1-416b-853d-9c09082444bf-kube-api-access-sjjlx\") pod \"e2e-worker-w6-ebb7fd-openshift-default-544f5768f-mngkl\" (UID: \"f55884d8-38e1-416b-853d-9c09082444bf\") " pod="openshift-ingress/e2e-worker-w6-ebb7fd-openshift-default-544f5768f-mngkl" Nov 04 02:53:20.412539 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:53:20.412495 4405 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-ebb7fd-openshift-default-544f5768f-mngkl" Nov 04 02:53:20.851042 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:53:20.851016 4405 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-ebb7fd-openshift-default-544f5768f-mngkl"] Nov 04 02:53:20.852237 ip-10-0-134-125 kubenswrapper[4405]: W1104 02:53:20.852211 4405 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf55884d8_38e1_416b_853d_9c09082444bf.slice/crio-5165ac3190835fadeb23be42ae6025ae7e78deea8a4ea0f4df4ab91c817ecaef WatchSource:0}: Error finding container 5165ac3190835fadeb23be42ae6025ae7e78deea8a4ea0f4df4ab91c817ecaef: Status 404 returned error can't find the container with id 5165ac3190835fadeb23be42ae6025ae7e78deea8a4ea0f4df4ab91c817ecaef Nov 04 02:53:20.854831 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:53:20.854801 4405 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 04 02:53:20.854934 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:53:20.854866 4405 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 04 02:53:20.854934 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:53:20.854896 4405 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 04 02:53:21.246591 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:53:21.246487 4405 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-ebb7fd-openshift-default-544f5768f-mngkl" event={"ID":"f55884d8-38e1-416b-853d-9c09082444bf","Type":"ContainerStarted","Data":"a6f9bfbf61349ba11b6f631437bbf329fa68d34d50afb063d904657629571c89"} Nov 04 02:53:21.246591 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:53:21.246562 4405 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-ebb7fd-openshift-default-544f5768f-mngkl" event={"ID":"f55884d8-38e1-416b-853d-9c09082444bf","Type":"ContainerStarted","Data":"5165ac3190835fadeb23be42ae6025ae7e78deea8a4ea0f4df4ab91c817ecaef"} Nov 04 02:53:21.368155 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:53:21.368097 4405 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w6-ebb7fd-openshift-default-544f5768f-mngkl" podStartSLOduration=1.36808277 podStartE2EDuration="1.36808277s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 02:53:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 02:53:21.366027209 +0000 UTC m=+1247.796727356" watchObservedRunningTime="2026-11-04 02:53:21.36808277 +0000 UTC m=+1247.798782889" Nov 04 02:53:21.414582 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:53:21.414541 4405 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w6-ebb7fd-openshift-default-544f5768f-mngkl" Nov 04 02:53:21.416768 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:53:21.416736 4405 patch_prober.go:29] interesting pod/e2e-worker-w6-ebb7fd-openshift-default-544f5768f-mngkl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused" start-of-body= Nov 04 02:53:21.416926 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:53:21.416796 4405 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-ebb7fd-openshift-default-544f5768f-mngkl" podUID="f55884d8-38e1-416b-853d-9c09082444bf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused" Nov 04 02:53:22.417693 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:53:22.417186 4405 patch_prober.go:29] interesting pod/e2e-worker-w6-ebb7fd-openshift-default-544f5768f-mngkl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused" start-of-body= Nov 04 02:53:22.417693 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:53:22.417253 4405 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-ebb7fd-openshift-default-544f5768f-mngkl" podUID="f55884d8-38e1-416b-853d-9c09082444bf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused" Nov 04 02:53:23.137761 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:53:23.137719 4405 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w0-fe4e0c/distinct-2-w0-fe4e0c-kserve-579bf4cc77-8pbq8"] Nov 04 02:53:23.141701 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:53:23.141662 4405 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-fe4e0c/distinct-2-w0-fe4e0c-kserve-579bf4cc77-8pbq8" Nov 04 02:53:23.249395 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:53:23.249354 4405 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2b2ab722-d1bb-4376-bede-f5f4ee4562aa-tmp-dir\") pod \"distinct-2-w0-fe4e0c-kserve-579bf4cc77-8pbq8\" (UID: \"2b2ab722-d1bb-4376-bede-f5f4ee4562aa\") " pod="e2e-models-e2e-worker-w0-fe4e0c/distinct-2-w0-fe4e0c-kserve-579bf4cc77-8pbq8" Nov 04 02:53:23.249603 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:53:23.249544 4405 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gdx2g\" (UniqueName: \"kubernetes.io/projected/2b2ab722-d1bb-4376-bede-f5f4ee4562aa-kube-api-access-gdx2g\") pod \"distinct-2-w0-fe4e0c-kserve-579bf4cc77-8pbq8\" (UID: \"2b2ab722-d1bb-4376-bede-f5f4ee4562aa\") " pod="e2e-models-e2e-worker-w0-fe4e0c/distinct-2-w0-fe4e0c-kserve-579bf4cc77-8pbq8" Nov 04 02:53:23.249653 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:53:23.249602 4405 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2b2ab722-d1bb-4376-bede-f5f4ee4562aa-dshm\") pod \"distinct-2-w0-fe4e0c-kserve-579bf4cc77-8pbq8\" (UID: \"2b2ab722-d1bb-4376-bede-f5f4ee4562aa\") " pod="e2e-models-e2e-worker-w0-fe4e0c/distinct-2-w0-fe4e0c-kserve-579bf4cc77-8pbq8" Nov 04 02:53:23.249740 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:53:23.249716 4405 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2b2ab722-d1bb-4376-bede-f5f4ee4562aa-tls-certs\") pod \"distinct-2-w0-fe4e0c-kserve-579bf4cc77-8pbq8\" (UID: \"2b2ab722-d1bb-4376-bede-f5f4ee4562aa\") " pod="e2e-models-e2e-worker-w0-fe4e0c/distinct-2-w0-fe4e0c-kserve-579bf4cc77-8pbq8" Nov 04 02:53:23.249779 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:53:23.249757 4405 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2b2ab722-d1bb-4376-bede-f5f4ee4562aa-model-cache\") pod \"distinct-2-w0-fe4e0c-kserve-579bf4cc77-8pbq8\" (UID: \"2b2ab722-d1bb-4376-bede-f5f4ee4562aa\") " pod="e2e-models-e2e-worker-w0-fe4e0c/distinct-2-w0-fe4e0c-kserve-579bf4cc77-8pbq8" Nov 04 02:53:23.249908 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:53:23.249893 4405 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2b2ab722-d1bb-4376-bede-f5f4ee4562aa-home\") pod \"distinct-2-w0-fe4e0c-kserve-579bf4cc77-8pbq8\" (UID: \"2b2ab722-d1bb-4376-bede-f5f4ee4562aa\") " pod="e2e-models-e2e-worker-w0-fe4e0c/distinct-2-w0-fe4e0c-kserve-579bf4cc77-8pbq8" Nov 04 02:53:23.270647 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:53:23.270611 4405 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-fe4e0c\"/\"openshift-service-ca.crt\"" Nov 04 02:53:23.270647 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:53:23.270610 4405 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-fe4e0c\"/\"kube-root-ca.crt\"" Nov 04 02:53:23.270888 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:53:23.270651 4405 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-fe4e0c\"/\"distinct-2-w0-fe4e0c-kserve-self-signed-certs\"" Nov 04 02:53:23.270888 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:53:23.270809 4405 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-fe4e0c\"/\"default-dockercfg-2rh2k\"" Nov 04 02:53:23.273702 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:53:23.273681 4405 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-fe4e0c/distinct-2-w0-fe4e0c-kserve-579bf4cc77-8pbq8"] Nov 04 02:53:23.358692 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:53:23.354548 4405 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2b2ab722-d1bb-4376-bede-f5f4ee4562aa-tmp-dir\") pod \"distinct-2-w0-fe4e0c-kserve-579bf4cc77-8pbq8\" (UID: \"2b2ab722-d1bb-4376-bede-f5f4ee4562aa\") " pod="e2e-models-e2e-worker-w0-fe4e0c/distinct-2-w0-fe4e0c-kserve-579bf4cc77-8pbq8" Nov 04 02:53:23.358692 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:53:23.354727 4405 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gdx2g\" (UniqueName: \"kubernetes.io/projected/2b2ab722-d1bb-4376-bede-f5f4ee4562aa-kube-api-access-gdx2g\") pod \"distinct-2-w0-fe4e0c-kserve-579bf4cc77-8pbq8\" (UID: \"2b2ab722-d1bb-4376-bede-f5f4ee4562aa\") " pod="e2e-models-e2e-worker-w0-fe4e0c/distinct-2-w0-fe4e0c-kserve-579bf4cc77-8pbq8" Nov 04 02:53:23.358692 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:53:23.354799 4405 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2b2ab722-d1bb-4376-bede-f5f4ee4562aa-dshm\") pod \"distinct-2-w0-fe4e0c-kserve-579bf4cc77-8pbq8\" (UID: \"2b2ab722-d1bb-4376-bede-f5f4ee4562aa\") " pod="e2e-models-e2e-worker-w0-fe4e0c/distinct-2-w0-fe4e0c-kserve-579bf4cc77-8pbq8" Nov 04 02:53:23.358692 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:53:23.354845 4405 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2b2ab722-d1bb-4376-bede-f5f4ee4562aa-tls-certs\") pod \"distinct-2-w0-fe4e0c-kserve-579bf4cc77-8pbq8\" (UID: \"2b2ab722-d1bb-4376-bede-f5f4ee4562aa\") " pod="e2e-models-e2e-worker-w0-fe4e0c/distinct-2-w0-fe4e0c-kserve-579bf4cc77-8pbq8" Nov 04 02:53:23.358692 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:53:23.354882 4405 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2b2ab722-d1bb-4376-bede-f5f4ee4562aa-model-cache\") pod \"distinct-2-w0-fe4e0c-kserve-579bf4cc77-8pbq8\" (UID: \"2b2ab722-d1bb-4376-bede-f5f4ee4562aa\") " pod="e2e-models-e2e-worker-w0-fe4e0c/distinct-2-w0-fe4e0c-kserve-579bf4cc77-8pbq8" Nov 04 02:53:23.358692 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:53:23.355065 4405 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2b2ab722-d1bb-4376-bede-f5f4ee4562aa-home\") pod \"distinct-2-w0-fe4e0c-kserve-579bf4cc77-8pbq8\" (UID: \"2b2ab722-d1bb-4376-bede-f5f4ee4562aa\") " pod="e2e-models-e2e-worker-w0-fe4e0c/distinct-2-w0-fe4e0c-kserve-579bf4cc77-8pbq8" Nov 04 02:53:23.358692 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:53:23.355578 4405 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2b2ab722-d1bb-4376-bede-f5f4ee4562aa-home\") pod \"distinct-2-w0-fe4e0c-kserve-579bf4cc77-8pbq8\" (UID: \"2b2ab722-d1bb-4376-bede-f5f4ee4562aa\") " pod="e2e-models-e2e-worker-w0-fe4e0c/distinct-2-w0-fe4e0c-kserve-579bf4cc77-8pbq8" Nov 04 02:53:23.358692 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:53:23.355871 4405 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2b2ab722-d1bb-4376-bede-f5f4ee4562aa-tmp-dir\") pod \"distinct-2-w0-fe4e0c-kserve-579bf4cc77-8pbq8\" (UID: \"2b2ab722-d1bb-4376-bede-f5f4ee4562aa\") " pod="e2e-models-e2e-worker-w0-fe4e0c/distinct-2-w0-fe4e0c-kserve-579bf4cc77-8pbq8" Nov 04 02:53:23.359906 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:53:23.359866 4405 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2b2ab722-d1bb-4376-bede-f5f4ee4562aa-dshm\") pod \"distinct-2-w0-fe4e0c-kserve-579bf4cc77-8pbq8\" (UID: \"2b2ab722-d1bb-4376-bede-f5f4ee4562aa\") " pod="e2e-models-e2e-worker-w0-fe4e0c/distinct-2-w0-fe4e0c-kserve-579bf4cc77-8pbq8" Nov 04 02:53:23.360110 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:53:23.360008 4405 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2b2ab722-d1bb-4376-bede-f5f4ee4562aa-model-cache\") pod \"distinct-2-w0-fe4e0c-kserve-579bf4cc77-8pbq8\" (UID: \"2b2ab722-d1bb-4376-bede-f5f4ee4562aa\") " pod="e2e-models-e2e-worker-w0-fe4e0c/distinct-2-w0-fe4e0c-kserve-579bf4cc77-8pbq8" Nov 04 02:53:23.367639 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:53:23.367586 4405 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2b2ab722-d1bb-4376-bede-f5f4ee4562aa-tls-certs\") pod \"distinct-2-w0-fe4e0c-kserve-579bf4cc77-8pbq8\" (UID: \"2b2ab722-d1bb-4376-bede-f5f4ee4562aa\") " pod="e2e-models-e2e-worker-w0-fe4e0c/distinct-2-w0-fe4e0c-kserve-579bf4cc77-8pbq8" Nov 04 02:53:23.428766 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:53:23.418198 4405 patch_prober.go:29] interesting pod/e2e-worker-w6-ebb7fd-openshift-default-544f5768f-mngkl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused" start-of-body= Nov 04 02:53:23.428766 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:53:23.418267 4405 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-ebb7fd-openshift-default-544f5768f-mngkl" podUID="f55884d8-38e1-416b-853d-9c09082444bf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused" Nov 04 02:53:23.428766 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:53:23.428740 4405 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gdx2g\" (UniqueName: \"kubernetes.io/projected/2b2ab722-d1bb-4376-bede-f5f4ee4562aa-kube-api-access-gdx2g\") pod \"distinct-2-w0-fe4e0c-kserve-579bf4cc77-8pbq8\" (UID: \"2b2ab722-d1bb-4376-bede-f5f4ee4562aa\") " pod="e2e-models-e2e-worker-w0-fe4e0c/distinct-2-w0-fe4e0c-kserve-579bf4cc77-8pbq8" Nov 04 02:53:23.455976 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:53:23.455933 4405 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-fe4e0c/distinct-2-w0-fe4e0c-kserve-579bf4cc77-8pbq8" Nov 04 02:53:23.945848 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:53:23.945815 4405 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-fe4e0c/distinct-2-w0-fe4e0c-kserve-579bf4cc77-8pbq8"] Nov 04 02:53:23.946518 ip-10-0-134-125 kubenswrapper[4405]: W1104 02:53:23.946494 4405 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod2b2ab722_d1bb_4376_bede_f5f4ee4562aa.slice/crio-4c0c112228670c01e1b4bea9d2afa162752248b29abe9da1d7bd83caa1a826c2 WatchSource:0}: Error finding container 4c0c112228670c01e1b4bea9d2afa162752248b29abe9da1d7bd83caa1a826c2: Status 404 returned error can't find the container with id 4c0c112228670c01e1b4bea9d2afa162752248b29abe9da1d7bd83caa1a826c2 Nov 04 02:53:24.268792 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:53:24.268699 4405 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-fe4e0c/distinct-2-w0-fe4e0c-kserve-579bf4cc77-8pbq8" event={"ID":"2b2ab722-d1bb-4376-bede-f5f4ee4562aa","Type":"ContainerStarted","Data":"f4e91b1a963343e6bc85130338df13366abb078cb3bcc2e9717901c36187b229"} Nov 04 02:53:24.268792 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:53:24.268743 4405 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-fe4e0c/distinct-2-w0-fe4e0c-kserve-579bf4cc77-8pbq8" event={"ID":"2b2ab722-d1bb-4376-bede-f5f4ee4562aa","Type":"ContainerStarted","Data":"4c0c112228670c01e1b4bea9d2afa162752248b29abe9da1d7bd83caa1a826c2"} Nov 04 02:53:24.386616 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:53:24.386562 4405 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w0-fe4e0c/distinct-2-w0-fe4e0c-kserve-579bf4cc77-8pbq8" podStartSLOduration=1.386545759 podStartE2EDuration="1.386545759s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 02:53:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 02:53:24.384420513 +0000 UTC m=+1250.815120633" watchObservedRunningTime="2026-11-04 02:53:24.386545759 +0000 UTC m=+1250.817245887" Nov 04 02:53:24.415141 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:53:24.415102 4405 patch_prober.go:29] interesting pod/e2e-worker-w6-ebb7fd-openshift-default-544f5768f-mngkl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused" start-of-body= Nov 04 02:53:24.415428 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:53:24.415399 4405 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-ebb7fd-openshift-default-544f5768f-mngkl" podUID="f55884d8-38e1-416b-853d-9c09082444bf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused" Nov 04 02:53:24.456273 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:53:24.456232 4405 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-fe4e0c/distinct-2-w0-fe4e0c-kserve-579bf4cc77-8pbq8" Nov 04 02:53:25.426696 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:53:25.417087 4405 patch_prober.go:29] interesting pod/e2e-worker-w6-ebb7fd-openshift-default-544f5768f-mngkl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused" start-of-body= Nov 04 02:53:25.426696 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:53:25.417147 4405 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-ebb7fd-openshift-default-544f5768f-mngkl" podUID="f55884d8-38e1-416b-853d-9c09082444bf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused" Nov 04 02:53:26.390716 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:53:26.386809 4405 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-e516ff/distinct-w1-e516ff-kserve-f4474cb9c-glpzw" podUID="8842ad09-22ff-4492-ba5d-d4cacf4eac75" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/health\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 04 02:53:26.416181 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:53:26.416138 4405 patch_prober.go:29] interesting pod/e2e-worker-w6-ebb7fd-openshift-default-544f5768f-mngkl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused" start-of-body= Nov 04 02:53:26.416466 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:53:26.416433 4405 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-ebb7fd-openshift-default-544f5768f-mngkl" podUID="f55884d8-38e1-416b-853d-9c09082444bf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused" Nov 04 02:53:26.475892 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:53:26.475825 4405 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-203c35/simulator-w4-203c35-kserve-f9c4bd695-999ns" podUID="eb0d8bdd-6808-4fd0-b9b5-567dff3e7ca5" containerName="main" probeResult="failure" output="Get \"https://10.134.0.46:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 02:53:26.665264 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:53:26.665160 4405 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-a0bcbc/premium-w2-a0bcbc-kserve-78dfd6d78c-lwth2" podUID="33b620c1-389c-48b4-b277-90252018c4b6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/ready\": context deadline exceeded" Nov 04 02:53:26.665895 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:53:26.665863 4405 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-a0bcbc/distinct-w2-a0bcbc-kserve-596f94c7cd-szl4g" podUID="bac32934-8a57-4919-b435-13f367cbea3e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": context deadline exceeded" Nov 04 02:53:26.881814 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:53:26.881765 4405 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-ebb7fd/distinct-2-w6-ebb7fd-kserve-6f68c5db86-zrpjs" podUID="b7b786e0-9c94-4126-9949-cd354ebbfb2a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.65:8000/ready\": context deadline exceeded" Nov 04 02:53:26.978745 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:53:26.974987 4405 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-ljhhw" podUID="f66b1861-d87b-4d57-ab27-f1a24544a534" containerName="main" probeResult="failure" output="Get \"https://10.134.0.26:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 02:53:27.080706 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:53:27.079279 4405 patch_prober.go:29] interesting pod/e2e-worker-w4-203c35-openshift-default-56497cdcc-2b66r container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.51:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 04 02:53:27.080706 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:53:27.079344 4405 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-203c35-openshift-default-56497cdcc-2b66r" podUID="147e25a7-dda1-4170-9453-44896e44e2bc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.51:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 04 02:53:27.101705 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:53:27.096185 4405 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-ebb7fd/distinct-w6-ebb7fd-kserve-55c47d8865-7z9pz" podUID="ccb8e181-5610-46da-bb8b-e0b07fa02ff1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": context deadline exceeded" Nov 04 02:53:27.195833 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:53:27.195775 4405 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-e516ff/distinct-w1-e516ff-kserve-f4474cb9c-glpzw" podUID="8842ad09-22ff-4492-ba5d-d4cacf4eac75" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 02:53:27.337694 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:53:27.332892 4405 prober.go:122] "Probe failed" probeType="Liveness" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-hhnmx" podUID="73142270-a8e7-4180-bfa5-6f720372736f" containerName="cert-manager-webhook" probeResult="failure" output="Get \"http://10.134.0.18:6080/livez\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 04 02:53:27.338423 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:53:27.338381 4405 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-cdcf67/distinct-w3-cdcf67-kserve-667cc7f7d7-5m2lq" podUID="52ed22cc-96e4-4ef0-b34d-ed58c89a9183" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 02:53:27.419145 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:53:27.419085 4405 patch_prober.go:29] interesting pod/e2e-worker-w6-ebb7fd-openshift-default-544f5768f-mngkl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused" start-of-body= Nov 04 02:53:27.419766 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:53:27.419729 4405 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-ebb7fd-openshift-default-544f5768f-mngkl" podUID="f55884d8-38e1-416b-853d-9c09082444bf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused" Nov 04 02:53:27.685755 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:53:27.666818 4405 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-a0bcbc/distinct-w2-a0bcbc-kserve-596f94c7cd-szl4g" podUID="bac32934-8a57-4919-b435-13f367cbea3e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 02:53:27.685755 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:53:27.667263 4405 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-a0bcbc/premium-w2-a0bcbc-kserve-78dfd6d78c-lwth2" podUID="33b620c1-389c-48b4-b277-90252018c4b6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/ready\": context deadline exceeded" Nov 04 02:53:27.685755 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:53:27.667863 4405 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-a0bcbc/distinct-w2-a0bcbc-kserve-596f94c7cd-szl4g" Nov 04 02:53:27.685755 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:53:27.667924 4405 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-a0bcbc/premium-w2-a0bcbc-kserve-78dfd6d78c-lwth2" Nov 04 02:53:27.917716 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:53:27.908778 4405 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-ebb7fd/distinct-2-w6-ebb7fd-kserve-6f68c5db86-zrpjs" podUID="b7b786e0-9c94-4126-9949-cd354ebbfb2a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 02:53:28.022038 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:53:28.021940 4405 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-ljhhw" podUID="f66b1861-d87b-4d57-ab27-f1a24544a534" containerName="main" probeResult="failure" output="Get \"https://10.134.0.26:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 02:53:28.378693 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:53:28.166823 4405 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-ebb7fd/distinct-w6-ebb7fd-kserve-55c47d8865-7z9pz" podUID="ccb8e181-5610-46da-bb8b-e0b07fa02ff1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 02:53:28.407003 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:53:28.193879 4405 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-a0bcbc/distinct-w2-a0bcbc-kserve-596f94c7cd-szl4g" podUID="bac32934-8a57-4919-b435-13f367cbea3e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 02:53:28.535326 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:53:28.200319 4405 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-e516ff/distinct-w1-e516ff-kserve-f4474cb9c-glpzw" podUID="8842ad09-22ff-4492-ba5d-d4cacf4eac75" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/ready\": context deadline exceeded" Nov 04 02:53:28.541840 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:53:28.212074 4405 patch_prober.go:29] interesting pod/e2e-worker-w1-e516ff-openshift-default-6f4b957485-xg2tv container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.60:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 04 02:53:28.550795 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:53:28.550747 4405 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-e516ff-openshift-default-6f4b957485-xg2tv" podUID="a4bcafa2-ec97-486c-819f-7be10e3110a9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.60:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 04 02:53:28.564690 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:53:28.365769 4405 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-cdcf67/distinct-w3-cdcf67-kserve-667cc7f7d7-5m2lq" podUID="52ed22cc-96e4-4ef0-b34d-ed58c89a9183" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 02:53:28.591839 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:53:28.509225 4405 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-203c35/simulator-w4-203c35-kserve-f9c4bd695-999ns" podUID="eb0d8bdd-6808-4fd0-b9b5-567dff3e7ca5" containerName="main" probeResult="failure" output="Get \"https://10.134.0.46:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 02:53:28.591839 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:53:28.287205 4405 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-ebb7fd/distinct-2-w6-ebb7fd-kserve-6f68c5db86-zrpjs" Nov 04 02:53:28.591839 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:53:28.590118 4405 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-ljhhw" Nov 04 02:53:28.591839 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:53:28.590174 4405 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-ebb7fd/distinct-w6-ebb7fd-kserve-55c47d8865-7z9pz" Nov 04 02:53:28.591839 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:53:28.590207 4405 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-e516ff/distinct-w1-e516ff-kserve-f4474cb9c-glpzw" Nov 04 02:53:28.591839 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:53:28.590239 4405 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-cdcf67/distinct-w3-cdcf67-kserve-667cc7f7d7-5m2lq" Nov 04 02:53:28.591839 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:53:28.215832 4405 patch_prober.go:29] interesting pod/e2e-worker-w4-203c35-openshift-default-56497cdcc-z59vn container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.61:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 04 02:53:28.591839 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:53:28.590296 4405 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-203c35-openshift-default-56497cdcc-z59vn" podUID="5ebc4fc9-8882-43c6-8713-1ee995a8ee00" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.61:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 04 02:53:28.591839 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:53:28.453846 4405 patch_prober.go:29] interesting pod/e2e-worker-w6-ebb7fd-openshift-default-544f5768f-mngkl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused" start-of-body= Nov 04 02:53:28.591839 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:53:28.590356 4405 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-ebb7fd-openshift-default-544f5768f-mngkl" podUID="f55884d8-38e1-416b-853d-9c09082444bf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused" Nov 04 02:53:28.702801 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:53:28.690547 4405 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-a0bcbc/premium-w2-a0bcbc-kserve-78dfd6d78c-lwth2" podUID="33b620c1-389c-48b4-b277-90252018c4b6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 02:53:28.768268 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:53:28.757869 4405 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-a0bcbc/distinct-w2-a0bcbc-kserve-596f94c7cd-szl4g" podUID="bac32934-8a57-4919-b435-13f367cbea3e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": context deadline exceeded" Nov 04 02:53:28.913104 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:53:28.912833 4405 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-ebb7fd/distinct-2-w6-ebb7fd-kserve-6f68c5db86-zrpjs" podUID="b7b786e0-9c94-4126-9949-cd354ebbfb2a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.65:8000/ready\": context deadline exceeded" Nov 04 02:53:28.927720 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:53:28.913724 4405 prober.go:122] "Probe failed" probeType="Liveness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-ljhhw" podUID="f66b1861-d87b-4d57-ab27-f1a24544a534" containerName="main" probeResult="failure" output="Get \"https://10.134.0.26:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 02:53:29.080417 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:53:29.080317 4405 prober.go:122] "Probe failed" probeType="Readiness" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-hhnmx" podUID="73142270-a8e7-4180-bfa5-6f720372736f" containerName="cert-manager-webhook" probeResult="failure" output="Get \"http://10.134.0.18:6080/healthz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 04 02:53:29.425694 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:53:29.420682 4405 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-ebb7fd/distinct-w6-ebb7fd-kserve-55c47d8865-7z9pz" podUID="ccb8e181-5610-46da-bb8b-e0b07fa02ff1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 02:53:29.475438 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:53:29.474248 4405 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-ljhhw" podUID="f66b1861-d87b-4d57-ab27-f1a24544a534" containerName="main" probeResult="failure" output="Get \"https://10.134.0.26:8000/ready\": context deadline exceeded" Nov 04 02:53:29.497633 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:53:29.497576 4405 patch_prober.go:29] interesting pod/e2e-worker-w6-ebb7fd-openshift-default-544f5768f-mngkl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused" start-of-body= Nov 04 02:53:29.497959 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:53:29.497908 4405 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-ebb7fd-openshift-default-544f5768f-mngkl" podUID="f55884d8-38e1-416b-853d-9c09082444bf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused" Nov 04 02:53:29.545806 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:53:29.545765 4405 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-e516ff/distinct-w1-e516ff-kserve-f4474cb9c-glpzw" podUID="8842ad09-22ff-4492-ba5d-d4cacf4eac75" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 02:53:29.598454 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:53:29.598407 4405 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-203c35/simulator-w4-203c35-kserve-f9c4bd695-999ns" podUID="eb0d8bdd-6808-4fd0-b9b5-567dff3e7ca5" containerName="main" probeResult="failure" output="Get \"https://10.134.0.46:8000/ready\": context deadline exceeded" Nov 04 02:53:29.618695 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:53:29.605022 4405 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-203c35/simulator-w4-203c35-kserve-f9c4bd695-999ns" Nov 04 02:53:29.618695 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:53:29.598751 4405 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-cdcf67/distinct-w3-cdcf67-kserve-667cc7f7d7-5m2lq" podUID="52ed22cc-96e4-4ef0-b34d-ed58c89a9183" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 02:53:29.795449 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:53:29.794783 4405 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-a0bcbc/premium-w2-a0bcbc-kserve-78dfd6d78c-lwth2" podUID="33b620c1-389c-48b4-b277-90252018c4b6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/ready\": context deadline exceeded" Nov 04 02:53:29.795449 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:53:29.795153 4405 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-a0bcbc/premium-w2-a0bcbc-kserve-78dfd6d78c-lwth2" podUID="33b620c1-389c-48b4-b277-90252018c4b6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 02:53:29.795449 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:53:29.795218 4405 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-a0bcbc/distinct-w2-a0bcbc-kserve-596f94c7cd-szl4g" podUID="bac32934-8a57-4919-b435-13f367cbea3e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": context deadline exceeded" Nov 04 02:53:29.810134 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:53:29.810097 4405 patch_prober.go:29] interesting pod/e2e-worker-w2-a0bcbc-openshift-default-65dc9d9b6c-fzwkp container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.66:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 04 02:53:29.810372 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:53:29.810343 4405 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-a0bcbc-openshift-default-65dc9d9b6c-fzwkp" podUID="700d7662-7d5b-40b0-afea-e55dc01922e6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.66:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 04 02:53:29.917739 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:53:29.910232 4405 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-cdcf67/distinct-w3-cdcf67-kserve-667cc7f7d7-5m2lq" podUID="52ed22cc-96e4-4ef0-b34d-ed58c89a9183" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 02:53:29.932696 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:53:29.929150 4405 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-ebb7fd/distinct-2-w6-ebb7fd-kserve-6f68c5db86-zrpjs" podUID="b7b786e0-9c94-4126-9949-cd354ebbfb2a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.65:8000/ready\": context deadline exceeded" Nov 04 02:53:30.417195 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:53:30.416999 4405 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-ebb7fd/distinct-w6-ebb7fd-kserve-55c47d8865-7z9pz" Nov 04 02:53:30.417409 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:53:30.417239 4405 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-cdcf67/distinct-w3-cdcf67-kserve-667cc7f7d7-5m2lq" Nov 04 02:53:30.479870 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:53:30.479795 4405 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-ljhhw" podUID="f66b1861-d87b-4d57-ab27-f1a24544a534" containerName="main" probeResult="failure" output="Get \"https://10.134.0.26:8000/ready\": context deadline exceeded" Nov 04 02:53:30.482487 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:53:30.482459 4405 patch_prober.go:29] interesting pod/e2e-worker-w6-ebb7fd-openshift-default-544f5768f-mngkl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused" start-of-body= Nov 04 02:53:30.482740 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:53:30.482715 4405 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-ebb7fd-openshift-default-544f5768f-mngkl" podUID="f55884d8-38e1-416b-853d-9c09082444bf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused" Nov 04 02:53:30.491334 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:53:30.491311 4405 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-203c35/simulator-w4-203c35-kserve-f9c4bd695-999ns" Nov 04 02:53:30.523746 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:53:30.522403 4405 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-ljhhw" Nov 04 02:53:30.523746 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:53:30.522464 4405 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-ebb7fd/distinct-2-w6-ebb7fd-kserve-6f68c5db86-zrpjs" Nov 04 02:53:30.526585 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:53:30.524943 4405 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-a0bcbc/distinct-w2-a0bcbc-kserve-596f94c7cd-szl4g" Nov 04 02:53:30.526585 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:53:30.525527 4405 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-e516ff/distinct-w1-e516ff-kserve-f4474cb9c-glpzw" Nov 04 02:53:30.526585 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:53:30.526061 4405 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-a0bcbc/premium-w2-a0bcbc-kserve-78dfd6d78c-lwth2" Nov 04 02:53:32.098374 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:53:32.098342 4405 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w6-ebb7fd-openshift-default-544f5768f-mngkl" Nov 04 02:53:32.098898 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:53:32.098708 4405 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w6-ebb7fd-openshift-default-544f5768f-mngkl" Nov 04 02:53:32.099830 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:53:32.099809 4405 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w6-ebb7fd-openshift-default-544f5768f-mngkl" Nov 04 02:53:33.456405 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:53:33.456372 4405 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w0-fe4e0c/distinct-2-w0-fe4e0c-kserve-579bf4cc77-8pbq8" Nov 04 02:53:33.469285 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:53:33.469261 4405 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w0-fe4e0c/distinct-2-w0-fe4e0c-kserve-579bf4cc77-8pbq8" Nov 04 02:53:33.618006 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:53:33.617974 4405 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-fe4e0c/distinct-2-w0-fe4e0c-kserve-579bf4cc77-8pbq8" Nov 04 02:53:36.883413 ip-10-0-134-125 kubenswrapper[4405]: E1104 02:53:36.883380 4405 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5f2vb" containerName="extractor" Nov 04 02:53:49.427612 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:53:49.427574 4405 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-e516ff-openshift-default-6f4b957485-w2plx"] Nov 04 02:53:49.433136 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:53:49.433113 4405 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-e516ff-openshift-default-6f4b957485-w2plx" Nov 04 02:53:49.541060 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:53:49.541010 4405 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a525d6f7-79b5-4409-8137-319328626c10-istio-envoy\") pod \"e2e-worker-w1-e516ff-openshift-default-6f4b957485-w2plx\" (UID: \"a525d6f7-79b5-4409-8137-319328626c10\") " pod="openshift-ingress/e2e-worker-w1-e516ff-openshift-default-6f4b957485-w2plx" Nov 04 02:53:49.541242 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:53:49.541201 4405 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a525d6f7-79b5-4409-8137-319328626c10-istiod-ca-cert\") pod \"e2e-worker-w1-e516ff-openshift-default-6f4b957485-w2plx\" (UID: \"a525d6f7-79b5-4409-8137-319328626c10\") " pod="openshift-ingress/e2e-worker-w1-e516ff-openshift-default-6f4b957485-w2plx" Nov 04 02:53:49.541367 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:53:49.541344 4405 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bnxc2\" (UniqueName: \"kubernetes.io/projected/a525d6f7-79b5-4409-8137-319328626c10-kube-api-access-bnxc2\") pod \"e2e-worker-w1-e516ff-openshift-default-6f4b957485-w2plx\" (UID: \"a525d6f7-79b5-4409-8137-319328626c10\") " pod="openshift-ingress/e2e-worker-w1-e516ff-openshift-default-6f4b957485-w2plx" Nov 04 02:53:49.541446 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:53:49.541410 4405 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a525d6f7-79b5-4409-8137-319328626c10-workload-socket\") pod \"e2e-worker-w1-e516ff-openshift-default-6f4b957485-w2plx\" (UID: \"a525d6f7-79b5-4409-8137-319328626c10\") " pod="openshift-ingress/e2e-worker-w1-e516ff-openshift-default-6f4b957485-w2plx" Nov 04 02:53:49.541524 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:53:49.541500 4405 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a525d6f7-79b5-4409-8137-319328626c10-credential-socket\") pod \"e2e-worker-w1-e516ff-openshift-default-6f4b957485-w2plx\" (UID: \"a525d6f7-79b5-4409-8137-319328626c10\") " pod="openshift-ingress/e2e-worker-w1-e516ff-openshift-default-6f4b957485-w2plx" Nov 04 02:53:49.541749 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:53:49.541729 4405 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a525d6f7-79b5-4409-8137-319328626c10-workload-certs\") pod \"e2e-worker-w1-e516ff-openshift-default-6f4b957485-w2plx\" (UID: \"a525d6f7-79b5-4409-8137-319328626c10\") " pod="openshift-ingress/e2e-worker-w1-e516ff-openshift-default-6f4b957485-w2plx" Nov 04 02:53:49.541839 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:53:49.541797 4405 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a525d6f7-79b5-4409-8137-319328626c10-istio-token\") pod \"e2e-worker-w1-e516ff-openshift-default-6f4b957485-w2plx\" (UID: \"a525d6f7-79b5-4409-8137-319328626c10\") " pod="openshift-ingress/e2e-worker-w1-e516ff-openshift-default-6f4b957485-w2plx" Nov 04 02:53:49.541931 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:53:49.541907 4405 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a525d6f7-79b5-4409-8137-319328626c10-istio-data\") pod \"e2e-worker-w1-e516ff-openshift-default-6f4b957485-w2plx\" (UID: \"a525d6f7-79b5-4409-8137-319328626c10\") " pod="openshift-ingress/e2e-worker-w1-e516ff-openshift-default-6f4b957485-w2plx" Nov 04 02:53:49.541985 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:53:49.541968 4405 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a525d6f7-79b5-4409-8137-319328626c10-istio-podinfo\") pod \"e2e-worker-w1-e516ff-openshift-default-6f4b957485-w2plx\" (UID: \"a525d6f7-79b5-4409-8137-319328626c10\") " pod="openshift-ingress/e2e-worker-w1-e516ff-openshift-default-6f4b957485-w2plx" Nov 04 02:53:49.636045 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:53:49.635998 4405 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-e516ff-openshift-default-6f4b957485-w2plx"] Nov 04 02:53:49.643658 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:53:49.643622 4405 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bnxc2\" (UniqueName: \"kubernetes.io/projected/a525d6f7-79b5-4409-8137-319328626c10-kube-api-access-bnxc2\") pod \"e2e-worker-w1-e516ff-openshift-default-6f4b957485-w2plx\" (UID: \"a525d6f7-79b5-4409-8137-319328626c10\") " pod="openshift-ingress/e2e-worker-w1-e516ff-openshift-default-6f4b957485-w2plx" Nov 04 02:53:49.643856 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:53:49.643711 4405 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a525d6f7-79b5-4409-8137-319328626c10-workload-socket\") pod \"e2e-worker-w1-e516ff-openshift-default-6f4b957485-w2plx\" (UID: \"a525d6f7-79b5-4409-8137-319328626c10\") " pod="openshift-ingress/e2e-worker-w1-e516ff-openshift-default-6f4b957485-w2plx" Nov 04 02:53:49.643856 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:53:49.643757 4405 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a525d6f7-79b5-4409-8137-319328626c10-credential-socket\") pod \"e2e-worker-w1-e516ff-openshift-default-6f4b957485-w2plx\" (UID: \"a525d6f7-79b5-4409-8137-319328626c10\") " pod="openshift-ingress/e2e-worker-w1-e516ff-openshift-default-6f4b957485-w2plx" Nov 04 02:53:49.643856 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:53:49.643796 4405 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a525d6f7-79b5-4409-8137-319328626c10-workload-certs\") pod \"e2e-worker-w1-e516ff-openshift-default-6f4b957485-w2plx\" (UID: \"a525d6f7-79b5-4409-8137-319328626c10\") " pod="openshift-ingress/e2e-worker-w1-e516ff-openshift-default-6f4b957485-w2plx" Nov 04 02:53:49.644039 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:53:49.643856 4405 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a525d6f7-79b5-4409-8137-319328626c10-istio-token\") pod \"e2e-worker-w1-e516ff-openshift-default-6f4b957485-w2plx\" (UID: \"a525d6f7-79b5-4409-8137-319328626c10\") " pod="openshift-ingress/e2e-worker-w1-e516ff-openshift-default-6f4b957485-w2plx" Nov 04 02:53:49.644039 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:53:49.643899 4405 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a525d6f7-79b5-4409-8137-319328626c10-istio-data\") pod \"e2e-worker-w1-e516ff-openshift-default-6f4b957485-w2plx\" (UID: \"a525d6f7-79b5-4409-8137-319328626c10\") " pod="openshift-ingress/e2e-worker-w1-e516ff-openshift-default-6f4b957485-w2plx" Nov 04 02:53:49.644039 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:53:49.643923 4405 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a525d6f7-79b5-4409-8137-319328626c10-istio-podinfo\") pod \"e2e-worker-w1-e516ff-openshift-default-6f4b957485-w2plx\" (UID: \"a525d6f7-79b5-4409-8137-319328626c10\") " pod="openshift-ingress/e2e-worker-w1-e516ff-openshift-default-6f4b957485-w2plx" Nov 04 02:53:49.644039 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:53:49.644000 4405 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a525d6f7-79b5-4409-8137-319328626c10-istio-envoy\") pod \"e2e-worker-w1-e516ff-openshift-default-6f4b957485-w2plx\" (UID: \"a525d6f7-79b5-4409-8137-319328626c10\") " pod="openshift-ingress/e2e-worker-w1-e516ff-openshift-default-6f4b957485-w2plx" Nov 04 02:53:49.644246 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:53:49.644119 4405 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a525d6f7-79b5-4409-8137-319328626c10-istiod-ca-cert\") pod \"e2e-worker-w1-e516ff-openshift-default-6f4b957485-w2plx\" (UID: \"a525d6f7-79b5-4409-8137-319328626c10\") " pod="openshift-ingress/e2e-worker-w1-e516ff-openshift-default-6f4b957485-w2plx" Nov 04 02:53:49.645066 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:53:49.645039 4405 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a525d6f7-79b5-4409-8137-319328626c10-istiod-ca-cert\") pod \"e2e-worker-w1-e516ff-openshift-default-6f4b957485-w2plx\" (UID: \"a525d6f7-79b5-4409-8137-319328626c10\") " pod="openshift-ingress/e2e-worker-w1-e516ff-openshift-default-6f4b957485-w2plx" Nov 04 02:53:49.645647 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:53:49.645623 4405 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a525d6f7-79b5-4409-8137-319328626c10-workload-socket\") pod \"e2e-worker-w1-e516ff-openshift-default-6f4b957485-w2plx\" (UID: \"a525d6f7-79b5-4409-8137-319328626c10\") " pod="openshift-ingress/e2e-worker-w1-e516ff-openshift-default-6f4b957485-w2plx" Nov 04 02:53:49.645911 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:53:49.645889 4405 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a525d6f7-79b5-4409-8137-319328626c10-credential-socket\") pod \"e2e-worker-w1-e516ff-openshift-default-6f4b957485-w2plx\" (UID: \"a525d6f7-79b5-4409-8137-319328626c10\") " pod="openshift-ingress/e2e-worker-w1-e516ff-openshift-default-6f4b957485-w2plx" Nov 04 02:53:49.646270 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:53:49.646240 4405 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a525d6f7-79b5-4409-8137-319328626c10-istio-data\") pod \"e2e-worker-w1-e516ff-openshift-default-6f4b957485-w2plx\" (UID: \"a525d6f7-79b5-4409-8137-319328626c10\") " pod="openshift-ingress/e2e-worker-w1-e516ff-openshift-default-6f4b957485-w2plx" Nov 04 02:53:49.646536 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:53:49.646513 4405 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a525d6f7-79b5-4409-8137-319328626c10-workload-certs\") pod \"e2e-worker-w1-e516ff-openshift-default-6f4b957485-w2plx\" (UID: \"a525d6f7-79b5-4409-8137-319328626c10\") " pod="openshift-ingress/e2e-worker-w1-e516ff-openshift-default-6f4b957485-w2plx" Nov 04 02:53:49.649827 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:53:49.649779 4405 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a525d6f7-79b5-4409-8137-319328626c10-istio-envoy\") pod \"e2e-worker-w1-e516ff-openshift-default-6f4b957485-w2plx\" (UID: \"a525d6f7-79b5-4409-8137-319328626c10\") " pod="openshift-ingress/e2e-worker-w1-e516ff-openshift-default-6f4b957485-w2plx" Nov 04 02:53:49.658521 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:53:49.658489 4405 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a525d6f7-79b5-4409-8137-319328626c10-istio-podinfo\") pod \"e2e-worker-w1-e516ff-openshift-default-6f4b957485-w2plx\" (UID: \"a525d6f7-79b5-4409-8137-319328626c10\") " pod="openshift-ingress/e2e-worker-w1-e516ff-openshift-default-6f4b957485-w2plx" Nov 04 02:53:49.663232 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:53:49.663207 4405 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-203c35-openshift-default-56497cdcc-w6c48"] Nov 04 02:53:49.667849 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:53:49.667824 4405 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-203c35-openshift-default-56497cdcc-w6c48" Nov 04 02:53:49.705781 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:53:49.705749 4405 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bnxc2\" (UniqueName: \"kubernetes.io/projected/a525d6f7-79b5-4409-8137-319328626c10-kube-api-access-bnxc2\") pod \"e2e-worker-w1-e516ff-openshift-default-6f4b957485-w2plx\" (UID: \"a525d6f7-79b5-4409-8137-319328626c10\") " pod="openshift-ingress/e2e-worker-w1-e516ff-openshift-default-6f4b957485-w2plx" Nov 04 02:53:49.706225 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:53:49.706205 4405 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a525d6f7-79b5-4409-8137-319328626c10-istio-token\") pod \"e2e-worker-w1-e516ff-openshift-default-6f4b957485-w2plx\" (UID: \"a525d6f7-79b5-4409-8137-319328626c10\") " pod="openshift-ingress/e2e-worker-w1-e516ff-openshift-default-6f4b957485-w2plx" Nov 04 02:53:49.751106 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:53:49.750903 4405 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-203c35-openshift-default-56497cdcc-w6c48"] Nov 04 02:53:49.752846 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:53:49.751419 4405 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-e516ff-openshift-default-6f4b957485-w2plx" Nov 04 02:53:49.846926 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:53:49.846891 4405 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/89ddc69b-94de-4443-a62a-fa5d360aa368-istio-podinfo\") pod \"e2e-worker-w4-203c35-openshift-default-56497cdcc-w6c48\" (UID: \"89ddc69b-94de-4443-a62a-fa5d360aa368\") " pod="openshift-ingress/e2e-worker-w4-203c35-openshift-default-56497cdcc-w6c48" Nov 04 02:53:49.847061 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:53:49.846951 4405 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/89ddc69b-94de-4443-a62a-fa5d360aa368-credential-socket\") pod \"e2e-worker-w4-203c35-openshift-default-56497cdcc-w6c48\" (UID: \"89ddc69b-94de-4443-a62a-fa5d360aa368\") " pod="openshift-ingress/e2e-worker-w4-203c35-openshift-default-56497cdcc-w6c48" Nov 04 02:53:49.847061 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:53:49.846983 4405 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/89ddc69b-94de-4443-a62a-fa5d360aa368-istio-token\") pod \"e2e-worker-w4-203c35-openshift-default-56497cdcc-w6c48\" (UID: \"89ddc69b-94de-4443-a62a-fa5d360aa368\") " pod="openshift-ingress/e2e-worker-w4-203c35-openshift-default-56497cdcc-w6c48" Nov 04 02:53:49.847061 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:53:49.847003 4405 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/89ddc69b-94de-4443-a62a-fa5d360aa368-workload-certs\") pod \"e2e-worker-w4-203c35-openshift-default-56497cdcc-w6c48\" (UID: \"89ddc69b-94de-4443-a62a-fa5d360aa368\") " pod="openshift-ingress/e2e-worker-w4-203c35-openshift-default-56497cdcc-w6c48" Nov 04 02:53:49.847061 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:53:49.847025 4405 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/89ddc69b-94de-4443-a62a-fa5d360aa368-workload-socket\") pod \"e2e-worker-w4-203c35-openshift-default-56497cdcc-w6c48\" (UID: \"89ddc69b-94de-4443-a62a-fa5d360aa368\") " pod="openshift-ingress/e2e-worker-w4-203c35-openshift-default-56497cdcc-w6c48" Nov 04 02:53:49.847061 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:53:49.847060 4405 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/89ddc69b-94de-4443-a62a-fa5d360aa368-istio-data\") pod \"e2e-worker-w4-203c35-openshift-default-56497cdcc-w6c48\" (UID: \"89ddc69b-94de-4443-a62a-fa5d360aa368\") " pod="openshift-ingress/e2e-worker-w4-203c35-openshift-default-56497cdcc-w6c48" Nov 04 02:53:49.847224 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:53:49.847156 4405 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qlgg7\" (UniqueName: \"kubernetes.io/projected/89ddc69b-94de-4443-a62a-fa5d360aa368-kube-api-access-qlgg7\") pod \"e2e-worker-w4-203c35-openshift-default-56497cdcc-w6c48\" (UID: \"89ddc69b-94de-4443-a62a-fa5d360aa368\") " pod="openshift-ingress/e2e-worker-w4-203c35-openshift-default-56497cdcc-w6c48" Nov 04 02:53:49.847224 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:53:49.847200 4405 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/89ddc69b-94de-4443-a62a-fa5d360aa368-istio-envoy\") pod \"e2e-worker-w4-203c35-openshift-default-56497cdcc-w6c48\" (UID: \"89ddc69b-94de-4443-a62a-fa5d360aa368\") " pod="openshift-ingress/e2e-worker-w4-203c35-openshift-default-56497cdcc-w6c48" Nov 04 02:53:49.847285 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:53:49.847229 4405 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/89ddc69b-94de-4443-a62a-fa5d360aa368-istiod-ca-cert\") pod \"e2e-worker-w4-203c35-openshift-default-56497cdcc-w6c48\" (UID: \"89ddc69b-94de-4443-a62a-fa5d360aa368\") " pod="openshift-ingress/e2e-worker-w4-203c35-openshift-default-56497cdcc-w6c48" Nov 04 02:53:49.948187 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:53:49.948104 4405 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/89ddc69b-94de-4443-a62a-fa5d360aa368-istio-token\") pod \"e2e-worker-w4-203c35-openshift-default-56497cdcc-w6c48\" (UID: \"89ddc69b-94de-4443-a62a-fa5d360aa368\") " pod="openshift-ingress/e2e-worker-w4-203c35-openshift-default-56497cdcc-w6c48" Nov 04 02:53:49.948187 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:53:49.948147 4405 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/89ddc69b-94de-4443-a62a-fa5d360aa368-workload-certs\") pod \"e2e-worker-w4-203c35-openshift-default-56497cdcc-w6c48\" (UID: \"89ddc69b-94de-4443-a62a-fa5d360aa368\") " pod="openshift-ingress/e2e-worker-w4-203c35-openshift-default-56497cdcc-w6c48" Nov 04 02:53:49.948187 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:53:49.948170 4405 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/89ddc69b-94de-4443-a62a-fa5d360aa368-workload-socket\") pod \"e2e-worker-w4-203c35-openshift-default-56497cdcc-w6c48\" (UID: \"89ddc69b-94de-4443-a62a-fa5d360aa368\") " pod="openshift-ingress/e2e-worker-w4-203c35-openshift-default-56497cdcc-w6c48" Nov 04 02:53:49.948471 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:53:49.948209 4405 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/89ddc69b-94de-4443-a62a-fa5d360aa368-istio-data\") pod \"e2e-worker-w4-203c35-openshift-default-56497cdcc-w6c48\" (UID: \"89ddc69b-94de-4443-a62a-fa5d360aa368\") " pod="openshift-ingress/e2e-worker-w4-203c35-openshift-default-56497cdcc-w6c48" Nov 04 02:53:49.948471 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:53:49.948319 4405 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qlgg7\" (UniqueName: \"kubernetes.io/projected/89ddc69b-94de-4443-a62a-fa5d360aa368-kube-api-access-qlgg7\") pod \"e2e-worker-w4-203c35-openshift-default-56497cdcc-w6c48\" (UID: \"89ddc69b-94de-4443-a62a-fa5d360aa368\") " pod="openshift-ingress/e2e-worker-w4-203c35-openshift-default-56497cdcc-w6c48" Nov 04 02:53:49.948471 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:53:49.948391 4405 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/89ddc69b-94de-4443-a62a-fa5d360aa368-istio-envoy\") pod \"e2e-worker-w4-203c35-openshift-default-56497cdcc-w6c48\" (UID: \"89ddc69b-94de-4443-a62a-fa5d360aa368\") " pod="openshift-ingress/e2e-worker-w4-203c35-openshift-default-56497cdcc-w6c48" Nov 04 02:53:49.948471 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:53:49.948434 4405 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/89ddc69b-94de-4443-a62a-fa5d360aa368-istiod-ca-cert\") pod \"e2e-worker-w4-203c35-openshift-default-56497cdcc-w6c48\" (UID: \"89ddc69b-94de-4443-a62a-fa5d360aa368\") " pod="openshift-ingress/e2e-worker-w4-203c35-openshift-default-56497cdcc-w6c48" Nov 04 02:53:49.948731 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:53:49.948478 4405 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/89ddc69b-94de-4443-a62a-fa5d360aa368-istio-podinfo\") pod \"e2e-worker-w4-203c35-openshift-default-56497cdcc-w6c48\" (UID: \"89ddc69b-94de-4443-a62a-fa5d360aa368\") " pod="openshift-ingress/e2e-worker-w4-203c35-openshift-default-56497cdcc-w6c48" Nov 04 02:53:49.948731 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:53:49.948522 4405 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/89ddc69b-94de-4443-a62a-fa5d360aa368-workload-certs\") pod \"e2e-worker-w4-203c35-openshift-default-56497cdcc-w6c48\" (UID: \"89ddc69b-94de-4443-a62a-fa5d360aa368\") " pod="openshift-ingress/e2e-worker-w4-203c35-openshift-default-56497cdcc-w6c48" Nov 04 02:53:49.948731 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:53:49.948531 4405 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/89ddc69b-94de-4443-a62a-fa5d360aa368-credential-socket\") pod \"e2e-worker-w4-203c35-openshift-default-56497cdcc-w6c48\" (UID: \"89ddc69b-94de-4443-a62a-fa5d360aa368\") " pod="openshift-ingress/e2e-worker-w4-203c35-openshift-default-56497cdcc-w6c48" Nov 04 02:53:49.948731 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:53:49.948619 4405 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/89ddc69b-94de-4443-a62a-fa5d360aa368-workload-socket\") pod \"e2e-worker-w4-203c35-openshift-default-56497cdcc-w6c48\" (UID: \"89ddc69b-94de-4443-a62a-fa5d360aa368\") " pod="openshift-ingress/e2e-worker-w4-203c35-openshift-default-56497cdcc-w6c48" Nov 04 02:53:49.949102 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:53:49.949079 4405 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/89ddc69b-94de-4443-a62a-fa5d360aa368-istio-data\") pod \"e2e-worker-w4-203c35-openshift-default-56497cdcc-w6c48\" (UID: \"89ddc69b-94de-4443-a62a-fa5d360aa368\") " pod="openshift-ingress/e2e-worker-w4-203c35-openshift-default-56497cdcc-w6c48" Nov 04 02:53:49.949217 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:53:49.949125 4405 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/89ddc69b-94de-4443-a62a-fa5d360aa368-credential-socket\") pod \"e2e-worker-w4-203c35-openshift-default-56497cdcc-w6c48\" (UID: \"89ddc69b-94de-4443-a62a-fa5d360aa368\") " pod="openshift-ingress/e2e-worker-w4-203c35-openshift-default-56497cdcc-w6c48" Nov 04 02:53:49.949268 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:53:49.949239 4405 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/89ddc69b-94de-4443-a62a-fa5d360aa368-istiod-ca-cert\") pod \"e2e-worker-w4-203c35-openshift-default-56497cdcc-w6c48\" (UID: \"89ddc69b-94de-4443-a62a-fa5d360aa368\") " pod="openshift-ingress/e2e-worker-w4-203c35-openshift-default-56497cdcc-w6c48" Nov 04 02:53:49.951388 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:53:49.951365 4405 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/89ddc69b-94de-4443-a62a-fa5d360aa368-istio-envoy\") pod \"e2e-worker-w4-203c35-openshift-default-56497cdcc-w6c48\" (UID: \"89ddc69b-94de-4443-a62a-fa5d360aa368\") " pod="openshift-ingress/e2e-worker-w4-203c35-openshift-default-56497cdcc-w6c48" Nov 04 02:53:49.951595 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:53:49.951576 4405 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/89ddc69b-94de-4443-a62a-fa5d360aa368-istio-podinfo\") pod \"e2e-worker-w4-203c35-openshift-default-56497cdcc-w6c48\" (UID: \"89ddc69b-94de-4443-a62a-fa5d360aa368\") " pod="openshift-ingress/e2e-worker-w4-203c35-openshift-default-56497cdcc-w6c48" Nov 04 02:53:50.008977 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:53:50.008942 4405 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/89ddc69b-94de-4443-a62a-fa5d360aa368-istio-token\") pod \"e2e-worker-w4-203c35-openshift-default-56497cdcc-w6c48\" (UID: \"89ddc69b-94de-4443-a62a-fa5d360aa368\") " pod="openshift-ingress/e2e-worker-w4-203c35-openshift-default-56497cdcc-w6c48" Nov 04 02:53:50.009136 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:53:50.008955 4405 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qlgg7\" (UniqueName: \"kubernetes.io/projected/89ddc69b-94de-4443-a62a-fa5d360aa368-kube-api-access-qlgg7\") pod \"e2e-worker-w4-203c35-openshift-default-56497cdcc-w6c48\" (UID: \"89ddc69b-94de-4443-a62a-fa5d360aa368\") " pod="openshift-ingress/e2e-worker-w4-203c35-openshift-default-56497cdcc-w6c48" Nov 04 02:53:50.195613 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:53:50.195576 4405 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-e516ff-openshift-default-6f4b957485-w2plx"] Nov 04 02:53:50.196077 ip-10-0-134-125 kubenswrapper[4405]: W1104 02:53:50.196048 4405 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda525d6f7_79b5_4409_8137_319328626c10.slice/crio-1f083139fb3a8410adcf1be621e2a23358604bd623962c53393d36ffee309259 WatchSource:0}: Error finding container 1f083139fb3a8410adcf1be621e2a23358604bd623962c53393d36ffee309259: Status 404 returned error can't find the container with id 1f083139fb3a8410adcf1be621e2a23358604bd623962c53393d36ffee309259 Nov 04 02:53:50.199106 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:53:50.199033 4405 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 04 02:53:50.199176 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:53:50.199119 4405 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 04 02:53:50.199176 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:53:50.199162 4405 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 04 02:53:50.293097 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:53:50.293062 4405 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-203c35-openshift-default-56497cdcc-w6c48" Nov 04 02:53:50.733065 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:53:50.732996 4405 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-203c35-openshift-default-56497cdcc-w6c48"] Nov 04 02:53:50.735182 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:53:50.735152 4405 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-e516ff-openshift-default-6f4b957485-w2plx" event={"ID":"a525d6f7-79b5-4409-8137-319328626c10","Type":"ContainerStarted","Data":"48cd8b87ccdd12489dc788842979decee8e4e3f709d453c6c0c8b9d98ec655f4"} Nov 04 02:53:50.735299 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:53:50.735200 4405 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-e516ff-openshift-default-6f4b957485-w2plx" event={"ID":"a525d6f7-79b5-4409-8137-319328626c10","Type":"ContainerStarted","Data":"1f083139fb3a8410adcf1be621e2a23358604bd623962c53393d36ffee309259"} Nov 04 02:53:50.735618 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:53:50.735577 4405 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 04 02:53:50.735771 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:53:50.735662 4405 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 04 02:53:50.735771 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:53:50.735723 4405 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 04 02:53:50.752029 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:53:50.752003 4405 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-e516ff-openshift-default-6f4b957485-w2plx" Nov 04 02:53:51.754144 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:53:51.754105 4405 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-203c35-openshift-default-56497cdcc-w6c48" event={"ID":"89ddc69b-94de-4443-a62a-fa5d360aa368","Type":"ContainerStarted","Data":"9c709331c40b10e0a93eacdbe663fc93145c774d2e72f230525dbf41228825c3"} Nov 04 02:53:51.754610 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:53:51.754592 4405 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-203c35-openshift-default-56497cdcc-w6c48" event={"ID":"89ddc69b-94de-4443-a62a-fa5d360aa368","Type":"ContainerStarted","Data":"ecde9492b94a720eaedb5dfa26aa3cef4a4e26b55be9b0fd80ac46517ed34105"} Nov 04 02:53:51.755124 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:53:51.755102 4405 patch_prober.go:29] interesting pod/e2e-worker-w1-e516ff-openshift-default-6f4b957485-w2plx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused" start-of-body= Nov 04 02:53:51.755224 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:53:51.755153 4405 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-e516ff-openshift-default-6f4b957485-w2plx" podUID="a525d6f7-79b5-4409-8137-319328626c10" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused" Nov 04 02:53:51.881225 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:53:51.880851 4405 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-e516ff-openshift-default-6f4b957485-w2plx" podStartSLOduration=2.8808329969999997 podStartE2EDuration="2.880832997s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 02:53:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 02:53:50.881396503 +0000 UTC m=+1277.312096624" watchObservedRunningTime="2026-11-04 02:53:51.880832997 +0000 UTC m=+1278.311533118" Nov 04 02:53:51.881489 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:53:51.881454 4405 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-203c35-openshift-default-56497cdcc-w6c48" podStartSLOduration=2.881444594 podStartE2EDuration="2.881444594s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 02:53:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 02:53:51.879924787 +0000 UTC m=+1278.310624910" watchObservedRunningTime="2026-11-04 02:53:51.881444594 +0000 UTC m=+1278.312144713" Nov 04 02:53:52.299693 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:53:52.299334 4405 patch_prober.go:29] interesting pod/e2e-worker-w4-203c35-openshift-default-56497cdcc-w6c48 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused" start-of-body= Nov 04 02:53:52.299693 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:53:52.299415 4405 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-203c35-openshift-default-56497cdcc-w6c48" podUID="89ddc69b-94de-4443-a62a-fa5d360aa368" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused" Nov 04 02:53:52.299997 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:53:52.299726 4405 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-203c35-openshift-default-56497cdcc-w6c48" Nov 04 02:53:52.756750 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:53:52.752079 4405 patch_prober.go:29] interesting pod/e2e-worker-w1-e516ff-openshift-default-6f4b957485-w2plx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused" start-of-body= Nov 04 02:53:52.756750 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:53:52.752151 4405 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-e516ff-openshift-default-6f4b957485-w2plx" podUID="a525d6f7-79b5-4409-8137-319328626c10" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused" Nov 04 02:53:53.298695 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:53:53.296074 4405 patch_prober.go:29] interesting pod/e2e-worker-w4-203c35-openshift-default-56497cdcc-w6c48 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused" start-of-body= Nov 04 02:53:53.298695 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:53:53.296136 4405 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-203c35-openshift-default-56497cdcc-w6c48" podUID="89ddc69b-94de-4443-a62a-fa5d360aa368" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused" Nov 04 02:53:53.752694 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:53:53.752158 4405 patch_prober.go:29] interesting pod/e2e-worker-w1-e516ff-openshift-default-6f4b957485-w2plx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused" start-of-body= Nov 04 02:53:53.752694 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:53:53.752229 4405 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-e516ff-openshift-default-6f4b957485-w2plx" podUID="a525d6f7-79b5-4409-8137-319328626c10" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused" Nov 04 02:53:54.297694 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:53:54.296247 4405 patch_prober.go:29] interesting pod/e2e-worker-w4-203c35-openshift-default-56497cdcc-w6c48 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused" start-of-body= Nov 04 02:53:54.297694 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:53:54.296346 4405 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-203c35-openshift-default-56497cdcc-w6c48" podUID="89ddc69b-94de-4443-a62a-fa5d360aa368" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused" Nov 04 02:53:54.752692 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:53:54.751968 4405 patch_prober.go:29] interesting pod/e2e-worker-w1-e516ff-openshift-default-6f4b957485-w2plx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused" start-of-body= Nov 04 02:53:54.752692 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:53:54.752047 4405 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-e516ff-openshift-default-6f4b957485-w2plx" podUID="a525d6f7-79b5-4409-8137-319328626c10" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused" Nov 04 02:53:55.297697 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:53:55.296078 4405 patch_prober.go:29] interesting pod/e2e-worker-w4-203c35-openshift-default-56497cdcc-w6c48 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused" start-of-body= Nov 04 02:53:55.297697 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:53:55.296154 4405 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-203c35-openshift-default-56497cdcc-w6c48" podUID="89ddc69b-94de-4443-a62a-fa5d360aa368" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused" Nov 04 02:53:55.758537 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:53:55.756109 4405 patch_prober.go:29] interesting pod/e2e-worker-w1-e516ff-openshift-default-6f4b957485-w2plx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused" start-of-body= Nov 04 02:53:55.758537 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:53:55.756169 4405 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-e516ff-openshift-default-6f4b957485-w2plx" podUID="a525d6f7-79b5-4409-8137-319328626c10" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused" Nov 04 02:53:56.309694 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:53:56.303033 4405 patch_prober.go:29] interesting pod/e2e-worker-w4-203c35-openshift-default-56497cdcc-w6c48 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused" start-of-body= Nov 04 02:53:56.309694 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:53:56.303094 4405 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-203c35-openshift-default-56497cdcc-w6c48" podUID="89ddc69b-94de-4443-a62a-fa5d360aa368" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused" Nov 04 02:53:56.752730 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:53:56.752694 4405 patch_prober.go:29] interesting pod/e2e-worker-w1-e516ff-openshift-default-6f4b957485-w2plx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused" start-of-body= Nov 04 02:53:56.752917 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:53:56.752753 4405 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-e516ff-openshift-default-6f4b957485-w2plx" podUID="a525d6f7-79b5-4409-8137-319328626c10" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused" Nov 04 02:53:57.296193 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:53:57.296152 4405 patch_prober.go:29] interesting pod/e2e-worker-w4-203c35-openshift-default-56497cdcc-w6c48 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused" start-of-body= Nov 04 02:53:57.296492 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:53:57.296463 4405 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-203c35-openshift-default-56497cdcc-w6c48" podUID="89ddc69b-94de-4443-a62a-fa5d360aa368" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused" Nov 04 02:53:58.287853 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:53:58.287811 4405 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-e516ff-openshift-default-6f4b957485-w2plx" Nov 04 02:53:58.289381 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:53:58.289357 4405 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-e516ff-openshift-default-6f4b957485-w2plx" Nov 04 02:53:58.290201 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:53:58.290180 4405 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-e516ff-openshift-default-6f4b957485-w2plx" Nov 04 02:53:58.293707 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:53:58.293651 4405 patch_prober.go:29] interesting pod/e2e-worker-w4-203c35-openshift-default-56497cdcc-w6c48 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused" start-of-body= Nov 04 02:53:58.293819 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:53:58.293725 4405 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-203c35-openshift-default-56497cdcc-w6c48" podUID="89ddc69b-94de-4443-a62a-fa5d360aa368" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused" Nov 04 02:53:59.300092 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:53:59.300009 4405 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-203c35-openshift-default-56497cdcc-w6c48" Nov 04 02:53:59.300473 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:53:59.300418 4405 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-203c35-openshift-default-56497cdcc-w6c48" Nov 04 02:53:59.301270 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:53:59.301251 4405 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-203c35-openshift-default-56497cdcc-w6c48" Nov 04 02:53:59.959944 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:53:59.959911 4405 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-bl5mg"] Nov 04 02:54:02.406694 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:54:02.405795 4405 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-ebb7fd/distinct-w6-ebb7fd-kserve-55c47d8865-7z9pz" podUID="ccb8e181-5610-46da-bb8b-e0b07fa02ff1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 02:54:02.519284 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:54:02.519231 4405 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-cdcf67/distinct-w3-cdcf67-kserve-667cc7f7d7-5m2lq" podUID="52ed22cc-96e4-4ef0-b34d-ed58c89a9183" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 02:54:02.520887 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:54:02.520853 4405 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-ebb7fd/distinct-2-w6-ebb7fd-kserve-6f68c5db86-zrpjs" podUID="b7b786e0-9c94-4126-9949-cd354ebbfb2a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 02:54:02.547844 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:54:02.547773 4405 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-e516ff/distinct-w1-e516ff-kserve-f4474cb9c-glpzw" podUID="8842ad09-22ff-4492-ba5d-d4cacf4eac75" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 02:54:02.576695 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:54:02.572802 4405 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-203c35/simulator-w4-203c35-kserve-f9c4bd695-999ns" podUID="eb0d8bdd-6808-4fd0-b9b5-567dff3e7ca5" containerName="main" probeResult="failure" output="Get \"https://10.134.0.46:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 02:54:02.630597 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:54:02.614778 4405 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-fe4e0c/distinct-2-w0-fe4e0c-kserve-579bf4cc77-8pbq8" podUID="2b2ab722-d1bb-4376-bede-f5f4ee4562aa" containerName="main" probeResult="failure" output="Get \"https://10.134.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 02:54:02.719760 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:54:02.708964 4405 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-a0bcbc/premium-w2-a0bcbc-kserve-78dfd6d78c-lwth2" podUID="33b620c1-389c-48b4-b277-90252018c4b6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 02:54:02.762925 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:54:02.762759 4405 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-a0bcbc/distinct-w2-a0bcbc-kserve-596f94c7cd-szl4g" podUID="bac32934-8a57-4919-b435-13f367cbea3e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": context deadline exceeded" Nov 04 02:54:03.416694 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:54:03.407779 4405 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-ebb7fd/distinct-w6-ebb7fd-kserve-55c47d8865-7z9pz" podUID="ccb8e181-5610-46da-bb8b-e0b07fa02ff1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 02:54:03.416694 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:54:03.408608 4405 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-ebb7fd/distinct-w6-ebb7fd-kserve-55c47d8865-7z9pz" Nov 04 02:54:03.478857 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:54:03.478801 4405 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-ljhhw" podUID="f66b1861-d87b-4d57-ab27-f1a24544a534" containerName="main" probeResult="failure" output="Get \"https://10.134.0.26:8000/ready\": context deadline exceeded" Nov 04 02:54:03.519822 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:54:03.519780 4405 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-cdcf67/distinct-w3-cdcf67-kserve-667cc7f7d7-5m2lq" podUID="52ed22cc-96e4-4ef0-b34d-ed58c89a9183" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 02:54:03.520685 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:54:03.520634 4405 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-cdcf67/distinct-w3-cdcf67-kserve-667cc7f7d7-5m2lq" Nov 04 02:54:03.542551 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:54:03.540839 4405 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-ebb7fd/distinct-2-w6-ebb7fd-kserve-6f68c5db86-zrpjs" podUID="b7b786e0-9c94-4126-9949-cd354ebbfb2a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 02:54:03.542551 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:54:03.541333 4405 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-ebb7fd/distinct-2-w6-ebb7fd-kserve-6f68c5db86-zrpjs" Nov 04 02:54:03.587727 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:54:03.559411 4405 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-e516ff/distinct-w1-e516ff-kserve-f4474cb9c-glpzw" podUID="8842ad09-22ff-4492-ba5d-d4cacf4eac75" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 02:54:03.587727 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:54:03.559883 4405 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-e516ff/distinct-w1-e516ff-kserve-f4474cb9c-glpzw" Nov 04 02:54:03.601707 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:54:03.600990 4405 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-203c35/simulator-w4-203c35-kserve-f9c4bd695-999ns" podUID="eb0d8bdd-6808-4fd0-b9b5-567dff3e7ca5" containerName="main" probeResult="failure" output="Get \"https://10.134.0.46:8000/ready\": context deadline exceeded" Nov 04 02:54:03.601707 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:54:03.601509 4405 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-203c35/simulator-w4-203c35-kserve-f9c4bd695-999ns" Nov 04 02:54:03.616972 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:54:03.616834 4405 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-fe4e0c/distinct-2-w0-fe4e0c-kserve-579bf4cc77-8pbq8" podUID="2b2ab722-d1bb-4376-bede-f5f4ee4562aa" containerName="main" probeResult="failure" output="Get \"https://10.134.0.68:8000/ready\": context deadline exceeded" Nov 04 02:54:03.625803 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:54:03.625764 4405 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-fe4e0c/distinct-2-w0-fe4e0c-kserve-579bf4cc77-8pbq8" Nov 04 02:54:04.049114 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:54:04.049056 4405 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-a0bcbc/premium-w2-a0bcbc-kserve-78dfd6d78c-lwth2" podUID="33b620c1-389c-48b4-b277-90252018c4b6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 02:54:04.115346 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:54:04.115305 4405 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-a0bcbc/distinct-w2-a0bcbc-kserve-596f94c7cd-szl4g" podUID="bac32934-8a57-4919-b435-13f367cbea3e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": context deadline exceeded" Nov 04 02:54:04.649695 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:54:04.648830 4405 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-ebb7fd/distinct-2-w6-ebb7fd-kserve-6f68c5db86-zrpjs" podUID="b7b786e0-9c94-4126-9949-cd354ebbfb2a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.65:8000/ready\": context deadline exceeded" Nov 04 02:54:04.649695 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:54:04.649390 4405 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-e516ff/distinct-w1-e516ff-kserve-f4474cb9c-glpzw" podUID="8842ad09-22ff-4492-ba5d-d4cacf4eac75" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/ready\": context deadline exceeded" Nov 04 02:54:04.768694 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:54:04.763211 4405 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-ebb7fd/distinct-w6-ebb7fd-kserve-55c47d8865-7z9pz" podUID="ccb8e181-5610-46da-bb8b-e0b07fa02ff1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 02:54:04.768694 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:54:04.763742 4405 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-203c35/simulator-w4-203c35-kserve-f9c4bd695-999ns" podUID="eb0d8bdd-6808-4fd0-b9b5-567dff3e7ca5" containerName="main" probeResult="failure" output="Get \"https://10.134.0.46:8000/ready\": context deadline exceeded" Nov 04 02:54:04.768694 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:54:04.765129 4405 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-fe4e0c/distinct-2-w0-fe4e0c-kserve-579bf4cc77-8pbq8" podUID="2b2ab722-d1bb-4376-bede-f5f4ee4562aa" containerName="main" probeResult="failure" output="Get \"https://10.134.0.68:8000/ready\": context deadline exceeded" Nov 04 02:54:04.893733 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:54:04.873212 4405 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-fe4e0c/distinct-2-w0-fe4e0c-kserve-579bf4cc77-8pbq8" podUID="2b2ab722-d1bb-4376-bede-f5f4ee4562aa" containerName="main" probeResult="failure" output="Get \"https://10.134.0.68:8000/health\": context deadline exceeded" Nov 04 02:54:04.893733 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:54:04.873411 4405 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-ljhhw" podUID="f66b1861-d87b-4d57-ab27-f1a24544a534" containerName="main" probeResult="failure" output="Get \"https://10.134.0.26:8000/ready\": context deadline exceeded" Nov 04 02:54:05.052021 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:54:05.051913 4405 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-a0bcbc/premium-w2-a0bcbc-kserve-78dfd6d78c-lwth2" podUID="33b620c1-389c-48b4-b277-90252018c4b6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 02:54:05.101699 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:54:05.093516 4405 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-cdcf67/distinct-w3-cdcf67-kserve-667cc7f7d7-5m2lq" podUID="52ed22cc-96e4-4ef0-b34d-ed58c89a9183" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/ready\": context deadline exceeded" Nov 04 02:54:05.228748 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:54:05.164831 4405 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-a0bcbc/distinct-w2-a0bcbc-kserve-596f94c7cd-szl4g" podUID="bac32934-8a57-4919-b435-13f367cbea3e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": context deadline exceeded" Nov 04 02:54:05.482696 ip-10-0-134-125 kubenswrapper[4405]: E1104 02:54:05.479260 4405 kubelet.go:2711] "Housekeeping took longer than expected" err="housekeeping took too long" expected="1s" actual="1.769s" Nov 04 02:54:05.482696 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:54:05.479357 4405 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-a0bcbc/premium-w2-a0bcbc-kserve-78dfd6d78c-lwth2" Nov 04 02:54:05.482696 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:54:05.479444 4405 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-a0bcbc/distinct-w2-a0bcbc-kserve-596f94c7cd-szl4g" Nov 04 02:54:05.482696 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:54:05.479489 4405 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-ljhhw" Nov 04 02:54:05.652697 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:54:05.649299 4405 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-ebb7fd/distinct-2-w6-ebb7fd-kserve-6f68c5db86-zrpjs" podUID="b7b786e0-9c94-4126-9949-cd354ebbfb2a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 02:54:05.693756 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:54:05.657293 4405 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-e516ff/distinct-w1-e516ff-kserve-f4474cb9c-glpzw" podUID="8842ad09-22ff-4492-ba5d-d4cacf4eac75" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/ready\": context deadline exceeded" Nov 04 02:54:05.782342 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:54:05.782238 4405 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-ebb7fd/distinct-w6-ebb7fd-kserve-55c47d8865-7z9pz" podUID="ccb8e181-5610-46da-bb8b-e0b07fa02ff1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 02:54:05.808956 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:54:05.797743 4405 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-fe4e0c/distinct-2-w0-fe4e0c-kserve-579bf4cc77-8pbq8" podUID="2b2ab722-d1bb-4376-bede-f5f4ee4562aa" containerName="main" probeResult="failure" output="Get \"https://10.134.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 02:54:05.816786 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:54:05.809660 4405 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-203c35/simulator-w4-203c35-kserve-f9c4bd695-999ns" podUID="eb0d8bdd-6808-4fd0-b9b5-567dff3e7ca5" containerName="main" probeResult="failure" output="Get \"https://10.134.0.46:8000/ready\": context deadline exceeded" Nov 04 02:54:05.883495 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:54:05.883452 4405 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-203c35/simulator-w4-203c35-kserve-f9c4bd695-999ns" podUID="eb0d8bdd-6808-4fd0-b9b5-567dff3e7ca5" containerName="main" probeResult="failure" output="Get \"https://10.134.0.46:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 02:54:05.909216 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:54:05.883424 4405 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-ljhhw" podUID="f66b1861-d87b-4d57-ab27-f1a24544a534" containerName="main" probeResult="failure" output="Get \"https://10.134.0.26:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 02:54:06.000694 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:54:06.000395 4405 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-203c35-openshift-default-56497cdcc-ljwqc"] Nov 04 02:54:06.076195 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:54:06.076105 4405 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-203c35-openshift-default-56497cdcc-ljwqc" Nov 04 02:54:06.124492 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:54:06.106902 4405 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-cdcf67/distinct-w3-cdcf67-kserve-667cc7f7d7-5m2lq" podUID="52ed22cc-96e4-4ef0-b34d-ed58c89a9183" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/ready\": context deadline exceeded" Nov 04 02:54:06.205695 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:54:06.173566 4405 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fec507be-e742-4226-a6c4-6fb41e6f3479-istiod-ca-cert\") pod \"e2e-worker-w4-203c35-openshift-default-56497cdcc-ljwqc\" (UID: \"fec507be-e742-4226-a6c4-6fb41e6f3479\") " pod="openshift-ingress/e2e-worker-w4-203c35-openshift-default-56497cdcc-ljwqc" Nov 04 02:54:06.206057 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:54:06.206011 4405 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fec507be-e742-4226-a6c4-6fb41e6f3479-credential-socket\") pod \"e2e-worker-w4-203c35-openshift-default-56497cdcc-ljwqc\" (UID: \"fec507be-e742-4226-a6c4-6fb41e6f3479\") " pod="openshift-ingress/e2e-worker-w4-203c35-openshift-default-56497cdcc-ljwqc" Nov 04 02:54:06.206486 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:54:06.206463 4405 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fec507be-e742-4226-a6c4-6fb41e6f3479-workload-socket\") pod \"e2e-worker-w4-203c35-openshift-default-56497cdcc-ljwqc\" (UID: \"fec507be-e742-4226-a6c4-6fb41e6f3479\") " pod="openshift-ingress/e2e-worker-w4-203c35-openshift-default-56497cdcc-ljwqc" Nov 04 02:54:06.303887 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:54:06.273546 4405 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ntjgq\" (UniqueName: \"kubernetes.io/projected/fec507be-e742-4226-a6c4-6fb41e6f3479-kube-api-access-ntjgq\") pod \"e2e-worker-w4-203c35-openshift-default-56497cdcc-ljwqc\" (UID: \"fec507be-e742-4226-a6c4-6fb41e6f3479\") " pod="openshift-ingress/e2e-worker-w4-203c35-openshift-default-56497cdcc-ljwqc" Nov 04 02:54:06.303887 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:54:06.273600 4405 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fec507be-e742-4226-a6c4-6fb41e6f3479-istio-envoy\") pod \"e2e-worker-w4-203c35-openshift-default-56497cdcc-ljwqc\" (UID: \"fec507be-e742-4226-a6c4-6fb41e6f3479\") " pod="openshift-ingress/e2e-worker-w4-203c35-openshift-default-56497cdcc-ljwqc" Nov 04 02:54:06.303887 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:54:06.273659 4405 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fec507be-e742-4226-a6c4-6fb41e6f3479-istio-podinfo\") pod \"e2e-worker-w4-203c35-openshift-default-56497cdcc-ljwqc\" (UID: \"fec507be-e742-4226-a6c4-6fb41e6f3479\") " pod="openshift-ingress/e2e-worker-w4-203c35-openshift-default-56497cdcc-ljwqc" Nov 04 02:54:06.303887 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:54:06.238486 4405 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-a0bcbc/distinct-w2-a0bcbc-kserve-596f94c7cd-szl4g" podUID="bac32934-8a57-4919-b435-13f367cbea3e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 02:54:06.303887 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:54:06.200179 4405 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-203c35-openshift-default-56497cdcc-ljwqc"] Nov 04 02:54:06.303887 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:54:06.273703 4405 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fec507be-e742-4226-a6c4-6fb41e6f3479-istio-token\") pod \"e2e-worker-w4-203c35-openshift-default-56497cdcc-ljwqc\" (UID: \"fec507be-e742-4226-a6c4-6fb41e6f3479\") " pod="openshift-ingress/e2e-worker-w4-203c35-openshift-default-56497cdcc-ljwqc" Nov 04 02:54:06.303887 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:54:06.273964 4405 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fec507be-e742-4226-a6c4-6fb41e6f3479-istio-data\") pod \"e2e-worker-w4-203c35-openshift-default-56497cdcc-ljwqc\" (UID: \"fec507be-e742-4226-a6c4-6fb41e6f3479\") " pod="openshift-ingress/e2e-worker-w4-203c35-openshift-default-56497cdcc-ljwqc" Nov 04 02:54:06.303887 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:54:06.273999 4405 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fec507be-e742-4226-a6c4-6fb41e6f3479-workload-certs\") pod \"e2e-worker-w4-203c35-openshift-default-56497cdcc-ljwqc\" (UID: \"fec507be-e742-4226-a6c4-6fb41e6f3479\") " pod="openshift-ingress/e2e-worker-w4-203c35-openshift-default-56497cdcc-ljwqc" Nov 04 02:54:06.348061 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:54:06.338584 4405 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-e516ff/distinct-w1-e516ff-kserve-f4474cb9c-glpzw" podUID="8842ad09-22ff-4492-ba5d-d4cacf4eac75" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 02:54:06.395754 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:54:06.380779 4405 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fec507be-e742-4226-a6c4-6fb41e6f3479-istio-data\") pod \"e2e-worker-w4-203c35-openshift-default-56497cdcc-ljwqc\" (UID: \"fec507be-e742-4226-a6c4-6fb41e6f3479\") " pod="openshift-ingress/e2e-worker-w4-203c35-openshift-default-56497cdcc-ljwqc" Nov 04 02:54:06.395754 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:54:06.380825 4405 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fec507be-e742-4226-a6c4-6fb41e6f3479-workload-certs\") pod \"e2e-worker-w4-203c35-openshift-default-56497cdcc-ljwqc\" (UID: \"fec507be-e742-4226-a6c4-6fb41e6f3479\") " pod="openshift-ingress/e2e-worker-w4-203c35-openshift-default-56497cdcc-ljwqc" Nov 04 02:54:06.395754 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:54:06.380955 4405 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fec507be-e742-4226-a6c4-6fb41e6f3479-istiod-ca-cert\") pod \"e2e-worker-w4-203c35-openshift-default-56497cdcc-ljwqc\" (UID: \"fec507be-e742-4226-a6c4-6fb41e6f3479\") " pod="openshift-ingress/e2e-worker-w4-203c35-openshift-default-56497cdcc-ljwqc" Nov 04 02:54:06.395754 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:54:06.380983 4405 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fec507be-e742-4226-a6c4-6fb41e6f3479-credential-socket\") pod \"e2e-worker-w4-203c35-openshift-default-56497cdcc-ljwqc\" (UID: \"fec507be-e742-4226-a6c4-6fb41e6f3479\") " pod="openshift-ingress/e2e-worker-w4-203c35-openshift-default-56497cdcc-ljwqc" Nov 04 02:54:06.395754 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:54:06.381018 4405 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fec507be-e742-4226-a6c4-6fb41e6f3479-workload-socket\") pod \"e2e-worker-w4-203c35-openshift-default-56497cdcc-ljwqc\" (UID: \"fec507be-e742-4226-a6c4-6fb41e6f3479\") " pod="openshift-ingress/e2e-worker-w4-203c35-openshift-default-56497cdcc-ljwqc" Nov 04 02:54:06.395754 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:54:06.381103 4405 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ntjgq\" (UniqueName: \"kubernetes.io/projected/fec507be-e742-4226-a6c4-6fb41e6f3479-kube-api-access-ntjgq\") pod \"e2e-worker-w4-203c35-openshift-default-56497cdcc-ljwqc\" (UID: \"fec507be-e742-4226-a6c4-6fb41e6f3479\") " pod="openshift-ingress/e2e-worker-w4-203c35-openshift-default-56497cdcc-ljwqc" Nov 04 02:54:06.395754 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:54:06.381132 4405 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fec507be-e742-4226-a6c4-6fb41e6f3479-istio-envoy\") pod \"e2e-worker-w4-203c35-openshift-default-56497cdcc-ljwqc\" (UID: \"fec507be-e742-4226-a6c4-6fb41e6f3479\") " pod="openshift-ingress/e2e-worker-w4-203c35-openshift-default-56497cdcc-ljwqc" Nov 04 02:54:06.395754 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:54:06.381180 4405 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fec507be-e742-4226-a6c4-6fb41e6f3479-istio-podinfo\") pod \"e2e-worker-w4-203c35-openshift-default-56497cdcc-ljwqc\" (UID: \"fec507be-e742-4226-a6c4-6fb41e6f3479\") " pod="openshift-ingress/e2e-worker-w4-203c35-openshift-default-56497cdcc-ljwqc" Nov 04 02:54:06.395754 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:54:06.381204 4405 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fec507be-e742-4226-a6c4-6fb41e6f3479-istio-token\") pod \"e2e-worker-w4-203c35-openshift-default-56497cdcc-ljwqc\" (UID: \"fec507be-e742-4226-a6c4-6fb41e6f3479\") " pod="openshift-ingress/e2e-worker-w4-203c35-openshift-default-56497cdcc-ljwqc" Nov 04 02:54:06.395754 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:54:06.381906 4405 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fec507be-e742-4226-a6c4-6fb41e6f3479-istio-data\") pod \"e2e-worker-w4-203c35-openshift-default-56497cdcc-ljwqc\" (UID: \"fec507be-e742-4226-a6c4-6fb41e6f3479\") " pod="openshift-ingress/e2e-worker-w4-203c35-openshift-default-56497cdcc-ljwqc" Nov 04 02:54:06.395754 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:54:06.382138 4405 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fec507be-e742-4226-a6c4-6fb41e6f3479-workload-certs\") pod \"e2e-worker-w4-203c35-openshift-default-56497cdcc-ljwqc\" (UID: \"fec507be-e742-4226-a6c4-6fb41e6f3479\") " pod="openshift-ingress/e2e-worker-w4-203c35-openshift-default-56497cdcc-ljwqc" Nov 04 02:54:06.396512 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:54:06.395808 4405 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fec507be-e742-4226-a6c4-6fb41e6f3479-credential-socket\") pod \"e2e-worker-w4-203c35-openshift-default-56497cdcc-ljwqc\" (UID: \"fec507be-e742-4226-a6c4-6fb41e6f3479\") " pod="openshift-ingress/e2e-worker-w4-203c35-openshift-default-56497cdcc-ljwqc" Nov 04 02:54:06.396512 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:54:06.396087 4405 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fec507be-e742-4226-a6c4-6fb41e6f3479-workload-socket\") pod \"e2e-worker-w4-203c35-openshift-default-56497cdcc-ljwqc\" (UID: \"fec507be-e742-4226-a6c4-6fb41e6f3479\") " pod="openshift-ingress/e2e-worker-w4-203c35-openshift-default-56497cdcc-ljwqc" Nov 04 02:54:06.401861 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:54:06.401835 4405 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fec507be-e742-4226-a6c4-6fb41e6f3479-istiod-ca-cert\") pod \"e2e-worker-w4-203c35-openshift-default-56497cdcc-ljwqc\" (UID: \"fec507be-e742-4226-a6c4-6fb41e6f3479\") " pod="openshift-ingress/e2e-worker-w4-203c35-openshift-default-56497cdcc-ljwqc" Nov 04 02:54:06.402400 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:54:06.400773 4405 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-a0bcbc/premium-w2-a0bcbc-kserve-78dfd6d78c-lwth2" podUID="33b620c1-389c-48b4-b277-90252018c4b6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/ready\": context deadline exceeded" Nov 04 02:54:06.520836 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:54:06.520767 4405 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fec507be-e742-4226-a6c4-6fb41e6f3479-istio-envoy\") pod \"e2e-worker-w4-203c35-openshift-default-56497cdcc-ljwqc\" (UID: \"fec507be-e742-4226-a6c4-6fb41e6f3479\") " pod="openshift-ingress/e2e-worker-w4-203c35-openshift-default-56497cdcc-ljwqc" Nov 04 02:54:06.618641 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:54:06.614424 4405 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fec507be-e742-4226-a6c4-6fb41e6f3479-istio-podinfo\") pod \"e2e-worker-w4-203c35-openshift-default-56497cdcc-ljwqc\" (UID: \"fec507be-e742-4226-a6c4-6fb41e6f3479\") " pod="openshift-ingress/e2e-worker-w4-203c35-openshift-default-56497cdcc-ljwqc" Nov 04 02:54:06.671247 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:54:06.671186 4405 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-e516ff/distinct-w1-e516ff-kserve-f4474cb9c-glpzw" podUID="8842ad09-22ff-4492-ba5d-d4cacf4eac75" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 02:54:06.673296 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:54:06.673252 4405 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-ebb7fd/distinct-2-w6-ebb7fd-kserve-6f68c5db86-zrpjs" podUID="b7b786e0-9c94-4126-9949-cd354ebbfb2a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 02:54:06.764325 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:54:06.754406 4405 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-ebb7fd/distinct-w6-ebb7fd-kserve-55c47d8865-7z9pz" Nov 04 02:54:06.767478 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:54:06.767441 4405 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ntjgq\" (UniqueName: \"kubernetes.io/projected/fec507be-e742-4226-a6c4-6fb41e6f3479-kube-api-access-ntjgq\") pod \"e2e-worker-w4-203c35-openshift-default-56497cdcc-ljwqc\" (UID: \"fec507be-e742-4226-a6c4-6fb41e6f3479\") " pod="openshift-ingress/e2e-worker-w4-203c35-openshift-default-56497cdcc-ljwqc" Nov 04 02:54:06.768373 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:54:06.768244 4405 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-fe4e0c/distinct-2-w0-fe4e0c-kserve-579bf4cc77-8pbq8" Nov 04 02:54:06.787726 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:54:06.785570 4405 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-ebb7fd/distinct-2-w6-ebb7fd-kserve-6f68c5db86-zrpjs" Nov 04 02:54:06.787726 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:54:06.785732 4405 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-cdcf67/distinct-w3-cdcf67-kserve-667cc7f7d7-5m2lq" Nov 04 02:54:06.787726 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:54:06.785792 4405 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-a0bcbc/premium-w2-a0bcbc-kserve-78dfd6d78c-lwth2" Nov 04 02:54:06.787726 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:54:06.785848 4405 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-203c35/simulator-w4-203c35-kserve-f9c4bd695-999ns" Nov 04 02:54:06.787726 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:54:06.785900 4405 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-e516ff/distinct-w1-e516ff-kserve-f4474cb9c-glpzw" Nov 04 02:54:06.791789 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:54:06.791764 4405 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-a0bcbc/distinct-w2-a0bcbc-kserve-596f94c7cd-szl4g" Nov 04 02:54:06.807690 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:54:06.803787 4405 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-ljhhw" Nov 04 02:54:06.840813 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:54:06.840782 4405 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fec507be-e742-4226-a6c4-6fb41e6f3479-istio-token\") pod \"e2e-worker-w4-203c35-openshift-default-56497cdcc-ljwqc\" (UID: \"fec507be-e742-4226-a6c4-6fb41e6f3479\") " pod="openshift-ingress/e2e-worker-w4-203c35-openshift-default-56497cdcc-ljwqc" Nov 04 02:54:06.892214 ip-10-0-134-125 kubenswrapper[4405]: E1104 02:54:06.892132 4405 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5f2vb" containerName="extractor" Nov 04 02:54:07.007029 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:54:07.007001 4405 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-203c35-openshift-default-56497cdcc-ljwqc" Nov 04 02:54:07.452884 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:54:07.452847 4405 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-203c35-openshift-default-56497cdcc-ljwqc"] Nov 04 02:54:07.456623 ip-10-0-134-125 kubenswrapper[4405]: W1104 02:54:07.456593 4405 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfec507be_e742_4226_a6c4_6fb41e6f3479.slice/crio-59cf3d394fb94451fe8c0a974b0dc71ad93e35cabc9c20b3428b27ee9c1a8612 WatchSource:0}: Error finding container 59cf3d394fb94451fe8c0a974b0dc71ad93e35cabc9c20b3428b27ee9c1a8612: Status 404 returned error can't find the container with id 59cf3d394fb94451fe8c0a974b0dc71ad93e35cabc9c20b3428b27ee9c1a8612 Nov 04 02:54:07.459406 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:54:07.459367 4405 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 04 02:54:07.459523 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:54:07.459439 4405 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 04 02:54:07.459523 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:54:07.459470 4405 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 04 02:54:08.169247 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:54:08.169213 4405 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-203c35-openshift-default-56497cdcc-ljwqc" event={"ID":"fec507be-e742-4226-a6c4-6fb41e6f3479","Type":"ContainerStarted","Data":"dc932a567f19c37532b049ce74104a0a9cefc967ab7c48ef6616658d921d5c76"} Nov 04 02:54:08.169659 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:54:08.169264 4405 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-203c35-openshift-default-56497cdcc-ljwqc" event={"ID":"fec507be-e742-4226-a6c4-6fb41e6f3479","Type":"ContainerStarted","Data":"59cf3d394fb94451fe8c0a974b0dc71ad93e35cabc9c20b3428b27ee9c1a8612"} Nov 04 02:54:08.331163 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:54:08.331107 4405 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-203c35-openshift-default-56497cdcc-ljwqc" podStartSLOduration=4.331091882 podStartE2EDuration="4.331091882s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 02:54:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 02:54:08.329008789 +0000 UTC m=+1294.759708908" watchObservedRunningTime="2026-11-04 02:54:08.331091882 +0000 UTC m=+1294.761792003" Nov 04 02:54:09.013694 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:54:09.012016 4405 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-203c35-openshift-default-56497cdcc-ljwqc" Nov 04 02:54:09.018020 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:54:09.017985 4405 patch_prober.go:29] interesting pod/e2e-worker-w4-203c35-openshift-default-56497cdcc-ljwqc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" start-of-body= Nov 04 02:54:09.019166 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:54:09.019114 4405 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-203c35-openshift-default-56497cdcc-ljwqc" podUID="fec507be-e742-4226-a6c4-6fb41e6f3479" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" Nov 04 02:54:10.029317 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:54:10.028861 4405 patch_prober.go:29] interesting pod/e2e-worker-w4-203c35-openshift-default-56497cdcc-ljwqc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" start-of-body= Nov 04 02:54:10.029317 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:54:10.028927 4405 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-203c35-openshift-default-56497cdcc-ljwqc" podUID="fec507be-e742-4226-a6c4-6fb41e6f3479" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" Nov 04 02:54:11.013094 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:54:11.013056 4405 patch_prober.go:29] interesting pod/e2e-worker-w4-203c35-openshift-default-56497cdcc-ljwqc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" start-of-body= Nov 04 02:54:11.013403 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:54:11.013375 4405 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-203c35-openshift-default-56497cdcc-ljwqc" podUID="fec507be-e742-4226-a6c4-6fb41e6f3479" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" Nov 04 02:54:12.016693 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:54:12.010036 4405 patch_prober.go:29] interesting pod/e2e-worker-w4-203c35-openshift-default-56497cdcc-ljwqc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" start-of-body= Nov 04 02:54:12.016693 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:54:12.010111 4405 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-203c35-openshift-default-56497cdcc-ljwqc" podUID="fec507be-e742-4226-a6c4-6fb41e6f3479" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" Nov 04 02:54:13.010133 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:54:13.010090 4405 patch_prober.go:29] interesting pod/e2e-worker-w4-203c35-openshift-default-56497cdcc-ljwqc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" start-of-body= Nov 04 02:54:13.010501 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:54:13.010466 4405 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-203c35-openshift-default-56497cdcc-ljwqc" podUID="fec507be-e742-4226-a6c4-6fb41e6f3479" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" Nov 04 02:54:14.008192 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:54:14.008155 4405 patch_prober.go:29] interesting pod/e2e-worker-w4-203c35-openshift-default-56497cdcc-ljwqc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" start-of-body= Nov 04 02:54:14.008600 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:54:14.008232 4405 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-203c35-openshift-default-56497cdcc-ljwqc" podUID="fec507be-e742-4226-a6c4-6fb41e6f3479" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" Nov 04 02:54:15.255494 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:54:15.255395 4405 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-203c35-openshift-default-56497cdcc-ljwqc" Nov 04 02:54:15.256087 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:54:15.256051 4405 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-203c35-openshift-default-56497cdcc-ljwqc" Nov 04 02:54:15.258313 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:54:15.258291 4405 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-203c35-openshift-default-56497cdcc-ljwqc" Nov 04 02:54:34.410602 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:54:34.410568 4405 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-fe4e0c-openshift-default-c7f4fc84c-vh6fd"] Nov 04 02:54:34.414945 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:54:34.414920 4405 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-fe4e0c-openshift-default-c7f4fc84c-vh6fd" Nov 04 02:54:34.470780 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:54:34.470744 4405 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/70d9a666-2667-41e8-8425-4e2ba085599d-istio-podinfo\") pod \"e2e-worker-w0-fe4e0c-openshift-default-c7f4fc84c-vh6fd\" (UID: \"70d9a666-2667-41e8-8425-4e2ba085599d\") " pod="openshift-ingress/e2e-worker-w0-fe4e0c-openshift-default-c7f4fc84c-vh6fd" Nov 04 02:54:34.470973 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:54:34.470821 4405 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/70d9a666-2667-41e8-8425-4e2ba085599d-istio-data\") pod \"e2e-worker-w0-fe4e0c-openshift-default-c7f4fc84c-vh6fd\" (UID: \"70d9a666-2667-41e8-8425-4e2ba085599d\") " pod="openshift-ingress/e2e-worker-w0-fe4e0c-openshift-default-c7f4fc84c-vh6fd" Nov 04 02:54:34.470973 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:54:34.470924 4405 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/70d9a666-2667-41e8-8425-4e2ba085599d-credential-socket\") pod \"e2e-worker-w0-fe4e0c-openshift-default-c7f4fc84c-vh6fd\" (UID: \"70d9a666-2667-41e8-8425-4e2ba085599d\") " pod="openshift-ingress/e2e-worker-w0-fe4e0c-openshift-default-c7f4fc84c-vh6fd" Nov 04 02:54:34.470973 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:54:34.470948 4405 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/70d9a666-2667-41e8-8425-4e2ba085599d-workload-socket\") pod \"e2e-worker-w0-fe4e0c-openshift-default-c7f4fc84c-vh6fd\" (UID: \"70d9a666-2667-41e8-8425-4e2ba085599d\") " pod="openshift-ingress/e2e-worker-w0-fe4e0c-openshift-default-c7f4fc84c-vh6fd" Nov 04 02:54:34.470973 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:54:34.470971 4405 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/70d9a666-2667-41e8-8425-4e2ba085599d-istio-token\") pod \"e2e-worker-w0-fe4e0c-openshift-default-c7f4fc84c-vh6fd\" (UID: \"70d9a666-2667-41e8-8425-4e2ba085599d\") " pod="openshift-ingress/e2e-worker-w0-fe4e0c-openshift-default-c7f4fc84c-vh6fd" Nov 04 02:54:34.471109 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:54:34.470989 4405 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/70d9a666-2667-41e8-8425-4e2ba085599d-istiod-ca-cert\") pod \"e2e-worker-w0-fe4e0c-openshift-default-c7f4fc84c-vh6fd\" (UID: \"70d9a666-2667-41e8-8425-4e2ba085599d\") " pod="openshift-ingress/e2e-worker-w0-fe4e0c-openshift-default-c7f4fc84c-vh6fd" Nov 04 02:54:34.471154 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:54:34.471133 4405 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k7l8h\" (UniqueName: \"kubernetes.io/projected/70d9a666-2667-41e8-8425-4e2ba085599d-kube-api-access-k7l8h\") pod \"e2e-worker-w0-fe4e0c-openshift-default-c7f4fc84c-vh6fd\" (UID: \"70d9a666-2667-41e8-8425-4e2ba085599d\") " pod="openshift-ingress/e2e-worker-w0-fe4e0c-openshift-default-c7f4fc84c-vh6fd" Nov 04 02:54:34.471190 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:54:34.471165 4405 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/70d9a666-2667-41e8-8425-4e2ba085599d-istio-envoy\") pod \"e2e-worker-w0-fe4e0c-openshift-default-c7f4fc84c-vh6fd\" (UID: \"70d9a666-2667-41e8-8425-4e2ba085599d\") " pod="openshift-ingress/e2e-worker-w0-fe4e0c-openshift-default-c7f4fc84c-vh6fd" Nov 04 02:54:34.471251 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:54:34.471236 4405 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/70d9a666-2667-41e8-8425-4e2ba085599d-workload-certs\") pod \"e2e-worker-w0-fe4e0c-openshift-default-c7f4fc84c-vh6fd\" (UID: \"70d9a666-2667-41e8-8425-4e2ba085599d\") " pod="openshift-ingress/e2e-worker-w0-fe4e0c-openshift-default-c7f4fc84c-vh6fd" Nov 04 02:54:34.524948 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:54:34.524908 4405 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-fe4e0c-openshift-default-c7f4fc84c-vh6fd"] Nov 04 02:54:34.572586 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:54:34.572534 4405 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k7l8h\" (UniqueName: \"kubernetes.io/projected/70d9a666-2667-41e8-8425-4e2ba085599d-kube-api-access-k7l8h\") pod \"e2e-worker-w0-fe4e0c-openshift-default-c7f4fc84c-vh6fd\" (UID: \"70d9a666-2667-41e8-8425-4e2ba085599d\") " pod="openshift-ingress/e2e-worker-w0-fe4e0c-openshift-default-c7f4fc84c-vh6fd" Nov 04 02:54:34.572838 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:54:34.572601 4405 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/70d9a666-2667-41e8-8425-4e2ba085599d-istio-envoy\") pod \"e2e-worker-w0-fe4e0c-openshift-default-c7f4fc84c-vh6fd\" (UID: \"70d9a666-2667-41e8-8425-4e2ba085599d\") " pod="openshift-ingress/e2e-worker-w0-fe4e0c-openshift-default-c7f4fc84c-vh6fd" Nov 04 02:54:34.572838 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:54:34.572711 4405 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/70d9a666-2667-41e8-8425-4e2ba085599d-workload-certs\") pod \"e2e-worker-w0-fe4e0c-openshift-default-c7f4fc84c-vh6fd\" (UID: \"70d9a666-2667-41e8-8425-4e2ba085599d\") " pod="openshift-ingress/e2e-worker-w0-fe4e0c-openshift-default-c7f4fc84c-vh6fd" Nov 04 02:54:34.572838 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:54:34.572787 4405 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/70d9a666-2667-41e8-8425-4e2ba085599d-istio-podinfo\") pod \"e2e-worker-w0-fe4e0c-openshift-default-c7f4fc84c-vh6fd\" (UID: \"70d9a666-2667-41e8-8425-4e2ba085599d\") " pod="openshift-ingress/e2e-worker-w0-fe4e0c-openshift-default-c7f4fc84c-vh6fd" Nov 04 02:54:34.573017 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:54:34.572851 4405 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/70d9a666-2667-41e8-8425-4e2ba085599d-istio-data\") pod \"e2e-worker-w0-fe4e0c-openshift-default-c7f4fc84c-vh6fd\" (UID: \"70d9a666-2667-41e8-8425-4e2ba085599d\") " pod="openshift-ingress/e2e-worker-w0-fe4e0c-openshift-default-c7f4fc84c-vh6fd" Nov 04 02:54:34.573017 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:54:34.572917 4405 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/70d9a666-2667-41e8-8425-4e2ba085599d-credential-socket\") pod \"e2e-worker-w0-fe4e0c-openshift-default-c7f4fc84c-vh6fd\" (UID: \"70d9a666-2667-41e8-8425-4e2ba085599d\") " pod="openshift-ingress/e2e-worker-w0-fe4e0c-openshift-default-c7f4fc84c-vh6fd" Nov 04 02:54:34.573017 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:54:34.572954 4405 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/70d9a666-2667-41e8-8425-4e2ba085599d-workload-socket\") pod \"e2e-worker-w0-fe4e0c-openshift-default-c7f4fc84c-vh6fd\" (UID: \"70d9a666-2667-41e8-8425-4e2ba085599d\") " pod="openshift-ingress/e2e-worker-w0-fe4e0c-openshift-default-c7f4fc84c-vh6fd" Nov 04 02:54:34.573017 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:54:34.572993 4405 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/70d9a666-2667-41e8-8425-4e2ba085599d-istio-token\") pod \"e2e-worker-w0-fe4e0c-openshift-default-c7f4fc84c-vh6fd\" (UID: \"70d9a666-2667-41e8-8425-4e2ba085599d\") " pod="openshift-ingress/e2e-worker-w0-fe4e0c-openshift-default-c7f4fc84c-vh6fd" Nov 04 02:54:34.573228 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:54:34.573020 4405 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/70d9a666-2667-41e8-8425-4e2ba085599d-istiod-ca-cert\") pod \"e2e-worker-w0-fe4e0c-openshift-default-c7f4fc84c-vh6fd\" (UID: \"70d9a666-2667-41e8-8425-4e2ba085599d\") " pod="openshift-ingress/e2e-worker-w0-fe4e0c-openshift-default-c7f4fc84c-vh6fd" Nov 04 02:54:34.573228 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:54:34.573196 4405 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/70d9a666-2667-41e8-8425-4e2ba085599d-workload-certs\") pod \"e2e-worker-w0-fe4e0c-openshift-default-c7f4fc84c-vh6fd\" (UID: \"70d9a666-2667-41e8-8425-4e2ba085599d\") " pod="openshift-ingress/e2e-worker-w0-fe4e0c-openshift-default-c7f4fc84c-vh6fd" Nov 04 02:54:34.573336 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:54:34.573270 4405 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/70d9a666-2667-41e8-8425-4e2ba085599d-istio-data\") pod \"e2e-worker-w0-fe4e0c-openshift-default-c7f4fc84c-vh6fd\" (UID: \"70d9a666-2667-41e8-8425-4e2ba085599d\") " pod="openshift-ingress/e2e-worker-w0-fe4e0c-openshift-default-c7f4fc84c-vh6fd" Nov 04 02:54:34.573437 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:54:34.573410 4405 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/70d9a666-2667-41e8-8425-4e2ba085599d-workload-socket\") pod \"e2e-worker-w0-fe4e0c-openshift-default-c7f4fc84c-vh6fd\" (UID: \"70d9a666-2667-41e8-8425-4e2ba085599d\") " pod="openshift-ingress/e2e-worker-w0-fe4e0c-openshift-default-c7f4fc84c-vh6fd" Nov 04 02:54:34.573572 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:54:34.573469 4405 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/70d9a666-2667-41e8-8425-4e2ba085599d-credential-socket\") pod \"e2e-worker-w0-fe4e0c-openshift-default-c7f4fc84c-vh6fd\" (UID: \"70d9a666-2667-41e8-8425-4e2ba085599d\") " pod="openshift-ingress/e2e-worker-w0-fe4e0c-openshift-default-c7f4fc84c-vh6fd" Nov 04 02:54:34.573821 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:54:34.573804 4405 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/70d9a666-2667-41e8-8425-4e2ba085599d-istiod-ca-cert\") pod \"e2e-worker-w0-fe4e0c-openshift-default-c7f4fc84c-vh6fd\" (UID: \"70d9a666-2667-41e8-8425-4e2ba085599d\") " pod="openshift-ingress/e2e-worker-w0-fe4e0c-openshift-default-c7f4fc84c-vh6fd" Nov 04 02:54:34.575568 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:54:34.575547 4405 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/70d9a666-2667-41e8-8425-4e2ba085599d-istio-podinfo\") pod \"e2e-worker-w0-fe4e0c-openshift-default-c7f4fc84c-vh6fd\" (UID: \"70d9a666-2667-41e8-8425-4e2ba085599d\") " pod="openshift-ingress/e2e-worker-w0-fe4e0c-openshift-default-c7f4fc84c-vh6fd" Nov 04 02:54:34.575655 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:54:34.575615 4405 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/70d9a666-2667-41e8-8425-4e2ba085599d-istio-envoy\") pod \"e2e-worker-w0-fe4e0c-openshift-default-c7f4fc84c-vh6fd\" (UID: \"70d9a666-2667-41e8-8425-4e2ba085599d\") " pod="openshift-ingress/e2e-worker-w0-fe4e0c-openshift-default-c7f4fc84c-vh6fd" Nov 04 02:54:34.633722 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:54:34.633662 4405 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/70d9a666-2667-41e8-8425-4e2ba085599d-istio-token\") pod \"e2e-worker-w0-fe4e0c-openshift-default-c7f4fc84c-vh6fd\" (UID: \"70d9a666-2667-41e8-8425-4e2ba085599d\") " pod="openshift-ingress/e2e-worker-w0-fe4e0c-openshift-default-c7f4fc84c-vh6fd" Nov 04 02:54:34.633871 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:54:34.633660 4405 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k7l8h\" (UniqueName: \"kubernetes.io/projected/70d9a666-2667-41e8-8425-4e2ba085599d-kube-api-access-k7l8h\") pod \"e2e-worker-w0-fe4e0c-openshift-default-c7f4fc84c-vh6fd\" (UID: \"70d9a666-2667-41e8-8425-4e2ba085599d\") " pod="openshift-ingress/e2e-worker-w0-fe4e0c-openshift-default-c7f4fc84c-vh6fd" Nov 04 02:54:34.728799 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:54:34.728712 4405 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-fe4e0c-openshift-default-c7f4fc84c-vh6fd" Nov 04 02:54:35.200339 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:54:35.187300 4405 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-fe4e0c-openshift-default-c7f4fc84c-vh6fd"] Nov 04 02:54:35.212952 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:54:35.211710 4405 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 04 02:54:35.212952 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:54:35.211814 4405 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 04 02:54:35.212952 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:54:35.211874 4405 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 04 02:54:35.392855 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:54:35.392795 4405 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-fe4e0c-openshift-default-c7f4fc84c-vh6fd" event={"ID":"70d9a666-2667-41e8-8425-4e2ba085599d","Type":"ContainerStarted","Data":"e086266869b21630287ce36176cac9278dbb5759714616463595d366b59d1915"} Nov 04 02:54:35.394554 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:54:35.394522 4405 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-fe4e0c-openshift-default-c7f4fc84c-vh6fd" event={"ID":"70d9a666-2667-41e8-8425-4e2ba085599d","Type":"ContainerStarted","Data":"e9d78936b5488080ab49f3fccb754690c90bf7ed5f4cc52eb1194e58ab902ee5"} Nov 04 02:54:35.437863 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:54:35.436282 4405 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w5-3b7476-openshift-default-59975457bf-gph88"] Nov 04 02:54:35.459240 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:54:35.458214 4405 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-3b7476-openshift-default-59975457bf-gph88" Nov 04 02:54:35.482596 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:54:35.479718 4405 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w5-3b7476-openshift-default-59975457bf-vxwkm"] Nov 04 02:54:35.485007 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:54:35.484822 4405 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-3b7476-openshift-default-59975457bf-vxwkm" Nov 04 02:54:35.543202 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:54:35.543115 4405 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-3b7476-openshift-default-59975457bf-gph88"] Nov 04 02:54:35.578991 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:54:35.578959 4405 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-3b7476-openshift-default-59975457bf-vxwkm"] Nov 04 02:54:35.599216 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:54:35.599180 4405 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f2c5af93-085e-462b-8377-a6e8139e66c8-workload-certs\") pod \"e2e-worker-w5-3b7476-openshift-default-59975457bf-vxwkm\" (UID: \"f2c5af93-085e-462b-8377-a6e8139e66c8\") " pod="openshift-ingress/e2e-worker-w5-3b7476-openshift-default-59975457bf-vxwkm" Nov 04 02:54:35.629794 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:54:35.629757 4405 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f2c5af93-085e-462b-8377-a6e8139e66c8-credential-socket\") pod \"e2e-worker-w5-3b7476-openshift-default-59975457bf-vxwkm\" (UID: \"f2c5af93-085e-462b-8377-a6e8139e66c8\") " pod="openshift-ingress/e2e-worker-w5-3b7476-openshift-default-59975457bf-vxwkm" Nov 04 02:54:35.633708 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:54:35.630045 4405 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9ddrz\" (UniqueName: \"kubernetes.io/projected/43a58ae9-e100-4e85-bbe0-fdc36af6f192-kube-api-access-9ddrz\") pod \"e2e-worker-w5-3b7476-openshift-default-59975457bf-gph88\" (UID: \"43a58ae9-e100-4e85-bbe0-fdc36af6f192\") " pod="openshift-ingress/e2e-worker-w5-3b7476-openshift-default-59975457bf-gph88" Nov 04 02:54:35.634060 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:54:35.634014 4405 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f2c5af93-085e-462b-8377-a6e8139e66c8-istio-podinfo\") pod \"e2e-worker-w5-3b7476-openshift-default-59975457bf-vxwkm\" (UID: \"f2c5af93-085e-462b-8377-a6e8139e66c8\") " pod="openshift-ingress/e2e-worker-w5-3b7476-openshift-default-59975457bf-vxwkm" Nov 04 02:54:35.634266 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:54:35.634226 4405 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/43a58ae9-e100-4e85-bbe0-fdc36af6f192-credential-socket\") pod \"e2e-worker-w5-3b7476-openshift-default-59975457bf-gph88\" (UID: \"43a58ae9-e100-4e85-bbe0-fdc36af6f192\") " pod="openshift-ingress/e2e-worker-w5-3b7476-openshift-default-59975457bf-gph88" Nov 04 02:54:35.652553 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:54:35.652498 4405 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/43a58ae9-e100-4e85-bbe0-fdc36af6f192-istio-data\") pod \"e2e-worker-w5-3b7476-openshift-default-59975457bf-gph88\" (UID: \"43a58ae9-e100-4e85-bbe0-fdc36af6f192\") " pod="openshift-ingress/e2e-worker-w5-3b7476-openshift-default-59975457bf-gph88" Nov 04 02:54:35.652948 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:54:35.652925 4405 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f2c5af93-085e-462b-8377-a6e8139e66c8-istio-data\") pod \"e2e-worker-w5-3b7476-openshift-default-59975457bf-vxwkm\" (UID: \"f2c5af93-085e-462b-8377-a6e8139e66c8\") " pod="openshift-ingress/e2e-worker-w5-3b7476-openshift-default-59975457bf-vxwkm" Nov 04 02:54:35.653165 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:54:35.653130 4405 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/43a58ae9-e100-4e85-bbe0-fdc36af6f192-istiod-ca-cert\") pod \"e2e-worker-w5-3b7476-openshift-default-59975457bf-gph88\" (UID: \"43a58ae9-e100-4e85-bbe0-fdc36af6f192\") " pod="openshift-ingress/e2e-worker-w5-3b7476-openshift-default-59975457bf-gph88" Nov 04 02:54:35.653376 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:54:35.653359 4405 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f2c5af93-085e-462b-8377-a6e8139e66c8-istio-token\") pod \"e2e-worker-w5-3b7476-openshift-default-59975457bf-vxwkm\" (UID: \"f2c5af93-085e-462b-8377-a6e8139e66c8\") " pod="openshift-ingress/e2e-worker-w5-3b7476-openshift-default-59975457bf-vxwkm" Nov 04 02:54:35.653561 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:54:35.653531 4405 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f2c5af93-085e-462b-8377-a6e8139e66c8-istiod-ca-cert\") pod \"e2e-worker-w5-3b7476-openshift-default-59975457bf-vxwkm\" (UID: \"f2c5af93-085e-462b-8377-a6e8139e66c8\") " pod="openshift-ingress/e2e-worker-w5-3b7476-openshift-default-59975457bf-vxwkm" Nov 04 02:54:35.653867 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:54:35.653831 4405 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/43a58ae9-e100-4e85-bbe0-fdc36af6f192-istio-envoy\") pod \"e2e-worker-w5-3b7476-openshift-default-59975457bf-gph88\" (UID: \"43a58ae9-e100-4e85-bbe0-fdc36af6f192\") " pod="openshift-ingress/e2e-worker-w5-3b7476-openshift-default-59975457bf-gph88" Nov 04 02:54:35.654105 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:54:35.654084 4405 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/43a58ae9-e100-4e85-bbe0-fdc36af6f192-istio-token\") pod \"e2e-worker-w5-3b7476-openshift-default-59975457bf-gph88\" (UID: \"43a58ae9-e100-4e85-bbe0-fdc36af6f192\") " pod="openshift-ingress/e2e-worker-w5-3b7476-openshift-default-59975457bf-gph88" Nov 04 02:54:35.654278 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:54:35.654259 4405 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-284m6\" (UniqueName: \"kubernetes.io/projected/f2c5af93-085e-462b-8377-a6e8139e66c8-kube-api-access-284m6\") pod \"e2e-worker-w5-3b7476-openshift-default-59975457bf-vxwkm\" (UID: \"f2c5af93-085e-462b-8377-a6e8139e66c8\") " pod="openshift-ingress/e2e-worker-w5-3b7476-openshift-default-59975457bf-vxwkm" Nov 04 02:54:35.654520 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:54:35.654502 4405 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/43a58ae9-e100-4e85-bbe0-fdc36af6f192-istio-podinfo\") pod \"e2e-worker-w5-3b7476-openshift-default-59975457bf-gph88\" (UID: \"43a58ae9-e100-4e85-bbe0-fdc36af6f192\") " pod="openshift-ingress/e2e-worker-w5-3b7476-openshift-default-59975457bf-gph88" Nov 04 02:54:35.654763 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:54:35.654655 4405 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/43a58ae9-e100-4e85-bbe0-fdc36af6f192-workload-certs\") pod \"e2e-worker-w5-3b7476-openshift-default-59975457bf-gph88\" (UID: \"43a58ae9-e100-4e85-bbe0-fdc36af6f192\") " pod="openshift-ingress/e2e-worker-w5-3b7476-openshift-default-59975457bf-gph88" Nov 04 02:54:35.654925 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:54:35.654903 4405 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/43a58ae9-e100-4e85-bbe0-fdc36af6f192-workload-socket\") pod \"e2e-worker-w5-3b7476-openshift-default-59975457bf-gph88\" (UID: \"43a58ae9-e100-4e85-bbe0-fdc36af6f192\") " pod="openshift-ingress/e2e-worker-w5-3b7476-openshift-default-59975457bf-gph88" Nov 04 02:54:35.655072 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:54:35.655055 4405 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f2c5af93-085e-462b-8377-a6e8139e66c8-istio-envoy\") pod \"e2e-worker-w5-3b7476-openshift-default-59975457bf-vxwkm\" (UID: \"f2c5af93-085e-462b-8377-a6e8139e66c8\") " pod="openshift-ingress/e2e-worker-w5-3b7476-openshift-default-59975457bf-vxwkm" Nov 04 02:54:35.655343 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:54:35.655325 4405 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f2c5af93-085e-462b-8377-a6e8139e66c8-workload-socket\") pod \"e2e-worker-w5-3b7476-openshift-default-59975457bf-vxwkm\" (UID: \"f2c5af93-085e-462b-8377-a6e8139e66c8\") " pod="openshift-ingress/e2e-worker-w5-3b7476-openshift-default-59975457bf-vxwkm" Nov 04 02:54:35.772447 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:54:35.771691 4405 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9ddrz\" (UniqueName: \"kubernetes.io/projected/43a58ae9-e100-4e85-bbe0-fdc36af6f192-kube-api-access-9ddrz\") pod \"e2e-worker-w5-3b7476-openshift-default-59975457bf-gph88\" (UID: \"43a58ae9-e100-4e85-bbe0-fdc36af6f192\") " pod="openshift-ingress/e2e-worker-w5-3b7476-openshift-default-59975457bf-gph88" Nov 04 02:54:35.772447 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:54:35.771791 4405 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f2c5af93-085e-462b-8377-a6e8139e66c8-istio-podinfo\") pod \"e2e-worker-w5-3b7476-openshift-default-59975457bf-vxwkm\" (UID: \"f2c5af93-085e-462b-8377-a6e8139e66c8\") " pod="openshift-ingress/e2e-worker-w5-3b7476-openshift-default-59975457bf-vxwkm" Nov 04 02:54:35.772447 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:54:35.771832 4405 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/43a58ae9-e100-4e85-bbe0-fdc36af6f192-credential-socket\") pod \"e2e-worker-w5-3b7476-openshift-default-59975457bf-gph88\" (UID: \"43a58ae9-e100-4e85-bbe0-fdc36af6f192\") " pod="openshift-ingress/e2e-worker-w5-3b7476-openshift-default-59975457bf-gph88" Nov 04 02:54:35.772447 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:54:35.771861 4405 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/43a58ae9-e100-4e85-bbe0-fdc36af6f192-istio-data\") pod \"e2e-worker-w5-3b7476-openshift-default-59975457bf-gph88\" (UID: \"43a58ae9-e100-4e85-bbe0-fdc36af6f192\") " pod="openshift-ingress/e2e-worker-w5-3b7476-openshift-default-59975457bf-gph88" Nov 04 02:54:35.772447 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:54:35.771947 4405 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f2c5af93-085e-462b-8377-a6e8139e66c8-istio-data\") pod \"e2e-worker-w5-3b7476-openshift-default-59975457bf-vxwkm\" (UID: \"f2c5af93-085e-462b-8377-a6e8139e66c8\") " pod="openshift-ingress/e2e-worker-w5-3b7476-openshift-default-59975457bf-vxwkm" Nov 04 02:54:35.772447 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:54:35.771995 4405 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/43a58ae9-e100-4e85-bbe0-fdc36af6f192-istiod-ca-cert\") pod \"e2e-worker-w5-3b7476-openshift-default-59975457bf-gph88\" (UID: \"43a58ae9-e100-4e85-bbe0-fdc36af6f192\") " pod="openshift-ingress/e2e-worker-w5-3b7476-openshift-default-59975457bf-gph88" Nov 04 02:54:35.772447 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:54:35.772058 4405 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f2c5af93-085e-462b-8377-a6e8139e66c8-istio-token\") pod \"e2e-worker-w5-3b7476-openshift-default-59975457bf-vxwkm\" (UID: \"f2c5af93-085e-462b-8377-a6e8139e66c8\") " pod="openshift-ingress/e2e-worker-w5-3b7476-openshift-default-59975457bf-vxwkm" Nov 04 02:54:35.772447 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:54:35.772107 4405 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f2c5af93-085e-462b-8377-a6e8139e66c8-istiod-ca-cert\") pod \"e2e-worker-w5-3b7476-openshift-default-59975457bf-vxwkm\" (UID: \"f2c5af93-085e-462b-8377-a6e8139e66c8\") " pod="openshift-ingress/e2e-worker-w5-3b7476-openshift-default-59975457bf-vxwkm" Nov 04 02:54:35.772447 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:54:35.772162 4405 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/43a58ae9-e100-4e85-bbe0-fdc36af6f192-istio-envoy\") pod \"e2e-worker-w5-3b7476-openshift-default-59975457bf-gph88\" (UID: \"43a58ae9-e100-4e85-bbe0-fdc36af6f192\") " pod="openshift-ingress/e2e-worker-w5-3b7476-openshift-default-59975457bf-gph88" Nov 04 02:54:35.772447 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:54:35.772210 4405 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/43a58ae9-e100-4e85-bbe0-fdc36af6f192-istio-token\") pod \"e2e-worker-w5-3b7476-openshift-default-59975457bf-gph88\" (UID: \"43a58ae9-e100-4e85-bbe0-fdc36af6f192\") " pod="openshift-ingress/e2e-worker-w5-3b7476-openshift-default-59975457bf-gph88" Nov 04 02:54:35.772447 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:54:35.772242 4405 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-284m6\" (UniqueName: \"kubernetes.io/projected/f2c5af93-085e-462b-8377-a6e8139e66c8-kube-api-access-284m6\") pod \"e2e-worker-w5-3b7476-openshift-default-59975457bf-vxwkm\" (UID: \"f2c5af93-085e-462b-8377-a6e8139e66c8\") " pod="openshift-ingress/e2e-worker-w5-3b7476-openshift-default-59975457bf-vxwkm" Nov 04 02:54:35.772447 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:54:35.772344 4405 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/43a58ae9-e100-4e85-bbe0-fdc36af6f192-istio-podinfo\") pod \"e2e-worker-w5-3b7476-openshift-default-59975457bf-gph88\" (UID: \"43a58ae9-e100-4e85-bbe0-fdc36af6f192\") " pod="openshift-ingress/e2e-worker-w5-3b7476-openshift-default-59975457bf-gph88" Nov 04 02:54:35.772447 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:54:35.772383 4405 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/43a58ae9-e100-4e85-bbe0-fdc36af6f192-workload-certs\") pod \"e2e-worker-w5-3b7476-openshift-default-59975457bf-gph88\" (UID: \"43a58ae9-e100-4e85-bbe0-fdc36af6f192\") " pod="openshift-ingress/e2e-worker-w5-3b7476-openshift-default-59975457bf-gph88" Nov 04 02:54:35.772447 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:54:35.772412 4405 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/43a58ae9-e100-4e85-bbe0-fdc36af6f192-workload-socket\") pod \"e2e-worker-w5-3b7476-openshift-default-59975457bf-gph88\" (UID: \"43a58ae9-e100-4e85-bbe0-fdc36af6f192\") " pod="openshift-ingress/e2e-worker-w5-3b7476-openshift-default-59975457bf-gph88" Nov 04 02:54:35.772447 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:54:35.772438 4405 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f2c5af93-085e-462b-8377-a6e8139e66c8-istio-envoy\") pod \"e2e-worker-w5-3b7476-openshift-default-59975457bf-vxwkm\" (UID: \"f2c5af93-085e-462b-8377-a6e8139e66c8\") " pod="openshift-ingress/e2e-worker-w5-3b7476-openshift-default-59975457bf-vxwkm" Nov 04 02:54:35.773410 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:54:35.772498 4405 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f2c5af93-085e-462b-8377-a6e8139e66c8-workload-socket\") pod \"e2e-worker-w5-3b7476-openshift-default-59975457bf-vxwkm\" (UID: \"f2c5af93-085e-462b-8377-a6e8139e66c8\") " pod="openshift-ingress/e2e-worker-w5-3b7476-openshift-default-59975457bf-vxwkm" Nov 04 02:54:35.773410 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:54:35.772528 4405 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f2c5af93-085e-462b-8377-a6e8139e66c8-workload-certs\") pod \"e2e-worker-w5-3b7476-openshift-default-59975457bf-vxwkm\" (UID: \"f2c5af93-085e-462b-8377-a6e8139e66c8\") " pod="openshift-ingress/e2e-worker-w5-3b7476-openshift-default-59975457bf-vxwkm" Nov 04 02:54:35.773410 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:54:35.772557 4405 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f2c5af93-085e-462b-8377-a6e8139e66c8-credential-socket\") pod \"e2e-worker-w5-3b7476-openshift-default-59975457bf-vxwkm\" (UID: \"f2c5af93-085e-462b-8377-a6e8139e66c8\") " pod="openshift-ingress/e2e-worker-w5-3b7476-openshift-default-59975457bf-vxwkm" Nov 04 02:54:35.773410 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:54:35.773226 4405 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f2c5af93-085e-462b-8377-a6e8139e66c8-credential-socket\") pod \"e2e-worker-w5-3b7476-openshift-default-59975457bf-vxwkm\" (UID: \"f2c5af93-085e-462b-8377-a6e8139e66c8\") " pod="openshift-ingress/e2e-worker-w5-3b7476-openshift-default-59975457bf-vxwkm" Nov 04 02:54:35.777681 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:54:35.775122 4405 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/43a58ae9-e100-4e85-bbe0-fdc36af6f192-credential-socket\") pod \"e2e-worker-w5-3b7476-openshift-default-59975457bf-gph88\" (UID: \"43a58ae9-e100-4e85-bbe0-fdc36af6f192\") " pod="openshift-ingress/e2e-worker-w5-3b7476-openshift-default-59975457bf-gph88" Nov 04 02:54:35.777681 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:54:35.775382 4405 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/43a58ae9-e100-4e85-bbe0-fdc36af6f192-istio-data\") pod \"e2e-worker-w5-3b7476-openshift-default-59975457bf-gph88\" (UID: \"43a58ae9-e100-4e85-bbe0-fdc36af6f192\") " pod="openshift-ingress/e2e-worker-w5-3b7476-openshift-default-59975457bf-gph88" Nov 04 02:54:35.777681 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:54:35.775622 4405 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f2c5af93-085e-462b-8377-a6e8139e66c8-istio-data\") pod \"e2e-worker-w5-3b7476-openshift-default-59975457bf-vxwkm\" (UID: \"f2c5af93-085e-462b-8377-a6e8139e66c8\") " pod="openshift-ingress/e2e-worker-w5-3b7476-openshift-default-59975457bf-vxwkm" Nov 04 02:54:35.777681 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:54:35.776364 4405 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/43a58ae9-e100-4e85-bbe0-fdc36af6f192-istiod-ca-cert\") pod \"e2e-worker-w5-3b7476-openshift-default-59975457bf-gph88\" (UID: \"43a58ae9-e100-4e85-bbe0-fdc36af6f192\") " pod="openshift-ingress/e2e-worker-w5-3b7476-openshift-default-59975457bf-gph88" Nov 04 02:54:35.777681 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:54:35.777309 4405 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f2c5af93-085e-462b-8377-a6e8139e66c8-istiod-ca-cert\") pod \"e2e-worker-w5-3b7476-openshift-default-59975457bf-vxwkm\" (UID: \"f2c5af93-085e-462b-8377-a6e8139e66c8\") " pod="openshift-ingress/e2e-worker-w5-3b7476-openshift-default-59975457bf-vxwkm" Nov 04 02:54:35.788464 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:54:35.788441 4405 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/43a58ae9-e100-4e85-bbe0-fdc36af6f192-workload-socket\") pod \"e2e-worker-w5-3b7476-openshift-default-59975457bf-gph88\" (UID: \"43a58ae9-e100-4e85-bbe0-fdc36af6f192\") " pod="openshift-ingress/e2e-worker-w5-3b7476-openshift-default-59975457bf-gph88" Nov 04 02:54:35.799795 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:54:35.799762 4405 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/43a58ae9-e100-4e85-bbe0-fdc36af6f192-workload-certs\") pod \"e2e-worker-w5-3b7476-openshift-default-59975457bf-gph88\" (UID: \"43a58ae9-e100-4e85-bbe0-fdc36af6f192\") " pod="openshift-ingress/e2e-worker-w5-3b7476-openshift-default-59975457bf-gph88" Nov 04 02:54:35.800437 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:54:35.800414 4405 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f2c5af93-085e-462b-8377-a6e8139e66c8-workload-socket\") pod \"e2e-worker-w5-3b7476-openshift-default-59975457bf-vxwkm\" (UID: \"f2c5af93-085e-462b-8377-a6e8139e66c8\") " pod="openshift-ingress/e2e-worker-w5-3b7476-openshift-default-59975457bf-vxwkm" Nov 04 02:54:35.812692 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:54:35.809199 4405 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f2c5af93-085e-462b-8377-a6e8139e66c8-workload-certs\") pod \"e2e-worker-w5-3b7476-openshift-default-59975457bf-vxwkm\" (UID: \"f2c5af93-085e-462b-8377-a6e8139e66c8\") " pod="openshift-ingress/e2e-worker-w5-3b7476-openshift-default-59975457bf-vxwkm" Nov 04 02:54:35.980769 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:54:35.904866 4405 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-fe4e0c-openshift-default-c7f4fc84c-vh6fd" podStartSLOduration=1.904845322 podStartE2EDuration="1.904845322s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 02:54:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 02:54:35.51541034 +0000 UTC m=+1321.946110460" watchObservedRunningTime="2026-11-04 02:54:35.904845322 +0000 UTC m=+1322.335545443" Nov 04 02:54:35.980769 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:54:35.937180 4405 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/43a58ae9-e100-4e85-bbe0-fdc36af6f192-istio-envoy\") pod \"e2e-worker-w5-3b7476-openshift-default-59975457bf-gph88\" (UID: \"43a58ae9-e100-4e85-bbe0-fdc36af6f192\") " pod="openshift-ingress/e2e-worker-w5-3b7476-openshift-default-59975457bf-gph88" Nov 04 02:54:35.980769 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:54:35.978064 4405 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-fe4e0c-openshift-default-c7f4fc84c-vh6fd" Nov 04 02:54:36.095323 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:54:36.095249 4405 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f2c5af93-085e-462b-8377-a6e8139e66c8-istio-token\") pod \"e2e-worker-w5-3b7476-openshift-default-59975457bf-vxwkm\" (UID: \"f2c5af93-085e-462b-8377-a6e8139e66c8\") " pod="openshift-ingress/e2e-worker-w5-3b7476-openshift-default-59975457bf-vxwkm" Nov 04 02:54:36.096225 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:54:36.096172 4405 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-284m6\" (UniqueName: \"kubernetes.io/projected/f2c5af93-085e-462b-8377-a6e8139e66c8-kube-api-access-284m6\") pod \"e2e-worker-w5-3b7476-openshift-default-59975457bf-vxwkm\" (UID: \"f2c5af93-085e-462b-8377-a6e8139e66c8\") " pod="openshift-ingress/e2e-worker-w5-3b7476-openshift-default-59975457bf-vxwkm" Nov 04 02:54:36.110685 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:54:36.101534 4405 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/43a58ae9-e100-4e85-bbe0-fdc36af6f192-istio-token\") pod \"e2e-worker-w5-3b7476-openshift-default-59975457bf-gph88\" (UID: \"43a58ae9-e100-4e85-bbe0-fdc36af6f192\") " pod="openshift-ingress/e2e-worker-w5-3b7476-openshift-default-59975457bf-gph88" Nov 04 02:54:36.132243 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:54:36.113908 4405 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f2c5af93-085e-462b-8377-a6e8139e66c8-istio-envoy\") pod \"e2e-worker-w5-3b7476-openshift-default-59975457bf-vxwkm\" (UID: \"f2c5af93-085e-462b-8377-a6e8139e66c8\") " pod="openshift-ingress/e2e-worker-w5-3b7476-openshift-default-59975457bf-vxwkm" Nov 04 02:54:36.158754 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:54:36.135294 4405 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/43a58ae9-e100-4e85-bbe0-fdc36af6f192-istio-podinfo\") pod \"e2e-worker-w5-3b7476-openshift-default-59975457bf-gph88\" (UID: \"43a58ae9-e100-4e85-bbe0-fdc36af6f192\") " pod="openshift-ingress/e2e-worker-w5-3b7476-openshift-default-59975457bf-gph88" Nov 04 02:54:36.158754 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:54:36.149951 4405 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f2c5af93-085e-462b-8377-a6e8139e66c8-istio-podinfo\") pod \"e2e-worker-w5-3b7476-openshift-default-59975457bf-vxwkm\" (UID: \"f2c5af93-085e-462b-8377-a6e8139e66c8\") " pod="openshift-ingress/e2e-worker-w5-3b7476-openshift-default-59975457bf-vxwkm" Nov 04 02:54:36.174327 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:54:36.173844 4405 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9ddrz\" (UniqueName: \"kubernetes.io/projected/43a58ae9-e100-4e85-bbe0-fdc36af6f192-kube-api-access-9ddrz\") pod \"e2e-worker-w5-3b7476-openshift-default-59975457bf-gph88\" (UID: \"43a58ae9-e100-4e85-bbe0-fdc36af6f192\") " pod="openshift-ingress/e2e-worker-w5-3b7476-openshift-default-59975457bf-gph88" Nov 04 02:54:36.209694 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:54:36.207314 4405 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-3b7476-openshift-default-59975457bf-gph88" Nov 04 02:54:36.317706 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:54:36.311408 4405 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-3b7476-openshift-default-59975457bf-vxwkm" Nov 04 02:54:36.774932 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:54:36.769616 4405 patch_prober.go:29] interesting pod/e2e-worker-w0-fe4e0c-openshift-default-c7f4fc84c-vh6fd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused" start-of-body= Nov 04 02:54:36.774932 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:54:36.769742 4405 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-fe4e0c-openshift-default-c7f4fc84c-vh6fd" podUID="70d9a666-2667-41e8-8425-4e2ba085599d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused" Nov 04 02:54:36.829694 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:54:36.803786 4405 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-203c35/simulator-w4-203c35-kserve-f9c4bd695-999ns" podUID="eb0d8bdd-6808-4fd0-b9b5-567dff3e7ca5" containerName="main" probeResult="failure" output="Get \"https://10.134.0.46:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 02:54:36.829694 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:54:36.817767 4405 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-a0bcbc/distinct-w2-a0bcbc-kserve-596f94c7cd-szl4g" podUID="bac32934-8a57-4919-b435-13f367cbea3e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 02:54:36.843229 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:54:36.843184 4405 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-fe4e0c/distinct-2-w0-fe4e0c-kserve-579bf4cc77-8pbq8" podUID="2b2ab722-d1bb-4376-bede-f5f4ee4562aa" containerName="main" probeResult="failure" output="Get \"https://10.134.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 02:54:36.996915 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:54:36.996870 4405 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-ljhhw" podUID="f66b1861-d87b-4d57-ab27-f1a24544a534" containerName="main" probeResult="failure" output="Get \"https://10.134.0.26:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 02:54:37.165709 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:54:37.159810 4405 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-cdcf67/distinct-w3-cdcf67-kserve-667cc7f7d7-5m2lq" podUID="52ed22cc-96e4-4ef0-b34d-ed58c89a9183" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 04 02:54:37.224707 ip-10-0-134-125 kubenswrapper[4405]: E1104 02:54:37.222047 4405 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5f2vb" containerName="extractor" Nov 04 02:54:37.785694 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:54:37.781648 4405 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-ebb7fd/distinct-2-w6-ebb7fd-kserve-6f68c5db86-zrpjs" podUID="b7b786e0-9c94-4126-9949-cd354ebbfb2a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 02:54:37.841694 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:54:37.825328 4405 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-203c35/simulator-w4-203c35-kserve-f9c4bd695-999ns" podUID="eb0d8bdd-6808-4fd0-b9b5-567dff3e7ca5" containerName="main" probeResult="failure" output="Get \"https://10.134.0.46:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 02:54:37.841694 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:54:37.801210 4405 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-e516ff/distinct-w1-e516ff-kserve-f4474cb9c-glpzw" podUID="8842ad09-22ff-4492-ba5d-d4cacf4eac75" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/ready\": context deadline exceeded" Nov 04 02:54:37.841694 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:54:37.821864 4405 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-ebb7fd/distinct-w6-ebb7fd-kserve-55c47d8865-7z9pz" podUID="ccb8e181-5610-46da-bb8b-e0b07fa02ff1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": context deadline exceeded" Nov 04 02:54:37.841694 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:54:37.823831 4405 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-a0bcbc/distinct-w2-a0bcbc-kserve-596f94c7cd-szl4g" podUID="bac32934-8a57-4919-b435-13f367cbea3e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 02:54:37.841694 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:54:37.821944 4405 patch_prober.go:29] interesting pod/e2e-worker-w0-fe4e0c-openshift-default-c7f4fc84c-vh6fd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused" start-of-body= Nov 04 02:54:37.841694 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:54:37.835201 4405 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-fe4e0c-openshift-default-c7f4fc84c-vh6fd" podUID="70d9a666-2667-41e8-8425-4e2ba085599d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused" Nov 04 02:54:37.841694 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:54:37.836515 4405 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-a0bcbc/premium-w2-a0bcbc-kserve-78dfd6d78c-lwth2" podUID="33b620c1-389c-48b4-b277-90252018c4b6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/ready\": context deadline exceeded" Nov 04 02:54:37.903479 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:54:37.854904 4405 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-fe4e0c/distinct-2-w0-fe4e0c-kserve-579bf4cc77-8pbq8" podUID="2b2ab722-d1bb-4376-bede-f5f4ee4562aa" containerName="main" probeResult="failure" output="Get \"https://10.134.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 02:54:37.964694 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:54:37.953145 4405 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-203c35/simulator-w4-203c35-kserve-f9c4bd695-999ns" Nov 04 02:54:37.977981 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:54:37.977952 4405 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-a0bcbc/distinct-w2-a0bcbc-kserve-596f94c7cd-szl4g" Nov 04 02:54:37.981693 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:54:37.978244 4405 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-fe4e0c/distinct-2-w0-fe4e0c-kserve-579bf4cc77-8pbq8" Nov 04 02:54:38.019900 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:54:38.019801 4405 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-ljhhw" podUID="f66b1861-d87b-4d57-ab27-f1a24544a534" containerName="main" probeResult="failure" output="Get \"https://10.134.0.26:8000/ready\": context deadline exceeded" Nov 04 02:54:38.020636 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:54:38.020610 4405 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-ljhhw" Nov 04 02:54:38.126707 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:54:38.092020 4405 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-a0bcbc/distinct-w2-a0bcbc-kserve-596f94c7cd-szl4g" podUID="bac32934-8a57-4919-b435-13f367cbea3e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/health\": context deadline exceeded" Nov 04 02:54:38.235948 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:54:38.193045 4405 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-3b7476-openshift-default-59975457bf-gph88"] Nov 04 02:54:38.235948 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:54:38.199830 4405 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-cdcf67/distinct-w3-cdcf67-kserve-667cc7f7d7-5m2lq" podUID="52ed22cc-96e4-4ef0-b34d-ed58c89a9183" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 02:54:38.235948 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:54:38.206864 4405 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-cdcf67/distinct-w3-cdcf67-kserve-667cc7f7d7-5m2lq" Nov 04 02:54:38.388732 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:54:38.345607 4405 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 04 02:54:38.388732 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:54:38.345731 4405 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 04 02:54:38.388732 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:54:38.345783 4405 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 04 02:54:38.779325 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:54:38.779291 4405 patch_prober.go:29] interesting pod/e2e-worker-w0-fe4e0c-openshift-default-c7f4fc84c-vh6fd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused" start-of-body= Nov 04 02:54:38.809759 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:54:38.779593 4405 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-fe4e0c-openshift-default-c7f4fc84c-vh6fd" podUID="70d9a666-2667-41e8-8425-4e2ba085599d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused" Nov 04 02:54:38.953697 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:54:38.862914 4405 prober.go:122] "Probe failed" probeType="Liveness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-ljhhw" podUID="f66b1861-d87b-4d57-ab27-f1a24544a534" containerName="main" probeResult="failure" output="Get \"https://10.134.0.26:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 02:54:38.953697 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:54:38.863777 4405 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-ebb7fd/distinct-2-w6-ebb7fd-kserve-6f68c5db86-zrpjs" podUID="b7b786e0-9c94-4126-9949-cd354ebbfb2a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 02:54:38.976103 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:54:38.892422 4405 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-e516ff/distinct-w1-e516ff-kserve-f4474cb9c-glpzw" podUID="8842ad09-22ff-4492-ba5d-d4cacf4eac75" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/ready\": context deadline exceeded" Nov 04 02:54:38.976103 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:54:38.892745 4405 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-203c35/simulator-w4-203c35-kserve-f9c4bd695-999ns" podUID="eb0d8bdd-6808-4fd0-b9b5-567dff3e7ca5" containerName="main" probeResult="failure" output="Get \"https://10.134.0.46:8000/ready\": context deadline exceeded" Nov 04 02:54:38.976103 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:54:38.893499 4405 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-a0bcbc/premium-w2-a0bcbc-kserve-78dfd6d78c-lwth2" podUID="33b620c1-389c-48b4-b277-90252018c4b6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 02:54:38.976103 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:54:38.893529 4405 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-ebb7fd/distinct-w6-ebb7fd-kserve-55c47d8865-7z9pz" podUID="ccb8e181-5610-46da-bb8b-e0b07fa02ff1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 02:54:38.976103 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:54:38.893549 4405 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-fe4e0c/distinct-2-w0-fe4e0c-kserve-579bf4cc77-8pbq8" podUID="2b2ab722-d1bb-4376-bede-f5f4ee4562aa" containerName="main" probeResult="failure" output="Get \"https://10.134.0.68:8000/ready\": context deadline exceeded" Nov 04 02:54:38.976103 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:54:38.894254 4405 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-a0bcbc/distinct-w2-a0bcbc-kserve-596f94c7cd-szl4g" podUID="bac32934-8a57-4919-b435-13f367cbea3e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": context deadline exceeded" Nov 04 02:54:38.976103 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:54:38.940814 4405 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-ebb7fd/distinct-w6-ebb7fd-kserve-55c47d8865-7z9pz" podUID="ccb8e181-5610-46da-bb8b-e0b07fa02ff1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 02:54:39.094047 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:54:39.023800 4405 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-ljhhw" podUID="f66b1861-d87b-4d57-ab27-f1a24544a534" containerName="main" probeResult="failure" output="Get \"https://10.134.0.26:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 02:54:39.094047 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:54:39.089809 4405 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-ebb7fd/distinct-2-w6-ebb7fd-kserve-6f68c5db86-zrpjs" Nov 04 02:54:39.094047 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:54:39.090962 4405 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-e516ff/distinct-w1-e516ff-kserve-f4474cb9c-glpzw" Nov 04 02:54:39.095846 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:54:39.095820 4405 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-a0bcbc/premium-w2-a0bcbc-kserve-78dfd6d78c-lwth2" Nov 04 02:54:39.648696 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:54:39.488424 4405 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-ebb7fd/distinct-w6-ebb7fd-kserve-55c47d8865-7z9pz" Nov 04 02:54:39.648696 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:54:39.635984 4405 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-cdcf67/distinct-w3-cdcf67-kserve-667cc7f7d7-5m2lq" podUID="52ed22cc-96e4-4ef0-b34d-ed58c89a9183" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 02:54:39.750784 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:54:39.745342 4405 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-a0bcbc/premium-w2-a0bcbc-kserve-78dfd6d78c-lwth2" podUID="33b620c1-389c-48b4-b277-90252018c4b6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/health\": context deadline exceeded" Nov 04 02:54:39.857768 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:54:39.857635 4405 patch_prober.go:29] interesting pod/e2e-worker-w0-fe4e0c-openshift-default-c7f4fc84c-vh6fd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused" start-of-body= Nov 04 02:54:39.907481 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:54:39.907375 4405 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-fe4e0c-openshift-default-c7f4fc84c-vh6fd" podUID="70d9a666-2667-41e8-8425-4e2ba085599d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused" Nov 04 02:54:40.107627 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:54:40.102845 4405 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-cdcf67/distinct-w3-cdcf67-kserve-667cc7f7d7-5m2lq" podUID="52ed22cc-96e4-4ef0-b34d-ed58c89a9183" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/health\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 04 02:54:40.107627 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:54:40.103101 4405 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-a0bcbc/distinct-w2-a0bcbc-kserve-596f94c7cd-szl4g" podUID="bac32934-8a57-4919-b435-13f367cbea3e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 02:54:40.107627 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:54:40.103500 4405 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-203c35/simulator-w4-203c35-kserve-f9c4bd695-999ns" podUID="eb0d8bdd-6808-4fd0-b9b5-567dff3e7ca5" containerName="main" probeResult="failure" output="Get \"https://10.134.0.46:8000/ready\": context deadline exceeded" Nov 04 02:54:40.182696 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:54:40.180845 4405 patch_prober.go:29] interesting pod/e2e-worker-w2-a0bcbc-openshift-default-65dc9d9b6c-m97sr container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.49:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 04 02:54:40.261576 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:54:40.180909 4405 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-a0bcbc-openshift-default-65dc9d9b6c-m97sr" podUID="9a748832-c6ef-484f-8635-8c9148a38149" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.49:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 04 02:54:40.390942 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:54:40.390803 4405 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/echo-simulator-kserve-64b6f8f96c-2lg6v" podUID="fe1fd7ca-7aad-4761-abe0-78bbd438ffad" containerName="main" probeResult="failure" output="Get \"https://10.134.0.27:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 02:54:40.419793 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:54:40.416412 4405 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-gnjj2" podUID="22d477a6-9ab8-4330-ae26-813caff99cb3" containerName="main" probeResult="failure" output="Get \"https://10.134.0.28:8000/ready\": context deadline exceeded" Nov 04 02:54:40.419793 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:54:40.416737 4405 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-ebb7fd/distinct-2-w6-ebb7fd-kserve-6f68c5db86-zrpjs" podUID="b7b786e0-9c94-4126-9949-cd354ebbfb2a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 02:54:40.453701 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:54:40.420089 4405 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-e516ff/distinct-w1-e516ff-kserve-f4474cb9c-glpzw" podUID="8842ad09-22ff-4492-ba5d-d4cacf4eac75" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 02:54:40.453701 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:54:40.420356 4405 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-ebb7fd/distinct-w6-ebb7fd-kserve-55c47d8865-7z9pz" podUID="ccb8e181-5610-46da-bb8b-e0b07fa02ff1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": context deadline exceeded" Nov 04 02:54:40.453701 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:54:40.420540 4405 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-a0bcbc/premium-w2-a0bcbc-kserve-78dfd6d78c-lwth2" podUID="33b620c1-389c-48b4-b277-90252018c4b6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/ready\": context deadline exceeded" Nov 04 02:54:40.453701 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:54:40.421215 4405 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-ljhhw" podUID="f66b1861-d87b-4d57-ab27-f1a24544a534" containerName="main" probeResult="failure" output="Get \"https://10.134.0.26:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 02:54:40.453701 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:54:40.421623 4405 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-fe4e0c/distinct-2-w0-fe4e0c-kserve-579bf4cc77-8pbq8" podUID="2b2ab722-d1bb-4376-bede-f5f4ee4562aa" containerName="main" probeResult="failure" output="Get \"https://10.134.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 02:54:40.818467 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:54:40.800203 4405 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-cdcf67/distinct-w3-cdcf67-kserve-667cc7f7d7-5m2lq" podUID="52ed22cc-96e4-4ef0-b34d-ed58c89a9183" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 02:54:40.845577 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:54:40.844087 4405 patch_prober.go:29] interesting pod/e2e-worker-w0-fe4e0c-openshift-default-c7f4fc84c-vh6fd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused" start-of-body= Nov 04 02:54:40.845577 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:54:40.844151 4405 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-fe4e0c-openshift-default-c7f4fc84c-vh6fd" podUID="70d9a666-2667-41e8-8425-4e2ba085599d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused" Nov 04 02:54:41.131366 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:54:41.130796 4405 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-203c35/simulator-w4-203c35-kserve-f9c4bd695-999ns" podUID="eb0d8bdd-6808-4fd0-b9b5-567dff3e7ca5" containerName="main" probeResult="failure" output="Get \"https://10.134.0.46:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 02:54:41.133872 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:54:41.133026 4405 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-a0bcbc/distinct-w2-a0bcbc-kserve-596f94c7cd-szl4g" podUID="bac32934-8a57-4919-b435-13f367cbea3e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": context deadline exceeded" Nov 04 02:54:41.138294 ip-10-0-134-125 kubenswrapper[4405]: E1104 02:54:41.138268 4405 kubelet.go:2711] "Housekeeping took longer than expected" err="housekeeping took too long" expected="1s" actual="1.361s" Nov 04 02:54:41.138577 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:54:41.138558 4405 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-3b7476-openshift-default-59975457bf-vxwkm" event={"ID":"f2c5af93-085e-462b-8377-a6e8139e66c8","Type":"ContainerStarted","Data":"05f95d319febc023166e3161b4d00c1f4193b61800628c9176f51e493753377e"} Nov 04 02:54:41.138760 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:54:41.138744 4405 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-3b7476-openshift-default-59975457bf-gph88" event={"ID":"43a58ae9-e100-4e85-bbe0-fdc36af6f192","Type":"ContainerStarted","Data":"1bb94f627433b1d2903194bb1ecd293f3762ced4bb07cf9c29a70a438528dc9c"} Nov 04 02:54:41.262747 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:54:41.242125 4405 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 04 02:54:41.262747 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:54:41.242235 4405 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 04 02:54:41.262747 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:54:41.242282 4405 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 04 02:54:41.366931 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:54:41.356265 4405 patch_prober.go:29] interesting pod/e2e-worker-w2-a0bcbc-openshift-default-65dc9d9b6c-j4n6p container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.54:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 04 02:54:41.366931 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:54:41.356384 4405 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-a0bcbc-openshift-default-65dc9d9b6c-j4n6p" podUID="55473cc8-d292-4f5f-bb17-3f08c2e19e8c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.54:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 04 02:54:41.439079 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:54:41.438977 4405 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-ebb7fd/distinct-2-w6-ebb7fd-kserve-6f68c5db86-zrpjs" podUID="b7b786e0-9c94-4126-9949-cd354ebbfb2a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 02:54:41.502706 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:54:41.501751 4405 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-fe4e0c/distinct-2-w0-fe4e0c-kserve-579bf4cc77-8pbq8" podUID="2b2ab722-d1bb-4376-bede-f5f4ee4562aa" containerName="main" probeResult="failure" output="Get \"https://10.134.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 02:54:41.513996 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:54:41.513946 4405 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-ljhhw" podUID="f66b1861-d87b-4d57-ab27-f1a24544a534" containerName="main" probeResult="failure" output="Get \"https://10.134.0.26:8000/ready\": context deadline exceeded" Nov 04 02:54:41.544472 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:54:41.531946 4405 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-ebb7fd/distinct-w6-ebb7fd-kserve-55c47d8865-7z9pz" podUID="ccb8e181-5610-46da-bb8b-e0b07fa02ff1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": context deadline exceeded" Nov 04 02:54:41.546002 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:54:41.531980 4405 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-e516ff/distinct-w1-e516ff-kserve-f4474cb9c-glpzw" podUID="8842ad09-22ff-4492-ba5d-d4cacf4eac75" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/ready\": context deadline exceeded" Nov 04 02:54:41.555706 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:54:41.534978 4405 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-a0bcbc/premium-w2-a0bcbc-kserve-78dfd6d78c-lwth2" podUID="33b620c1-389c-48b4-b277-90252018c4b6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 02:54:41.556171 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:54:41.536818 4405 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-203c35/simulator-w4-203c35-kserve-f9c4bd695-999ns" Nov 04 02:54:41.704757 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:54:41.677734 4405 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-ebb7fd/distinct-2-w6-ebb7fd-kserve-6f68c5db86-zrpjs" podUID="b7b786e0-9c94-4126-9949-cd354ebbfb2a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.65:8000/health\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 04 02:54:41.785640 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:54:41.785608 4405 patch_prober.go:29] interesting pod/e2e-worker-w0-fe4e0c-openshift-default-c7f4fc84c-vh6fd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused" start-of-body= Nov 04 02:54:41.785911 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:54:41.785886 4405 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-fe4e0c-openshift-default-c7f4fc84c-vh6fd" podUID="70d9a666-2667-41e8-8425-4e2ba085599d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused" Nov 04 02:54:41.808095 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:54:41.808036 4405 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-cdcf67/distinct-w3-cdcf67-kserve-667cc7f7d7-5m2lq" podUID="52ed22cc-96e4-4ef0-b34d-ed58c89a9183" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 02:54:41.866098 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:54:41.866066 4405 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-3b7476-openshift-default-59975457bf-vxwkm"] Nov 04 02:54:41.866388 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:54:41.866371 4405 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-a0bcbc/distinct-w2-a0bcbc-kserve-596f94c7cd-szl4g" Nov 04 02:54:41.866557 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:54:41.866546 4405 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-ebb7fd/distinct-w6-ebb7fd-kserve-55c47d8865-7z9pz" Nov 04 02:54:41.869019 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:54:41.868845 4405 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-e516ff/distinct-w1-e516ff-kserve-f4474cb9c-glpzw" Nov 04 02:54:41.871514 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:54:41.871472 4405 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-ebb7fd/distinct-2-w6-ebb7fd-kserve-6f68c5db86-zrpjs" Nov 04 02:54:41.871642 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:54:41.871566 4405 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-a0bcbc/premium-w2-a0bcbc-kserve-78dfd6d78c-lwth2" Nov 04 02:54:41.906530 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:54:41.905901 4405 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-ljhhw" Nov 04 02:54:41.933707 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:54:41.932073 4405 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-cdcf67/distinct-w3-cdcf67-kserve-667cc7f7d7-5m2lq" Nov 04 02:54:41.933707 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:54:41.933364 4405 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-fe4e0c/distinct-2-w0-fe4e0c-kserve-579bf4cc77-8pbq8" Nov 04 02:54:42.732326 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:54:42.732191 4405 patch_prober.go:29] interesting pod/e2e-worker-w0-fe4e0c-openshift-default-c7f4fc84c-vh6fd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused" start-of-body= Nov 04 02:54:42.732326 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:54:42.732262 4405 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-fe4e0c-openshift-default-c7f4fc84c-vh6fd" podUID="70d9a666-2667-41e8-8425-4e2ba085599d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused" Nov 04 02:54:43.189696 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:54:43.189202 4405 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-3b7476-openshift-default-59975457bf-gph88" event={"ID":"43a58ae9-e100-4e85-bbe0-fdc36af6f192","Type":"ContainerStarted","Data":"c64cebb6a88c524d1824f445d30518a0d02ce7ef085793001775e7011068f093"} Nov 04 02:54:43.194249 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:54:43.194193 4405 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-3b7476-openshift-default-59975457bf-vxwkm" event={"ID":"f2c5af93-085e-462b-8377-a6e8139e66c8","Type":"ContainerStarted","Data":"fd6959d2c3e3f6141a6342ce834069df4084a2f029a4812d29904c9c0ac88013"} Nov 04 02:54:43.212509 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:54:43.212474 4405 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w5-3b7476-openshift-default-59975457bf-gph88" Nov 04 02:54:43.245363 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:54:43.245324 4405 patch_prober.go:29] interesting pod/e2e-worker-w5-3b7476-openshift-default-59975457bf-gph88 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused" start-of-body= Nov 04 02:54:43.245692 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:54:43.245635 4405 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-3b7476-openshift-default-59975457bf-gph88" podUID="43a58ae9-e100-4e85-bbe0-fdc36af6f192" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused" Nov 04 02:54:43.336604 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:54:43.334890 4405 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w5-3b7476-openshift-default-59975457bf-vxwkm" Nov 04 02:54:43.337060 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:54:43.337035 4405 patch_prober.go:29] interesting pod/e2e-worker-w5-3b7476-openshift-default-59975457bf-vxwkm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" start-of-body= Nov 04 02:54:43.337279 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:54:43.337247 4405 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-3b7476-openshift-default-59975457bf-vxwkm" podUID="f2c5af93-085e-462b-8377-a6e8139e66c8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" Nov 04 02:54:43.362382 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:54:43.362315 4405 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w5-3b7476-openshift-default-59975457bf-gph88" podStartSLOduration=8.362290666 podStartE2EDuration="8.362290666s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 02:54:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 02:54:43.317183869 +0000 UTC m=+1329.747883990" watchObservedRunningTime="2026-11-04 02:54:43.362290666 +0000 UTC m=+1329.792990786" Nov 04 02:54:43.769236 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:54:43.769093 4405 patch_prober.go:29] interesting pod/e2e-worker-w0-fe4e0c-openshift-default-c7f4fc84c-vh6fd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused" start-of-body= Nov 04 02:54:43.769236 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:54:43.769157 4405 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-fe4e0c-openshift-default-c7f4fc84c-vh6fd" podUID="70d9a666-2667-41e8-8425-4e2ba085599d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused" Nov 04 02:54:44.275202 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:54:44.275166 4405 patch_prober.go:29] interesting pod/e2e-worker-w5-3b7476-openshift-default-59975457bf-gph88 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused" start-of-body= Nov 04 02:54:44.275451 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:54:44.275425 4405 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-3b7476-openshift-default-59975457bf-gph88" podUID="43a58ae9-e100-4e85-bbe0-fdc36af6f192" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused" Nov 04 02:54:44.330706 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:54:44.330145 4405 patch_prober.go:29] interesting pod/e2e-worker-w5-3b7476-openshift-default-59975457bf-vxwkm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" start-of-body= Nov 04 02:54:44.330706 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:54:44.330208 4405 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-3b7476-openshift-default-59975457bf-vxwkm" podUID="f2c5af93-085e-462b-8377-a6e8139e66c8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" Nov 04 02:54:44.730695 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:54:44.729816 4405 patch_prober.go:29] interesting pod/e2e-worker-w0-fe4e0c-openshift-default-c7f4fc84c-vh6fd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused" start-of-body= Nov 04 02:54:44.730695 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:54:44.729881 4405 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-fe4e0c-openshift-default-c7f4fc84c-vh6fd" podUID="70d9a666-2667-41e8-8425-4e2ba085599d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused" Nov 04 02:54:45.224692 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:54:45.221729 4405 patch_prober.go:29] interesting pod/e2e-worker-w5-3b7476-openshift-default-59975457bf-gph88 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused" start-of-body= Nov 04 02:54:45.224692 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:54:45.221792 4405 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-3b7476-openshift-default-59975457bf-gph88" podUID="43a58ae9-e100-4e85-bbe0-fdc36af6f192" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused" Nov 04 02:54:45.329939 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:54:45.329900 4405 patch_prober.go:29] interesting pod/e2e-worker-w5-3b7476-openshift-default-59975457bf-vxwkm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" start-of-body= Nov 04 02:54:45.330206 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:54:45.330181 4405 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-3b7476-openshift-default-59975457bf-vxwkm" podUID="f2c5af93-085e-462b-8377-a6e8139e66c8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" Nov 04 02:54:45.785699 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:54:45.767815 4405 patch_prober.go:29] interesting pod/e2e-worker-w0-fe4e0c-openshift-default-c7f4fc84c-vh6fd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused" start-of-body= Nov 04 02:54:45.785699 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:54:45.767893 4405 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-fe4e0c-openshift-default-c7f4fc84c-vh6fd" podUID="70d9a666-2667-41e8-8425-4e2ba085599d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused" Nov 04 02:54:46.226695 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:54:46.222806 4405 patch_prober.go:29] interesting pod/e2e-worker-w5-3b7476-openshift-default-59975457bf-gph88 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused" start-of-body= Nov 04 02:54:46.226695 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:54:46.222886 4405 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-3b7476-openshift-default-59975457bf-gph88" podUID="43a58ae9-e100-4e85-bbe0-fdc36af6f192" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused" Nov 04 02:54:46.336700 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:54:46.334264 4405 patch_prober.go:29] interesting pod/e2e-worker-w5-3b7476-openshift-default-59975457bf-vxwkm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" start-of-body= Nov 04 02:54:46.336700 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:54:46.334349 4405 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-3b7476-openshift-default-59975457bf-vxwkm" podUID="f2c5af93-085e-462b-8377-a6e8139e66c8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" Nov 04 02:54:46.734649 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:54:46.734612 4405 patch_prober.go:29] interesting pod/e2e-worker-w0-fe4e0c-openshift-default-c7f4fc84c-vh6fd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused" start-of-body= Nov 04 02:54:46.734992 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:54:46.734952 4405 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-fe4e0c-openshift-default-c7f4fc84c-vh6fd" podUID="70d9a666-2667-41e8-8425-4e2ba085599d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused" Nov 04 02:54:47.214159 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:54:47.214119 4405 patch_prober.go:29] interesting pod/e2e-worker-w5-3b7476-openshift-default-59975457bf-gph88 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused" start-of-body= Nov 04 02:54:47.214495 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:54:47.214457 4405 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-3b7476-openshift-default-59975457bf-gph88" podUID="43a58ae9-e100-4e85-bbe0-fdc36af6f192" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused" Nov 04 02:54:47.369695 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:54:47.369564 4405 patch_prober.go:29] interesting pod/e2e-worker-w5-3b7476-openshift-default-59975457bf-vxwkm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" start-of-body= Nov 04 02:54:47.369695 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:54:47.369633 4405 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-3b7476-openshift-default-59975457bf-vxwkm" podUID="f2c5af93-085e-462b-8377-a6e8139e66c8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" Nov 04 02:54:48.211706 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:54:48.211085 4405 patch_prober.go:29] interesting pod/e2e-worker-w5-3b7476-openshift-default-59975457bf-gph88 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused" start-of-body= Nov 04 02:54:48.211706 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:54:48.211162 4405 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-3b7476-openshift-default-59975457bf-gph88" podUID="43a58ae9-e100-4e85-bbe0-fdc36af6f192" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused" Nov 04 02:54:48.258395 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:54:48.258350 4405 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-fe4e0c-openshift-default-c7f4fc84c-vh6fd" Nov 04 02:54:48.259811 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:54:48.259785 4405 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-fe4e0c-openshift-default-c7f4fc84c-vh6fd" Nov 04 02:54:48.260424 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:54:48.260407 4405 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-fe4e0c-openshift-default-c7f4fc84c-vh6fd" Nov 04 02:54:48.328355 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:54:48.328318 4405 patch_prober.go:29] interesting pod/e2e-worker-w5-3b7476-openshift-default-59975457bf-vxwkm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" start-of-body= Nov 04 02:54:48.328542 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:54:48.328369 4405 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-3b7476-openshift-default-59975457bf-vxwkm" podUID="f2c5af93-085e-462b-8377-a6e8139e66c8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" Nov 04 02:54:48.389218 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:54:48.389143 4405 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w5-3b7476-openshift-default-59975457bf-vxwkm" podStartSLOduration=13.389126613 podStartE2EDuration="13.389126613s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 02:54:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 02:54:43.46311497 +0000 UTC m=+1329.893815093" watchObservedRunningTime="2026-11-04 02:54:48.389126613 +0000 UTC m=+1334.819826735" Nov 04 02:54:49.210300 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:54:49.210256 4405 patch_prober.go:29] interesting pod/e2e-worker-w5-3b7476-openshift-default-59975457bf-gph88 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused" start-of-body= Nov 04 02:54:49.210485 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:54:49.210336 4405 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-3b7476-openshift-default-59975457bf-gph88" podUID="43a58ae9-e100-4e85-bbe0-fdc36af6f192" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused" Nov 04 02:54:49.329062 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:54:49.329026 4405 patch_prober.go:29] interesting pod/e2e-worker-w5-3b7476-openshift-default-59975457bf-vxwkm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" start-of-body= Nov 04 02:54:49.329240 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:54:49.329091 4405 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-3b7476-openshift-default-59975457bf-vxwkm" podUID="f2c5af93-085e-462b-8377-a6e8139e66c8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" Nov 04 02:54:50.251628 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:54:50.249068 4405 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w5-3b7476-openshift-default-59975457bf-gph88" Nov 04 02:54:50.251628 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:54:50.250432 4405 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w5-3b7476-openshift-default-59975457bf-gph88" Nov 04 02:54:50.253509 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:54:50.253473 4405 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w5-3b7476-openshift-default-59975457bf-gph88" Nov 04 02:54:50.333226 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:54:50.333197 4405 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w5-3b7476-openshift-default-59975457bf-vxwkm" Nov 04 02:54:50.333496 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:54:50.333473 4405 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w5-3b7476-openshift-default-59975457bf-vxwkm" Nov 04 02:54:50.334281 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:54:50.334259 4405 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w5-3b7476-openshift-default-59975457bf-vxwkm" Nov 04 02:54:50.855587 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:54:50.855552 4405 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w6-ebb7fd-openshift-default-544f5768f-47dj7"] Nov 04 02:54:50.861019 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:54:50.860990 4405 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-ebb7fd-openshift-default-544f5768f-47dj7" Nov 04 02:54:50.932738 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:54:50.932697 4405 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8762bab6-0f89-4b6b-a87a-dc1555608862-istio-podinfo\") pod \"e2e-worker-w6-ebb7fd-openshift-default-544f5768f-47dj7\" (UID: \"8762bab6-0f89-4b6b-a87a-dc1555608862\") " pod="openshift-ingress/e2e-worker-w6-ebb7fd-openshift-default-544f5768f-47dj7" Nov 04 02:54:50.932993 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:54:50.932974 4405 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rfsbz\" (UniqueName: \"kubernetes.io/projected/8762bab6-0f89-4b6b-a87a-dc1555608862-kube-api-access-rfsbz\") pod \"e2e-worker-w6-ebb7fd-openshift-default-544f5768f-47dj7\" (UID: \"8762bab6-0f89-4b6b-a87a-dc1555608862\") " pod="openshift-ingress/e2e-worker-w6-ebb7fd-openshift-default-544f5768f-47dj7" Nov 04 02:54:50.933144 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:54:50.933129 4405 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8762bab6-0f89-4b6b-a87a-dc1555608862-istio-envoy\") pod \"e2e-worker-w6-ebb7fd-openshift-default-544f5768f-47dj7\" (UID: \"8762bab6-0f89-4b6b-a87a-dc1555608862\") " pod="openshift-ingress/e2e-worker-w6-ebb7fd-openshift-default-544f5768f-47dj7" Nov 04 02:54:50.933399 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:54:50.933376 4405 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8762bab6-0f89-4b6b-a87a-dc1555608862-workload-certs\") pod \"e2e-worker-w6-ebb7fd-openshift-default-544f5768f-47dj7\" (UID: \"8762bab6-0f89-4b6b-a87a-dc1555608862\") " pod="openshift-ingress/e2e-worker-w6-ebb7fd-openshift-default-544f5768f-47dj7" Nov 04 02:54:50.933529 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:54:50.933501 4405 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8762bab6-0f89-4b6b-a87a-dc1555608862-istio-token\") pod \"e2e-worker-w6-ebb7fd-openshift-default-544f5768f-47dj7\" (UID: \"8762bab6-0f89-4b6b-a87a-dc1555608862\") " pod="openshift-ingress/e2e-worker-w6-ebb7fd-openshift-default-544f5768f-47dj7" Nov 04 02:54:50.933636 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:54:50.933606 4405 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8762bab6-0f89-4b6b-a87a-dc1555608862-istiod-ca-cert\") pod \"e2e-worker-w6-ebb7fd-openshift-default-544f5768f-47dj7\" (UID: \"8762bab6-0f89-4b6b-a87a-dc1555608862\") " pod="openshift-ingress/e2e-worker-w6-ebb7fd-openshift-default-544f5768f-47dj7" Nov 04 02:54:50.933747 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:54:50.933638 4405 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8762bab6-0f89-4b6b-a87a-dc1555608862-credential-socket\") pod \"e2e-worker-w6-ebb7fd-openshift-default-544f5768f-47dj7\" (UID: \"8762bab6-0f89-4b6b-a87a-dc1555608862\") " pod="openshift-ingress/e2e-worker-w6-ebb7fd-openshift-default-544f5768f-47dj7" Nov 04 02:54:50.933747 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:54:50.933705 4405 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8762bab6-0f89-4b6b-a87a-dc1555608862-istio-data\") pod \"e2e-worker-w6-ebb7fd-openshift-default-544f5768f-47dj7\" (UID: \"8762bab6-0f89-4b6b-a87a-dc1555608862\") " pod="openshift-ingress/e2e-worker-w6-ebb7fd-openshift-default-544f5768f-47dj7" Nov 04 02:54:50.933861 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:54:50.933801 4405 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8762bab6-0f89-4b6b-a87a-dc1555608862-workload-socket\") pod \"e2e-worker-w6-ebb7fd-openshift-default-544f5768f-47dj7\" (UID: \"8762bab6-0f89-4b6b-a87a-dc1555608862\") " pod="openshift-ingress/e2e-worker-w6-ebb7fd-openshift-default-544f5768f-47dj7" Nov 04 02:54:50.934331 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:54:50.934310 4405 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-ebb7fd-openshift-default-544f5768f-47dj7"] Nov 04 02:54:51.035339 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:54:51.035307 4405 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8762bab6-0f89-4b6b-a87a-dc1555608862-istio-envoy\") pod \"e2e-worker-w6-ebb7fd-openshift-default-544f5768f-47dj7\" (UID: \"8762bab6-0f89-4b6b-a87a-dc1555608862\") " pod="openshift-ingress/e2e-worker-w6-ebb7fd-openshift-default-544f5768f-47dj7" Nov 04 02:54:51.035522 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:54:51.035405 4405 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8762bab6-0f89-4b6b-a87a-dc1555608862-workload-certs\") pod \"e2e-worker-w6-ebb7fd-openshift-default-544f5768f-47dj7\" (UID: \"8762bab6-0f89-4b6b-a87a-dc1555608862\") " pod="openshift-ingress/e2e-worker-w6-ebb7fd-openshift-default-544f5768f-47dj7" Nov 04 02:54:51.035522 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:54:51.035448 4405 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8762bab6-0f89-4b6b-a87a-dc1555608862-istio-token\") pod \"e2e-worker-w6-ebb7fd-openshift-default-544f5768f-47dj7\" (UID: \"8762bab6-0f89-4b6b-a87a-dc1555608862\") " pod="openshift-ingress/e2e-worker-w6-ebb7fd-openshift-default-544f5768f-47dj7" Nov 04 02:54:51.035522 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:54:51.035497 4405 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8762bab6-0f89-4b6b-a87a-dc1555608862-istiod-ca-cert\") pod \"e2e-worker-w6-ebb7fd-openshift-default-544f5768f-47dj7\" (UID: \"8762bab6-0f89-4b6b-a87a-dc1555608862\") " pod="openshift-ingress/e2e-worker-w6-ebb7fd-openshift-default-544f5768f-47dj7" Nov 04 02:54:51.035522 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:54:51.035517 4405 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8762bab6-0f89-4b6b-a87a-dc1555608862-credential-socket\") pod \"e2e-worker-w6-ebb7fd-openshift-default-544f5768f-47dj7\" (UID: \"8762bab6-0f89-4b6b-a87a-dc1555608862\") " pod="openshift-ingress/e2e-worker-w6-ebb7fd-openshift-default-544f5768f-47dj7" Nov 04 02:54:51.035727 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:54:51.035550 4405 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8762bab6-0f89-4b6b-a87a-dc1555608862-istio-data\") pod \"e2e-worker-w6-ebb7fd-openshift-default-544f5768f-47dj7\" (UID: \"8762bab6-0f89-4b6b-a87a-dc1555608862\") " pod="openshift-ingress/e2e-worker-w6-ebb7fd-openshift-default-544f5768f-47dj7" Nov 04 02:54:51.035727 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:54:51.035703 4405 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8762bab6-0f89-4b6b-a87a-dc1555608862-workload-socket\") pod \"e2e-worker-w6-ebb7fd-openshift-default-544f5768f-47dj7\" (UID: \"8762bab6-0f89-4b6b-a87a-dc1555608862\") " pod="openshift-ingress/e2e-worker-w6-ebb7fd-openshift-default-544f5768f-47dj7" Nov 04 02:54:51.035843 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:54:51.035745 4405 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8762bab6-0f89-4b6b-a87a-dc1555608862-istio-podinfo\") pod \"e2e-worker-w6-ebb7fd-openshift-default-544f5768f-47dj7\" (UID: \"8762bab6-0f89-4b6b-a87a-dc1555608862\") " pod="openshift-ingress/e2e-worker-w6-ebb7fd-openshift-default-544f5768f-47dj7" Nov 04 02:54:51.035843 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:54:51.035783 4405 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rfsbz\" (UniqueName: \"kubernetes.io/projected/8762bab6-0f89-4b6b-a87a-dc1555608862-kube-api-access-rfsbz\") pod \"e2e-worker-w6-ebb7fd-openshift-default-544f5768f-47dj7\" (UID: \"8762bab6-0f89-4b6b-a87a-dc1555608862\") " pod="openshift-ingress/e2e-worker-w6-ebb7fd-openshift-default-544f5768f-47dj7" Nov 04 02:54:51.035947 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:54:51.035918 4405 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8762bab6-0f89-4b6b-a87a-dc1555608862-credential-socket\") pod \"e2e-worker-w6-ebb7fd-openshift-default-544f5768f-47dj7\" (UID: \"8762bab6-0f89-4b6b-a87a-dc1555608862\") " pod="openshift-ingress/e2e-worker-w6-ebb7fd-openshift-default-544f5768f-47dj7" Nov 04 02:54:51.036027 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:54:51.036003 4405 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8762bab6-0f89-4b6b-a87a-dc1555608862-istio-data\") pod \"e2e-worker-w6-ebb7fd-openshift-default-544f5768f-47dj7\" (UID: \"8762bab6-0f89-4b6b-a87a-dc1555608862\") " pod="openshift-ingress/e2e-worker-w6-ebb7fd-openshift-default-544f5768f-47dj7" Nov 04 02:54:51.036136 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:54:51.036120 4405 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8762bab6-0f89-4b6b-a87a-dc1555608862-workload-certs\") pod \"e2e-worker-w6-ebb7fd-openshift-default-544f5768f-47dj7\" (UID: \"8762bab6-0f89-4b6b-a87a-dc1555608862\") " pod="openshift-ingress/e2e-worker-w6-ebb7fd-openshift-default-544f5768f-47dj7" Nov 04 02:54:51.036174 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:54:51.036109 4405 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8762bab6-0f89-4b6b-a87a-dc1555608862-workload-socket\") pod \"e2e-worker-w6-ebb7fd-openshift-default-544f5768f-47dj7\" (UID: \"8762bab6-0f89-4b6b-a87a-dc1555608862\") " pod="openshift-ingress/e2e-worker-w6-ebb7fd-openshift-default-544f5768f-47dj7" Nov 04 02:54:51.036371 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:54:51.036345 4405 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8762bab6-0f89-4b6b-a87a-dc1555608862-istiod-ca-cert\") pod \"e2e-worker-w6-ebb7fd-openshift-default-544f5768f-47dj7\" (UID: \"8762bab6-0f89-4b6b-a87a-dc1555608862\") " pod="openshift-ingress/e2e-worker-w6-ebb7fd-openshift-default-544f5768f-47dj7" Nov 04 02:54:51.038367 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:54:51.038345 4405 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8762bab6-0f89-4b6b-a87a-dc1555608862-istio-podinfo\") pod \"e2e-worker-w6-ebb7fd-openshift-default-544f5768f-47dj7\" (UID: \"8762bab6-0f89-4b6b-a87a-dc1555608862\") " pod="openshift-ingress/e2e-worker-w6-ebb7fd-openshift-default-544f5768f-47dj7" Nov 04 02:54:51.038658 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:54:51.038639 4405 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8762bab6-0f89-4b6b-a87a-dc1555608862-istio-envoy\") pod \"e2e-worker-w6-ebb7fd-openshift-default-544f5768f-47dj7\" (UID: \"8762bab6-0f89-4b6b-a87a-dc1555608862\") " pod="openshift-ingress/e2e-worker-w6-ebb7fd-openshift-default-544f5768f-47dj7" Nov 04 02:54:51.096345 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:54:51.096309 4405 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8762bab6-0f89-4b6b-a87a-dc1555608862-istio-token\") pod \"e2e-worker-w6-ebb7fd-openshift-default-544f5768f-47dj7\" (UID: \"8762bab6-0f89-4b6b-a87a-dc1555608862\") " pod="openshift-ingress/e2e-worker-w6-ebb7fd-openshift-default-544f5768f-47dj7" Nov 04 02:54:51.100193 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:54:51.100168 4405 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rfsbz\" (UniqueName: \"kubernetes.io/projected/8762bab6-0f89-4b6b-a87a-dc1555608862-kube-api-access-rfsbz\") pod \"e2e-worker-w6-ebb7fd-openshift-default-544f5768f-47dj7\" (UID: \"8762bab6-0f89-4b6b-a87a-dc1555608862\") " pod="openshift-ingress/e2e-worker-w6-ebb7fd-openshift-default-544f5768f-47dj7" Nov 04 02:54:51.179058 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:54:51.178979 4405 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-ebb7fd-openshift-default-544f5768f-47dj7" Nov 04 02:54:51.607783 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:54:51.607750 4405 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-ebb7fd-openshift-default-544f5768f-47dj7"] Nov 04 02:54:51.645237 ip-10-0-134-125 kubenswrapper[4405]: W1104 02:54:51.645194 4405 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8762bab6_0f89_4b6b_a87a_dc1555608862.slice/crio-7cf83655ba6da668b44f80c5d84840fac97b28ad4e7f11f43159ec3ddd6802a4 WatchSource:0}: Error finding container 7cf83655ba6da668b44f80c5d84840fac97b28ad4e7f11f43159ec3ddd6802a4: Status 404 returned error can't find the container with id 7cf83655ba6da668b44f80c5d84840fac97b28ad4e7f11f43159ec3ddd6802a4 Nov 04 02:54:51.648300 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:54:51.648262 4405 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 04 02:54:51.648422 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:54:51.648334 4405 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 04 02:54:51.648422 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:54:51.648371 4405 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 04 02:54:52.364713 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:54:52.364649 4405 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-ebb7fd-openshift-default-544f5768f-47dj7" event={"ID":"8762bab6-0f89-4b6b-a87a-dc1555608862","Type":"ContainerStarted","Data":"fc4bfde43f7d0c1abd2d9b987a382e6165987a5b3dd18701500856f6b51184aa"} Nov 04 02:54:52.365208 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:54:52.364740 4405 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-ebb7fd-openshift-default-544f5768f-47dj7" event={"ID":"8762bab6-0f89-4b6b-a87a-dc1555608862","Type":"ContainerStarted","Data":"7cf83655ba6da668b44f80c5d84840fac97b28ad4e7f11f43159ec3ddd6802a4"} Nov 04 02:54:52.585297 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:54:52.585239 4405 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w6-ebb7fd-openshift-default-544f5768f-47dj7" podStartSLOduration=2.585224575 podStartE2EDuration="2.585224575s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 02:54:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 02:54:52.533506257 +0000 UTC m=+1338.964206378" watchObservedRunningTime="2026-11-04 02:54:52.585224575 +0000 UTC m=+1339.015924694" Nov 04 02:54:53.179498 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:54:53.179455 4405 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w6-ebb7fd-openshift-default-544f5768f-47dj7" Nov 04 02:54:53.180502 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:54:53.180477 4405 patch_prober.go:29] interesting pod/e2e-worker-w6-ebb7fd-openshift-default-544f5768f-47dj7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused" start-of-body= Nov 04 02:54:53.180569 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:54:53.180540 4405 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-ebb7fd-openshift-default-544f5768f-47dj7" podUID="8762bab6-0f89-4b6b-a87a-dc1555608862" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused" Nov 04 02:54:54.179458 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:54:54.179421 4405 patch_prober.go:29] interesting pod/e2e-worker-w6-ebb7fd-openshift-default-544f5768f-47dj7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused" start-of-body= Nov 04 02:54:54.179614 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:54:54.179472 4405 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-ebb7fd-openshift-default-544f5768f-47dj7" podUID="8762bab6-0f89-4b6b-a87a-dc1555608862" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused" Nov 04 02:54:55.179750 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:54:55.179718 4405 patch_prober.go:29] interesting pod/e2e-worker-w6-ebb7fd-openshift-default-544f5768f-47dj7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused" start-of-body= Nov 04 02:54:55.180155 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:54:55.179773 4405 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-ebb7fd-openshift-default-544f5768f-47dj7" podUID="8762bab6-0f89-4b6b-a87a-dc1555608862" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused" Nov 04 02:54:56.180419 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:54:56.180379 4405 patch_prober.go:29] interesting pod/e2e-worker-w6-ebb7fd-openshift-default-544f5768f-47dj7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused" start-of-body= Nov 04 02:54:56.180826 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:54:56.180451 4405 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-ebb7fd-openshift-default-544f5768f-47dj7" podUID="8762bab6-0f89-4b6b-a87a-dc1555608862" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused" Nov 04 02:54:57.179969 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:54:57.179930 4405 patch_prober.go:29] interesting pod/e2e-worker-w6-ebb7fd-openshift-default-544f5768f-47dj7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused" start-of-body= Nov 04 02:54:57.180128 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:54:57.179991 4405 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-ebb7fd-openshift-default-544f5768f-47dj7" podUID="8762bab6-0f89-4b6b-a87a-dc1555608862" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused" Nov 04 02:54:58.180364 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:54:58.180326 4405 patch_prober.go:29] interesting pod/e2e-worker-w6-ebb7fd-openshift-default-544f5768f-47dj7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused" start-of-body= Nov 04 02:54:58.180816 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:54:58.180385 4405 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-ebb7fd-openshift-default-544f5768f-47dj7" podUID="8762bab6-0f89-4b6b-a87a-dc1555608862" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused" Nov 04 02:54:59.263614 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:54:59.263580 4405 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w6-ebb7fd-openshift-default-544f5768f-47dj7" Nov 04 02:54:59.264231 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:54:59.264103 4405 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w6-ebb7fd-openshift-default-544f5768f-47dj7" Nov 04 02:54:59.265081 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:54:59.265063 4405 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w6-ebb7fd-openshift-default-544f5768f-47dj7" Nov 04 02:55:01.986322 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:55:01.986288 4405 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-bl5mg"] Nov 04 02:55:04.431696 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:55:04.430811 4405 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-ebb7fd/distinct-2-w6-ebb7fd-kserve-6f68c5db86-zrpjs" podUID="b7b786e0-9c94-4126-9949-cd354ebbfb2a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 02:55:04.431696 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:55:04.431257 4405 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-a0bcbc/premium-w2-a0bcbc-kserve-78dfd6d78c-lwth2" podUID="33b620c1-389c-48b4-b277-90252018c4b6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/ready\": context deadline exceeded" Nov 04 02:55:04.447063 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:55:04.439231 4405 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-e516ff/distinct-w1-e516ff-kserve-f4474cb9c-glpzw" podUID="8842ad09-22ff-4492-ba5d-d4cacf4eac75" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 02:55:04.464697 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:55:04.463774 4405 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-fe4e0c/distinct-2-w0-fe4e0c-kserve-579bf4cc77-8pbq8" podUID="2b2ab722-d1bb-4376-bede-f5f4ee4562aa" containerName="main" probeResult="failure" output="Get \"https://10.134.0.68:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 02:55:04.505707 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:55:04.505155 4405 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-cdcf67/distinct-w3-cdcf67-kserve-667cc7f7d7-5m2lq" podUID="52ed22cc-96e4-4ef0-b34d-ed58c89a9183" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/ready\": context deadline exceeded" Nov 04 02:55:04.555707 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:55:04.551272 4405 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-ljhhw" podUID="f66b1861-d87b-4d57-ab27-f1a24544a534" containerName="main" probeResult="failure" output="Get \"https://10.134.0.26:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 02:55:04.784867 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:55:04.784759 4405 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-a0bcbc/distinct-w2-a0bcbc-kserve-596f94c7cd-szl4g" podUID="bac32934-8a57-4919-b435-13f367cbea3e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": context deadline exceeded" Nov 04 02:55:04.918692 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:55:04.917770 4405 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-ebb7fd/distinct-w6-ebb7fd-kserve-55c47d8865-7z9pz" podUID="ccb8e181-5610-46da-bb8b-e0b07fa02ff1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 02:55:04.939694 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:55:04.938754 4405 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-fe4e0c/distinct-2-w0-fe4e0c-kserve-579bf4cc77-8pbq8" podUID="2b2ab722-d1bb-4376-bede-f5f4ee4562aa" containerName="main" probeResult="failure" output="Get \"https://10.134.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 02:55:04.964693 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:55:04.963774 4405 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-203c35/simulator-w4-203c35-kserve-f9c4bd695-999ns" podUID="eb0d8bdd-6808-4fd0-b9b5-567dff3e7ca5" containerName="main" probeResult="failure" output="Get \"https://10.134.0.46:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 02:55:05.466973 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:55:05.432759 4405 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-a0bcbc/premium-w2-a0bcbc-kserve-78dfd6d78c-lwth2" podUID="33b620c1-389c-48b4-b277-90252018c4b6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 02:55:05.466973 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:55:05.433226 4405 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-ebb7fd/distinct-2-w6-ebb7fd-kserve-6f68c5db86-zrpjs" podUID="b7b786e0-9c94-4126-9949-cd354ebbfb2a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.65:8000/ready\": context deadline exceeded" Nov 04 02:55:05.466973 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:55:05.433951 4405 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-a0bcbc/premium-w2-a0bcbc-kserve-78dfd6d78c-lwth2" Nov 04 02:55:05.466973 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:55:05.434008 4405 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-ebb7fd/distinct-2-w6-ebb7fd-kserve-6f68c5db86-zrpjs" Nov 04 02:55:05.466973 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:55:05.449989 4405 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-e516ff/distinct-w1-e516ff-kserve-f4474cb9c-glpzw" podUID="8842ad09-22ff-4492-ba5d-d4cacf4eac75" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 02:55:05.466973 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:55:05.450431 4405 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-e516ff/distinct-w1-e516ff-kserve-f4474cb9c-glpzw" Nov 04 02:55:05.535694 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:55:05.508826 4405 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-cdcf67/distinct-w3-cdcf67-kserve-667cc7f7d7-5m2lq" podUID="52ed22cc-96e4-4ef0-b34d-ed58c89a9183" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/ready\": context deadline exceeded" Nov 04 02:55:05.535694 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:55:05.509332 4405 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-cdcf67/distinct-w3-cdcf67-kserve-667cc7f7d7-5m2lq" Nov 04 02:55:05.565710 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:55:05.565157 4405 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-ljhhw" podUID="f66b1861-d87b-4d57-ab27-f1a24544a534" containerName="main" probeResult="failure" output="Get \"https://10.134.0.26:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 02:55:05.565917 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:55:05.565779 4405 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-ljhhw" Nov 04 02:55:05.979699 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:55:05.979194 4405 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-fe4e0c/distinct-2-w0-fe4e0c-kserve-579bf4cc77-8pbq8" podUID="2b2ab722-d1bb-4376-bede-f5f4ee4562aa" containerName="main" probeResult="failure" output="Get \"https://10.134.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 02:55:06.007783 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:55:05.985374 4405 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-ebb7fd/distinct-w6-ebb7fd-kserve-55c47d8865-7z9pz" podUID="ccb8e181-5610-46da-bb8b-e0b07fa02ff1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": context deadline exceeded" Nov 04 02:55:06.050694 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:55:06.027138 4405 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-fe4e0c/distinct-2-w0-fe4e0c-kserve-579bf4cc77-8pbq8" Nov 04 02:55:06.051416 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:55:06.051387 4405 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-ebb7fd/distinct-w6-ebb7fd-kserve-55c47d8865-7z9pz" Nov 04 02:55:06.169697 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:55:06.156923 4405 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-203c35/simulator-w4-203c35-kserve-f9c4bd695-999ns" podUID="eb0d8bdd-6808-4fd0-b9b5-567dff3e7ca5" containerName="main" probeResult="failure" output="Get \"https://10.134.0.46:8000/health\": context deadline exceeded" Nov 04 02:55:06.529716 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:55:06.525931 4405 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-a0bcbc/premium-w2-a0bcbc-kserve-78dfd6d78c-lwth2" podUID="33b620c1-389c-48b4-b277-90252018c4b6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 02:55:06.532002 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:55:06.530633 4405 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-ebb7fd/distinct-2-w6-ebb7fd-kserve-6f68c5db86-zrpjs" podUID="b7b786e0-9c94-4126-9949-cd354ebbfb2a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 02:55:06.656603 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:55:06.656550 4405 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-cdcf67/distinct-w3-cdcf67-kserve-667cc7f7d7-5m2lq" podUID="52ed22cc-96e4-4ef0-b34d-ed58c89a9183" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 02:55:06.741721 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:55:06.703161 4405 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-203c35/simulator-w4-203c35-kserve-f9c4bd695-999ns" podUID="eb0d8bdd-6808-4fd0-b9b5-567dff3e7ca5" containerName="main" probeResult="failure" output="Get \"https://10.134.0.46:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 04 02:55:06.937435 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:55:06.935855 4405 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-e516ff/distinct-w1-e516ff-kserve-f4474cb9c-glpzw" podUID="8842ad09-22ff-4492-ba5d-d4cacf4eac75" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/ready\": context deadline exceeded" Nov 04 02:55:06.993005 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:55:06.988344 4405 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-ebb7fd/distinct-w6-ebb7fd-kserve-55c47d8865-7z9pz" podUID="ccb8e181-5610-46da-bb8b-e0b07fa02ff1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 02:55:06.993005 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:55:06.992986 4405 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-fe4e0c/distinct-2-w0-fe4e0c-kserve-579bf4cc77-8pbq8" podUID="2b2ab722-d1bb-4376-bede-f5f4ee4562aa" containerName="main" probeResult="failure" output="Get \"https://10.134.0.68:8000/ready\": context deadline exceeded" Nov 04 02:55:07.026748 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:55:07.016828 4405 patch_prober.go:29] interesting pod/e2e-worker-w5-3b7476-openshift-default-59975457bf-vxwkm container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.74:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 04 02:55:07.026748 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:55:07.016934 4405 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w5-3b7476-openshift-default-59975457bf-vxwkm" podUID="f2c5af93-085e-462b-8377-a6e8139e66c8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.74:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 04 02:55:07.070053 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:55:07.070002 4405 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-a0bcbc/distinct-w2-a0bcbc-kserve-596f94c7cd-szl4g" podUID="bac32934-8a57-4919-b435-13f367cbea3e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 02:55:07.139694 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:55:07.137941 4405 patch_prober.go:29] interesting pod/e2e-worker-w3-cdcf67-openshift-default-6ff8f5cbc7-rwgvq container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.34:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 04 02:55:07.139694 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:55:07.138010 4405 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-cdcf67-openshift-default-6ff8f5cbc7-rwgvq" podUID="dad304e5-de0a-44b2-a137-8139a2aa2f52" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.34:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 04 02:55:07.143731 ip-10-0-134-125 kubenswrapper[4405]: E1104 02:55:07.140320 4405 kubelet.go:2711] "Housekeeping took longer than expected" err="housekeeping took too long" expected="1s" actual="1.089s" Nov 04 02:55:07.143731 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:55:07.140416 4405 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-203c35/simulator-w4-203c35-kserve-f9c4bd695-999ns" Nov 04 02:55:07.143731 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:55:07.140469 4405 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-a0bcbc/distinct-w2-a0bcbc-kserve-596f94c7cd-szl4g" Nov 04 02:55:07.220818 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:55:07.213013 4405 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-e516ff/distinct-w1-e516ff-kserve-f4474cb9c-glpzw" podUID="8842ad09-22ff-4492-ba5d-d4cacf4eac75" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/health\": context deadline exceeded" Nov 04 02:55:07.254739 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:55:07.222457 4405 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-ljhhw" podUID="f66b1861-d87b-4d57-ab27-f1a24544a534" containerName="main" probeResult="failure" output="Get \"https://10.134.0.26:8000/ready\": context deadline exceeded" Nov 04 02:55:07.308402 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:55:07.308352 4405 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-gnjj2" podUID="22d477a6-9ab8-4330-ae26-813caff99cb3" containerName="main" probeResult="failure" output="Get \"https://10.134.0.28:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 02:55:07.540153 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:55:07.540056 4405 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-ebb7fd/distinct-2-w6-ebb7fd-kserve-6f68c5db86-zrpjs" podUID="b7b786e0-9c94-4126-9949-cd354ebbfb2a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.65:8000/ready\": context deadline exceeded" Nov 04 02:55:07.541107 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:55:07.541072 4405 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-a0bcbc/premium-w2-a0bcbc-kserve-78dfd6d78c-lwth2" podUID="33b620c1-389c-48b4-b277-90252018c4b6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/ready\": context deadline exceeded" Nov 04 02:55:07.799757 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:55:07.791081 4405 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-cdcf67/distinct-w3-cdcf67-kserve-667cc7f7d7-5m2lq" podUID="52ed22cc-96e4-4ef0-b34d-ed58c89a9183" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/ready\": context deadline exceeded" Nov 04 02:55:07.810614 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:55:07.800010 4405 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-203c35/simulator-w4-203c35-kserve-f9c4bd695-999ns" podUID="eb0d8bdd-6808-4fd0-b9b5-567dff3e7ca5" containerName="main" probeResult="failure" output="Get \"https://10.134.0.46:8000/ready\": context deadline exceeded" Nov 04 02:55:07.937340 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:55:07.937293 4405 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-e516ff/distinct-w1-e516ff-kserve-f4474cb9c-glpzw" podUID="8842ad09-22ff-4492-ba5d-d4cacf4eac75" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 02:55:07.997820 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:55:07.997769 4405 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-ebb7fd/distinct-w6-ebb7fd-kserve-55c47d8865-7z9pz" podUID="ccb8e181-5610-46da-bb8b-e0b07fa02ff1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 02:55:08.271695 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:55:08.262127 4405 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-fe4e0c/distinct-2-w0-fe4e0c-kserve-579bf4cc77-8pbq8" podUID="2b2ab722-d1bb-4376-bede-f5f4ee4562aa" containerName="main" probeResult="failure" output="Get \"https://10.134.0.68:8000/ready\": context deadline exceeded" Nov 04 02:55:08.301697 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:55:08.294807 4405 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-ljhhw" podUID="f66b1861-d87b-4d57-ab27-f1a24544a534" containerName="main" probeResult="failure" output="Get \"https://10.134.0.26:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 02:55:08.506695 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:55:08.391795 4405 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-a0bcbc/distinct-w2-a0bcbc-kserve-596f94c7cd-szl4g" podUID="bac32934-8a57-4919-b435-13f367cbea3e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 02:55:08.506695 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:55:08.401546 4405 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-a0bcbc/distinct-w2-a0bcbc-kserve-596f94c7cd-szl4g" podUID="bac32934-8a57-4919-b435-13f367cbea3e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": context deadline exceeded" Nov 04 02:55:08.584893 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:55:08.576164 4405 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-ebb7fd/distinct-2-w6-ebb7fd-kserve-6f68c5db86-zrpjs" podUID="b7b786e0-9c94-4126-9949-cd354ebbfb2a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 02:55:08.584893 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:55:08.576615 4405 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-a0bcbc/premium-w2-a0bcbc-kserve-78dfd6d78c-lwth2" podUID="33b620c1-389c-48b4-b277-90252018c4b6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 02:55:08.798696 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:55:08.791348 4405 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-cdcf67/distinct-w3-cdcf67-kserve-667cc7f7d7-5m2lq" podUID="52ed22cc-96e4-4ef0-b34d-ed58c89a9183" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 02:55:08.880919 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:55:08.857924 4405 prober.go:122] "Probe failed" probeType="Liveness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-ljhhw" podUID="f66b1861-d87b-4d57-ab27-f1a24544a534" containerName="main" probeResult="failure" output="Get \"https://10.134.0.26:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 02:55:08.927828 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:55:08.927775 4405 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-203c35/simulator-w4-203c35-kserve-f9c4bd695-999ns" podUID="eb0d8bdd-6808-4fd0-b9b5-567dff3e7ca5" containerName="main" probeResult="failure" output="Get \"https://10.134.0.46:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 02:55:08.937051 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:55:08.937006 4405 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-ebb7fd/distinct-w6-ebb7fd-kserve-55c47d8865-7z9pz" podUID="ccb8e181-5610-46da-bb8b-e0b07fa02ff1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 02:55:08.956827 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:55:08.956769 4405 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-e516ff/distinct-w1-e516ff-kserve-f4474cb9c-glpzw" podUID="8842ad09-22ff-4492-ba5d-d4cacf4eac75" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 02:55:09.011248 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:55:09.011205 4405 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-ebb7fd/distinct-w6-ebb7fd-kserve-55c47d8865-7z9pz" podUID="ccb8e181-5610-46da-bb8b-e0b07fa02ff1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 02:55:09.066693 ip-10-0-134-125 kubenswrapper[4405]: E1104 02:55:09.065889 4405 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5f2vb" containerName="extractor" Nov 04 02:55:09.265128 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:55:09.265026 4405 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-fe4e0c/distinct-2-w0-fe4e0c-kserve-579bf4cc77-8pbq8" podUID="2b2ab722-d1bb-4376-bede-f5f4ee4562aa" containerName="main" probeResult="failure" output="Get \"https://10.134.0.68:8000/ready\": context deadline exceeded" Nov 04 02:55:09.305293 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:55:09.303890 4405 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-ljhhw" podUID="f66b1861-d87b-4d57-ab27-f1a24544a534" containerName="main" probeResult="failure" output="Get \"https://10.134.0.26:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 02:55:09.413298 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:55:09.411786 4405 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-a0bcbc/distinct-w2-a0bcbc-kserve-596f94c7cd-szl4g" podUID="bac32934-8a57-4919-b435-13f367cbea3e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 02:55:09.599695 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:55:09.584779 4405 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-a0bcbc/premium-w2-a0bcbc-kserve-78dfd6d78c-lwth2" podUID="33b620c1-389c-48b4-b277-90252018c4b6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 02:55:09.599695 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:55:09.586313 4405 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-ebb7fd/distinct-2-w6-ebb7fd-kserve-6f68c5db86-zrpjs" podUID="b7b786e0-9c94-4126-9949-cd354ebbfb2a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.65:8000/ready\": context deadline exceeded" Nov 04 02:55:09.750871 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:55:09.750766 4405 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-a0bcbc/premium-w2-a0bcbc-kserve-78dfd6d78c-lwth2" podUID="33b620c1-389c-48b4-b277-90252018c4b6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 02:55:09.834513 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:55:09.832865 4405 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-cdcf67/distinct-w3-cdcf67-kserve-667cc7f7d7-5m2lq" podUID="52ed22cc-96e4-4ef0-b34d-ed58c89a9183" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 02:55:09.902918 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:55:09.902755 4405 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-cdcf67/distinct-w3-cdcf67-kserve-667cc7f7d7-5m2lq" podUID="52ed22cc-96e4-4ef0-b34d-ed58c89a9183" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 02:55:09.933707 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:55:09.932834 4405 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-203c35/simulator-w4-203c35-kserve-f9c4bd695-999ns" podUID="eb0d8bdd-6808-4fd0-b9b5-567dff3e7ca5" containerName="main" probeResult="failure" output="Get \"https://10.134.0.46:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 02:55:09.957820 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:55:09.957767 4405 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-e516ff/distinct-w1-e516ff-kserve-f4474cb9c-glpzw" podUID="8842ad09-22ff-4492-ba5d-d4cacf4eac75" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 02:55:10.020697 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:55:10.019831 4405 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-ebb7fd/distinct-w6-ebb7fd-kserve-55c47d8865-7z9pz" podUID="ccb8e181-5610-46da-bb8b-e0b07fa02ff1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 02:55:10.191981 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:55:10.191785 4405 patch_prober.go:29] interesting pod/e2e-worker-w2-a0bcbc-openshift-default-65dc9d9b6c-m97sr container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.49:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 04 02:55:10.191981 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:55:10.191854 4405 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-a0bcbc-openshift-default-65dc9d9b6c-m97sr" podUID="9a748832-c6ef-484f-8635-8c9148a38149" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.49:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 04 02:55:10.345698 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:55:10.336753 4405 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-fe4e0c/distinct-2-w0-fe4e0c-kserve-579bf4cc77-8pbq8" podUID="2b2ab722-d1bb-4376-bede-f5f4ee4562aa" containerName="main" probeResult="failure" output="Get \"https://10.134.0.68:8000/ready\": context deadline exceeded" Nov 04 02:55:10.364743 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:55:10.347775 4405 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-ljhhw" podUID="f66b1861-d87b-4d57-ab27-f1a24544a534" containerName="main" probeResult="failure" output="Get \"https://10.134.0.26:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 02:55:10.412121 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:55:10.412075 4405 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-a0bcbc/distinct-w2-a0bcbc-kserve-596f94c7cd-szl4g" podUID="bac32934-8a57-4919-b435-13f367cbea3e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 02:55:10.599573 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:55:10.599522 4405 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-a0bcbc/premium-w2-a0bcbc-kserve-78dfd6d78c-lwth2" podUID="33b620c1-389c-48b4-b277-90252018c4b6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/ready\": context deadline exceeded" Nov 04 02:55:10.747181 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:55:10.615506 4405 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-ebb7fd/distinct-2-w6-ebb7fd-kserve-6f68c5db86-zrpjs" podUID="b7b786e0-9c94-4126-9949-cd354ebbfb2a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 02:55:10.843945 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:55:10.843895 4405 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-cdcf67/distinct-w3-cdcf67-kserve-667cc7f7d7-5m2lq" podUID="52ed22cc-96e4-4ef0-b34d-ed58c89a9183" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 02:55:10.937920 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:55:10.937807 4405 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-203c35/simulator-w4-203c35-kserve-f9c4bd695-999ns" podUID="eb0d8bdd-6808-4fd0-b9b5-567dff3e7ca5" containerName="main" probeResult="failure" output="Get \"https://10.134.0.46:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 02:55:10.969383 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:55:10.969343 4405 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-e516ff/distinct-w1-e516ff-kserve-f4474cb9c-glpzw" podUID="8842ad09-22ff-4492-ba5d-d4cacf4eac75" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/ready\": context deadline exceeded" Nov 04 02:55:11.054697 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:55:11.039969 4405 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-ebb7fd/distinct-w6-ebb7fd-kserve-55c47d8865-7z9pz" podUID="ccb8e181-5610-46da-bb8b-e0b07fa02ff1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": context deadline exceeded" Nov 04 02:55:11.376694 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:55:11.372630 4405 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-fe4e0c/distinct-2-w0-fe4e0c-kserve-579bf4cc77-8pbq8" podUID="2b2ab722-d1bb-4376-bede-f5f4ee4562aa" containerName="main" probeResult="failure" output="Get \"https://10.134.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 02:55:11.385695 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:55:11.373937 4405 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-ljhhw" podUID="f66b1861-d87b-4d57-ab27-f1a24544a534" containerName="main" probeResult="failure" output="Get \"https://10.134.0.26:8000/ready\": context deadline exceeded" Nov 04 02:55:11.417267 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:55:11.416805 4405 patch_prober.go:29] interesting pod/e2e-worker-w2-a0bcbc-openshift-default-65dc9d9b6c-j4n6p container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.54:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 04 02:55:11.417267 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:55:11.416875 4405 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-a0bcbc-openshift-default-65dc9d9b6c-j4n6p" podUID="55473cc8-d292-4f5f-bb17-3f08c2e19e8c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.54:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 04 02:55:11.417267 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:55:11.416960 4405 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-a0bcbc/distinct-w2-a0bcbc-kserve-596f94c7cd-szl4g" podUID="bac32934-8a57-4919-b435-13f367cbea3e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 02:55:11.618813 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:55:11.618764 4405 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-ebb7fd/distinct-2-w6-ebb7fd-kserve-6f68c5db86-zrpjs" podUID="b7b786e0-9c94-4126-9949-cd354ebbfb2a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.65:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 02:55:11.759991 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:55:11.759884 4405 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-ebb7fd/distinct-2-w6-ebb7fd-kserve-6f68c5db86-zrpjs" podUID="b7b786e0-9c94-4126-9949-cd354ebbfb2a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 02:55:11.778394 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:55:11.778350 4405 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-a0bcbc/premium-w2-a0bcbc-kserve-78dfd6d78c-lwth2" podUID="33b620c1-389c-48b4-b277-90252018c4b6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/ready\": context deadline exceeded" Nov 04 02:55:11.861699 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:55:11.846749 4405 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-cdcf67/distinct-w3-cdcf67-kserve-667cc7f7d7-5m2lq" podUID="52ed22cc-96e4-4ef0-b34d-ed58c89a9183" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 02:55:11.865412 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:55:11.858097 4405 patch_prober.go:29] interesting pod/e2e-worker-w3-cdcf67-openshift-default-6ff8f5cbc7-975ss container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.56:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 04 02:55:11.884570 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:55:11.884524 4405 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-cdcf67-openshift-default-6ff8f5cbc7-975ss" podUID="e5ccd15d-11fd-4f67-9bd5-7a4cfa35d602" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.56:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 04 02:55:11.958026 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:55:11.957651 4405 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-203c35/simulator-w4-203c35-kserve-f9c4bd695-999ns" podUID="eb0d8bdd-6808-4fd0-b9b5-567dff3e7ca5" containerName="main" probeResult="failure" output="Get \"https://10.134.0.46:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 02:55:12.000931 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:55:12.000880 4405 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-e516ff/distinct-w1-e516ff-kserve-f4474cb9c-glpzw" podUID="8842ad09-22ff-4492-ba5d-d4cacf4eac75" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 02:55:12.063763 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:55:12.044415 4405 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-ebb7fd/distinct-w6-ebb7fd-kserve-55c47d8865-7z9pz" podUID="ccb8e181-5610-46da-bb8b-e0b07fa02ff1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 02:55:12.090808 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:55:12.090770 4405 patch_prober.go:29] interesting pod/e2e-worker-w4-203c35-openshift-default-56497cdcc-2b66r container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.51:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 04 02:55:12.098697 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:55:12.094741 4405 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-203c35-openshift-default-56497cdcc-2b66r" podUID="147e25a7-dda1-4170-9453-44896e44e2bc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.51:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 04 02:55:12.115100 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:55:12.107811 4405 patch_prober.go:29] interesting pod/e2e-worker-w1-e516ff-openshift-default-6f4b957485-7pg2l container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.55:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 04 02:55:12.163467 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:55:12.115351 4405 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-e516ff-openshift-default-6f4b957485-7pg2l" podUID="8fc7b14e-09f9-4630-bfc6-b98eab67753f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.55:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 04 02:55:12.381935 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:55:12.381889 4405 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-fe4e0c/distinct-2-w0-fe4e0c-kserve-579bf4cc77-8pbq8" podUID="2b2ab722-d1bb-4376-bede-f5f4ee4562aa" containerName="main" probeResult="failure" output="Get \"https://10.134.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 02:55:12.386256 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:55:12.386211 4405 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-ljhhw" podUID="f66b1861-d87b-4d57-ab27-f1a24544a534" containerName="main" probeResult="failure" output="Get \"https://10.134.0.26:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 02:55:12.475689 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:55:12.438747 4405 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-a0bcbc/distinct-w2-a0bcbc-kserve-596f94c7cd-szl4g" podUID="bac32934-8a57-4919-b435-13f367cbea3e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 02:55:12.801772 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:55:12.797190 4405 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-ebb7fd/distinct-2-w6-ebb7fd-kserve-6f68c5db86-zrpjs" podUID="b7b786e0-9c94-4126-9949-cd354ebbfb2a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.65:8000/ready\": context deadline exceeded" Nov 04 02:55:12.859707 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:55:12.859251 4405 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-a0bcbc/premium-w2-a0bcbc-kserve-78dfd6d78c-lwth2" podUID="33b620c1-389c-48b4-b277-90252018c4b6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 02:55:12.869579 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:55:12.868315 4405 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-cdcf67/distinct-w3-cdcf67-kserve-667cc7f7d7-5m2lq" podUID="52ed22cc-96e4-4ef0-b34d-ed58c89a9183" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 02:55:12.903278 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:55:12.903242 4405 patch_prober.go:29] interesting pod/e2e-worker-w1-e516ff-openshift-default-6f4b957485-xg2tv container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.60:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 04 02:55:12.903579 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:55:12.903550 4405 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-e516ff-openshift-default-6f4b957485-xg2tv" podUID="a4bcafa2-ec97-486c-819f-7be10e3110a9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.60:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 04 02:55:12.958394 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:55:12.958346 4405 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-203c35/simulator-w4-203c35-kserve-f9c4bd695-999ns" podUID="eb0d8bdd-6808-4fd0-b9b5-567dff3e7ca5" containerName="main" probeResult="failure" output="Get \"https://10.134.0.46:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 02:55:13.174696 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:55:13.166515 4405 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-ebb7fd/distinct-w6-ebb7fd-kserve-55c47d8865-7z9pz" podUID="ccb8e181-5610-46da-bb8b-e0b07fa02ff1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 02:55:13.189221 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:55:13.188719 4405 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-e516ff/distinct-w1-e516ff-kserve-f4474cb9c-glpzw" podUID="8842ad09-22ff-4492-ba5d-d4cacf4eac75" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/ready\": context deadline exceeded" Nov 04 02:55:13.387694 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:55:13.383820 4405 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-fe4e0c/distinct-2-w0-fe4e0c-kserve-579bf4cc77-8pbq8" podUID="2b2ab722-d1bb-4376-bede-f5f4ee4562aa" containerName="main" probeResult="failure" output="Get \"https://10.134.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 02:55:13.414619 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:55:13.390017 4405 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-ljhhw" podUID="f66b1861-d87b-4d57-ab27-f1a24544a534" containerName="main" probeResult="failure" output="Get \"https://10.134.0.26:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 02:55:13.425246 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:55:13.403334 4405 patch_prober.go:29] interesting pod/e2e-worker-w4-203c35-openshift-default-56497cdcc-z59vn container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.61:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 04 02:55:13.450694 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:55:13.425482 4405 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-203c35-openshift-default-56497cdcc-z59vn" podUID="5ebc4fc9-8882-43c6-8713-1ee995a8ee00" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.61:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 04 02:55:13.451060 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:55:13.451014 4405 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-a0bcbc/distinct-w2-a0bcbc-kserve-596f94c7cd-szl4g" podUID="bac32934-8a57-4919-b435-13f367cbea3e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": context deadline exceeded" Nov 04 02:55:13.813258 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:55:13.806254 4405 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-ebb7fd/distinct-2-w6-ebb7fd-kserve-6f68c5db86-zrpjs" podUID="b7b786e0-9c94-4126-9949-cd354ebbfb2a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 02:55:13.884011 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:55:13.882146 4405 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-cdcf67/distinct-w3-cdcf67-kserve-667cc7f7d7-5m2lq" podUID="52ed22cc-96e4-4ef0-b34d-ed58c89a9183" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/ready\": context deadline exceeded" Nov 04 02:55:13.884011 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:55:13.882486 4405 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-a0bcbc/premium-w2-a0bcbc-kserve-78dfd6d78c-lwth2" podUID="33b620c1-389c-48b4-b277-90252018c4b6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/ready\": context deadline exceeded" Nov 04 02:55:13.984856 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:55:13.984822 4405 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-ebb7fd/distinct-2-w6-ebb7fd-kserve-6f68c5db86-zrpjs" Nov 04 02:55:13.985051 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:55:13.984882 4405 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-203c35/simulator-w4-203c35-kserve-f9c4bd695-999ns" Nov 04 02:55:13.985051 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:55:13.984913 4405 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-cdcf67/distinct-w3-cdcf67-kserve-667cc7f7d7-5m2lq" Nov 04 02:55:13.998283 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:55:13.998255 4405 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-a0bcbc/premium-w2-a0bcbc-kserve-78dfd6d78c-lwth2" Nov 04 02:55:14.040694 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:55:14.025908 4405 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-a0bcbc/distinct-w2-a0bcbc-kserve-596f94c7cd-szl4g" Nov 04 02:55:14.040694 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:55:14.032426 4405 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-ljhhw" Nov 04 02:55:14.053865 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:55:14.052571 4405 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-ebb7fd/distinct-w6-ebb7fd-kserve-55c47d8865-7z9pz" Nov 04 02:55:14.072243 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:55:14.072133 4405 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-e516ff/distinct-w1-e516ff-kserve-f4474cb9c-glpzw" Nov 04 02:55:14.079350 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:55:14.079324 4405 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-fe4e0c/distinct-2-w0-fe4e0c-kserve-579bf4cc77-8pbq8" Nov 04 02:55:36.886762 ip-10-0-134-125 kubenswrapper[4405]: E1104 02:55:36.886623 4405 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5f2vb" containerName="extractor" Nov 04 02:55:37.333613 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:55:37.333579 4405 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-87191e54-openshift-default-7c59fcc9cc-6s8gq"] Nov 04 02:55:37.337853 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:55:37.337828 4405 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-87191e54-openshift-default-7c59fcc9cc-6s8gq" Nov 04 02:55:37.390734 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:55:37.390023 4405 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-87191e54-openshift-default-dockercfg-6kqm4\"" Nov 04 02:55:37.429393 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:55:37.429255 4405 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/65bede99-6a95-4ac2-84a9-2ffdaef6375d-workload-socket\") pod \"e2e-ait-87191e54-openshift-default-7c59fcc9cc-6s8gq\" (UID: \"65bede99-6a95-4ac2-84a9-2ffdaef6375d\") " pod="openshift-ingress/e2e-ait-87191e54-openshift-default-7c59fcc9cc-6s8gq" Nov 04 02:55:37.429393 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:55:37.429301 4405 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/65bede99-6a95-4ac2-84a9-2ffdaef6375d-istio-token\") pod \"e2e-ait-87191e54-openshift-default-7c59fcc9cc-6s8gq\" (UID: \"65bede99-6a95-4ac2-84a9-2ffdaef6375d\") " pod="openshift-ingress/e2e-ait-87191e54-openshift-default-7c59fcc9cc-6s8gq" Nov 04 02:55:37.429393 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:55:37.429367 4405 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/65bede99-6a95-4ac2-84a9-2ffdaef6375d-istiod-ca-cert\") pod \"e2e-ait-87191e54-openshift-default-7c59fcc9cc-6s8gq\" (UID: \"65bede99-6a95-4ac2-84a9-2ffdaef6375d\") " pod="openshift-ingress/e2e-ait-87191e54-openshift-default-7c59fcc9cc-6s8gq" Nov 04 02:55:37.429707 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:55:37.429440 4405 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/65bede99-6a95-4ac2-84a9-2ffdaef6375d-workload-certs\") pod \"e2e-ait-87191e54-openshift-default-7c59fcc9cc-6s8gq\" (UID: \"65bede99-6a95-4ac2-84a9-2ffdaef6375d\") " pod="openshift-ingress/e2e-ait-87191e54-openshift-default-7c59fcc9cc-6s8gq" Nov 04 02:55:37.429707 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:55:37.429478 4405 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/65bede99-6a95-4ac2-84a9-2ffdaef6375d-istio-data\") pod \"e2e-ait-87191e54-openshift-default-7c59fcc9cc-6s8gq\" (UID: \"65bede99-6a95-4ac2-84a9-2ffdaef6375d\") " pod="openshift-ingress/e2e-ait-87191e54-openshift-default-7c59fcc9cc-6s8gq" Nov 04 02:55:37.429707 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:55:37.429508 4405 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/65bede99-6a95-4ac2-84a9-2ffdaef6375d-istio-podinfo\") pod \"e2e-ait-87191e54-openshift-default-7c59fcc9cc-6s8gq\" (UID: \"65bede99-6a95-4ac2-84a9-2ffdaef6375d\") " pod="openshift-ingress/e2e-ait-87191e54-openshift-default-7c59fcc9cc-6s8gq" Nov 04 02:55:37.429707 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:55:37.429576 4405 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/65bede99-6a95-4ac2-84a9-2ffdaef6375d-credential-socket\") pod \"e2e-ait-87191e54-openshift-default-7c59fcc9cc-6s8gq\" (UID: \"65bede99-6a95-4ac2-84a9-2ffdaef6375d\") " pod="openshift-ingress/e2e-ait-87191e54-openshift-default-7c59fcc9cc-6s8gq" Nov 04 02:55:37.432954 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:55:37.429845 4405 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/65bede99-6a95-4ac2-84a9-2ffdaef6375d-istio-envoy\") pod \"e2e-ait-87191e54-openshift-default-7c59fcc9cc-6s8gq\" (UID: \"65bede99-6a95-4ac2-84a9-2ffdaef6375d\") " pod="openshift-ingress/e2e-ait-87191e54-openshift-default-7c59fcc9cc-6s8gq" Nov 04 02:55:37.432954 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:55:37.429906 4405 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mrj4j\" (UniqueName: \"kubernetes.io/projected/65bede99-6a95-4ac2-84a9-2ffdaef6375d-kube-api-access-mrj4j\") pod \"e2e-ait-87191e54-openshift-default-7c59fcc9cc-6s8gq\" (UID: \"65bede99-6a95-4ac2-84a9-2ffdaef6375d\") " pod="openshift-ingress/e2e-ait-87191e54-openshift-default-7c59fcc9cc-6s8gq" Nov 04 02:55:37.453834 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:55:37.453795 4405 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-87191e54-openshift-default-7c59fcc9cc-6s8gq"] Nov 04 02:55:37.532578 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:55:37.532532 4405 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/65bede99-6a95-4ac2-84a9-2ffdaef6375d-workload-certs\") pod \"e2e-ait-87191e54-openshift-default-7c59fcc9cc-6s8gq\" (UID: \"65bede99-6a95-4ac2-84a9-2ffdaef6375d\") " pod="openshift-ingress/e2e-ait-87191e54-openshift-default-7c59fcc9cc-6s8gq" Nov 04 02:55:37.532983 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:55:37.532956 4405 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/65bede99-6a95-4ac2-84a9-2ffdaef6375d-istio-data\") pod \"e2e-ait-87191e54-openshift-default-7c59fcc9cc-6s8gq\" (UID: \"65bede99-6a95-4ac2-84a9-2ffdaef6375d\") " pod="openshift-ingress/e2e-ait-87191e54-openshift-default-7c59fcc9cc-6s8gq" Nov 04 02:55:37.533169 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:55:37.533149 4405 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/65bede99-6a95-4ac2-84a9-2ffdaef6375d-istio-podinfo\") pod \"e2e-ait-87191e54-openshift-default-7c59fcc9cc-6s8gq\" (UID: \"65bede99-6a95-4ac2-84a9-2ffdaef6375d\") " pod="openshift-ingress/e2e-ait-87191e54-openshift-default-7c59fcc9cc-6s8gq" Nov 04 02:55:37.533379 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:55:37.533354 4405 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/65bede99-6a95-4ac2-84a9-2ffdaef6375d-credential-socket\") pod \"e2e-ait-87191e54-openshift-default-7c59fcc9cc-6s8gq\" (UID: \"65bede99-6a95-4ac2-84a9-2ffdaef6375d\") " pod="openshift-ingress/e2e-ait-87191e54-openshift-default-7c59fcc9cc-6s8gq" Nov 04 02:55:37.533817 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:55:37.533783 4405 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/65bede99-6a95-4ac2-84a9-2ffdaef6375d-istio-envoy\") pod \"e2e-ait-87191e54-openshift-default-7c59fcc9cc-6s8gq\" (UID: \"65bede99-6a95-4ac2-84a9-2ffdaef6375d\") " pod="openshift-ingress/e2e-ait-87191e54-openshift-default-7c59fcc9cc-6s8gq" Nov 04 02:55:37.534000 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:55:37.533981 4405 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mrj4j\" (UniqueName: \"kubernetes.io/projected/65bede99-6a95-4ac2-84a9-2ffdaef6375d-kube-api-access-mrj4j\") pod \"e2e-ait-87191e54-openshift-default-7c59fcc9cc-6s8gq\" (UID: \"65bede99-6a95-4ac2-84a9-2ffdaef6375d\") " pod="openshift-ingress/e2e-ait-87191e54-openshift-default-7c59fcc9cc-6s8gq" Nov 04 02:55:37.534235 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:55:37.534214 4405 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/65bede99-6a95-4ac2-84a9-2ffdaef6375d-workload-socket\") pod \"e2e-ait-87191e54-openshift-default-7c59fcc9cc-6s8gq\" (UID: \"65bede99-6a95-4ac2-84a9-2ffdaef6375d\") " pod="openshift-ingress/e2e-ait-87191e54-openshift-default-7c59fcc9cc-6s8gq" Nov 04 02:55:37.534374 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:55:37.534352 4405 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/65bede99-6a95-4ac2-84a9-2ffdaef6375d-istio-token\") pod \"e2e-ait-87191e54-openshift-default-7c59fcc9cc-6s8gq\" (UID: \"65bede99-6a95-4ac2-84a9-2ffdaef6375d\") " pod="openshift-ingress/e2e-ait-87191e54-openshift-default-7c59fcc9cc-6s8gq" Nov 04 02:55:37.536347 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:55:37.536323 4405 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/65bede99-6a95-4ac2-84a9-2ffdaef6375d-istiod-ca-cert\") pod \"e2e-ait-87191e54-openshift-default-7c59fcc9cc-6s8gq\" (UID: \"65bede99-6a95-4ac2-84a9-2ffdaef6375d\") " pod="openshift-ingress/e2e-ait-87191e54-openshift-default-7c59fcc9cc-6s8gq" Nov 04 02:55:37.540258 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:55:37.540175 4405 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/65bede99-6a95-4ac2-84a9-2ffdaef6375d-istiod-ca-cert\") pod \"e2e-ait-87191e54-openshift-default-7c59fcc9cc-6s8gq\" (UID: \"65bede99-6a95-4ac2-84a9-2ffdaef6375d\") " pod="openshift-ingress/e2e-ait-87191e54-openshift-default-7c59fcc9cc-6s8gq" Nov 04 02:55:37.558721 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:55:37.551762 4405 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/65bede99-6a95-4ac2-84a9-2ffdaef6375d-workload-certs\") pod \"e2e-ait-87191e54-openshift-default-7c59fcc9cc-6s8gq\" (UID: \"65bede99-6a95-4ac2-84a9-2ffdaef6375d\") " pod="openshift-ingress/e2e-ait-87191e54-openshift-default-7c59fcc9cc-6s8gq" Nov 04 02:55:37.558721 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:55:37.552860 4405 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/65bede99-6a95-4ac2-84a9-2ffdaef6375d-istio-data\") pod \"e2e-ait-87191e54-openshift-default-7c59fcc9cc-6s8gq\" (UID: \"65bede99-6a95-4ac2-84a9-2ffdaef6375d\") " pod="openshift-ingress/e2e-ait-87191e54-openshift-default-7c59fcc9cc-6s8gq" Nov 04 02:55:37.558721 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:55:37.553693 4405 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/65bede99-6a95-4ac2-84a9-2ffdaef6375d-credential-socket\") pod \"e2e-ait-87191e54-openshift-default-7c59fcc9cc-6s8gq\" (UID: \"65bede99-6a95-4ac2-84a9-2ffdaef6375d\") " pod="openshift-ingress/e2e-ait-87191e54-openshift-default-7c59fcc9cc-6s8gq" Nov 04 02:55:37.558721 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:55:37.553818 4405 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/65bede99-6a95-4ac2-84a9-2ffdaef6375d-workload-socket\") pod \"e2e-ait-87191e54-openshift-default-7c59fcc9cc-6s8gq\" (UID: \"65bede99-6a95-4ac2-84a9-2ffdaef6375d\") " pod="openshift-ingress/e2e-ait-87191e54-openshift-default-7c59fcc9cc-6s8gq" Nov 04 02:55:37.576620 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:55:37.575229 4405 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/65bede99-6a95-4ac2-84a9-2ffdaef6375d-istio-podinfo\") pod \"e2e-ait-87191e54-openshift-default-7c59fcc9cc-6s8gq\" (UID: \"65bede99-6a95-4ac2-84a9-2ffdaef6375d\") " pod="openshift-ingress/e2e-ait-87191e54-openshift-default-7c59fcc9cc-6s8gq" Nov 04 02:55:37.579854 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:55:37.579826 4405 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/65bede99-6a95-4ac2-84a9-2ffdaef6375d-istio-envoy\") pod \"e2e-ait-87191e54-openshift-default-7c59fcc9cc-6s8gq\" (UID: \"65bede99-6a95-4ac2-84a9-2ffdaef6375d\") " pod="openshift-ingress/e2e-ait-87191e54-openshift-default-7c59fcc9cc-6s8gq" Nov 04 02:55:37.618374 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:55:37.618301 4405 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/65bede99-6a95-4ac2-84a9-2ffdaef6375d-istio-token\") pod \"e2e-ait-87191e54-openshift-default-7c59fcc9cc-6s8gq\" (UID: \"65bede99-6a95-4ac2-84a9-2ffdaef6375d\") " pod="openshift-ingress/e2e-ait-87191e54-openshift-default-7c59fcc9cc-6s8gq" Nov 04 02:55:37.618520 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:55:37.618388 4405 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mrj4j\" (UniqueName: \"kubernetes.io/projected/65bede99-6a95-4ac2-84a9-2ffdaef6375d-kube-api-access-mrj4j\") pod \"e2e-ait-87191e54-openshift-default-7c59fcc9cc-6s8gq\" (UID: \"65bede99-6a95-4ac2-84a9-2ffdaef6375d\") " pod="openshift-ingress/e2e-ait-87191e54-openshift-default-7c59fcc9cc-6s8gq" Nov 04 02:55:37.652578 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:55:37.652534 4405 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-87191e54-openshift-default-7c59fcc9cc-6s8gq" Nov 04 02:55:38.105959 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:55:38.105928 4405 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-87191e54-openshift-default-7c59fcc9cc-6s8gq"] Nov 04 02:55:38.107297 ip-10-0-134-125 kubenswrapper[4405]: W1104 02:55:38.107264 4405 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod65bede99_6a95_4ac2_84a9_2ffdaef6375d.slice/crio-2add8c4ed047724876e03323c864274c6e37d02559efe443ae15a22ca2a0d753 WatchSource:0}: Error finding container 2add8c4ed047724876e03323c864274c6e37d02559efe443ae15a22ca2a0d753: Status 404 returned error can't find the container with id 2add8c4ed047724876e03323c864274c6e37d02559efe443ae15a22ca2a0d753 Nov 04 02:55:38.110410 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:55:38.110371 4405 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 04 02:55:38.110506 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:55:38.110457 4405 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 04 02:55:38.110506 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:55:38.110498 4405 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 04 02:55:39.002493 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:55:39.002458 4405 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-87191e54-openshift-default-7c59fcc9cc-6s8gq" event={"ID":"65bede99-6a95-4ac2-84a9-2ffdaef6375d","Type":"ContainerStarted","Data":"4ecb3c5f8ee794f0d2eb13e06c3b61c5179f14d4a2be11269ab328d86637081a"} Nov 04 02:55:39.002741 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:55:39.002520 4405 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-87191e54-openshift-default-7c59fcc9cc-6s8gq" event={"ID":"65bede99-6a95-4ac2-84a9-2ffdaef6375d","Type":"ContainerStarted","Data":"2add8c4ed047724876e03323c864274c6e37d02559efe443ae15a22ca2a0d753"} Nov 04 02:55:39.435121 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:55:39.434736 4405 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-87191e54-openshift-default-7c59fcc9cc-6s8gq" podStartSLOduration=2.434716304 podStartE2EDuration="2.434716304s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 02:55:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 02:55:39.384431848 +0000 UTC m=+1385.815131995" watchObservedRunningTime="2026-11-04 02:55:39.434716304 +0000 UTC m=+1385.865416427" Nov 04 02:55:39.653409 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:55:39.653372 4405 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-87191e54-openshift-default-7c59fcc9cc-6s8gq" Nov 04 02:55:39.660268 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:55:39.660228 4405 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-87191e54-openshift-default-7c59fcc9cc-6s8gq" Nov 04 02:55:40.011842 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:55:40.011809 4405 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-87191e54-openshift-default-7c59fcc9cc-6s8gq" Nov 04 02:55:40.013868 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:55:40.013839 4405 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-87191e54-openshift-default-7c59fcc9cc-6s8gq" Nov 04 02:55:44.747700 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:55:44.747626 4405 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-cdcf67/distinct-w3-cdcf67-kserve-667cc7f7d7-5m2lq" podUID="52ed22cc-96e4-4ef0-b34d-ed58c89a9183" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 02:55:44.953594 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:55:44.953551 4405 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-e516ff/distinct-w1-e516ff-kserve-f4474cb9c-glpzw" podUID="8842ad09-22ff-4492-ba5d-d4cacf4eac75" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 02:55:44.962692 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:55:44.961754 4405 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-203c35/simulator-w4-203c35-kserve-f9c4bd695-999ns" podUID="eb0d8bdd-6808-4fd0-b9b5-567dff3e7ca5" containerName="main" probeResult="failure" output="Get \"https://10.134.0.46:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 02:55:45.018757 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:55:45.006784 4405 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-ebb7fd/distinct-w6-ebb7fd-kserve-55c47d8865-7z9pz" podUID="ccb8e181-5610-46da-bb8b-e0b07fa02ff1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": context deadline exceeded" Nov 04 02:55:45.018757 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:55:45.007267 4405 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-fe4e0c/distinct-2-w0-fe4e0c-kserve-579bf4cc77-8pbq8" podUID="2b2ab722-d1bb-4376-bede-f5f4ee4562aa" containerName="main" probeResult="failure" output="Get \"https://10.134.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 02:55:45.099075 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:55:45.098219 4405 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-a0bcbc/distinct-w2-a0bcbc-kserve-596f94c7cd-szl4g" podUID="bac32934-8a57-4919-b435-13f367cbea3e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 02:55:45.236792 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:55:45.235773 4405 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-ljhhw" podUID="f66b1861-d87b-4d57-ab27-f1a24544a534" containerName="main" probeResult="failure" output="Get \"https://10.134.0.26:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 02:55:45.356867 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:55:45.356826 4405 patch_prober.go:29] interesting pod/e2e-worker-w4-203c35-openshift-default-56497cdcc-w6c48 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.70:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 04 02:55:45.357195 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:55:45.357166 4405 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-203c35-openshift-default-56497cdcc-w6c48" podUID="89ddc69b-94de-4443-a62a-fa5d360aa368" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.70:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 04 02:55:45.534829 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:55:45.534769 4405 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-a0bcbc/premium-w2-a0bcbc-kserve-78dfd6d78c-lwth2" podUID="33b620c1-389c-48b4-b277-90252018c4b6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 02:55:45.553823 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:55:45.553771 4405 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-ebb7fd/distinct-2-w6-ebb7fd-kserve-6f68c5db86-zrpjs" podUID="b7b786e0-9c94-4126-9949-cd354ebbfb2a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 02:55:45.759761 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:55:45.749882 4405 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-cdcf67/distinct-w3-cdcf67-kserve-667cc7f7d7-5m2lq" podUID="52ed22cc-96e4-4ef0-b34d-ed58c89a9183" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 02:55:45.759761 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:55:45.750612 4405 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-cdcf67/distinct-w3-cdcf67-kserve-667cc7f7d7-5m2lq" Nov 04 02:55:45.874818 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:55:45.874768 4405 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-203c35/simulator-w4-203c35-kserve-f9c4bd695-999ns" podUID="eb0d8bdd-6808-4fd0-b9b5-567dff3e7ca5" containerName="main" probeResult="failure" output="Get \"https://10.134.0.46:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 02:55:46.090281 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:55:45.961841 4405 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-e516ff/distinct-w1-e516ff-kserve-f4474cb9c-glpzw" podUID="8842ad09-22ff-4492-ba5d-d4cacf4eac75" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/ready\": context deadline exceeded" Nov 04 02:55:46.090722 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:55:45.961888 4405 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-203c35/simulator-w4-203c35-kserve-f9c4bd695-999ns" podUID="eb0d8bdd-6808-4fd0-b9b5-567dff3e7ca5" containerName="main" probeResult="failure" output="Get \"https://10.134.0.46:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 02:55:46.115697 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:55:46.053767 4405 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-fe4e0c/distinct-2-w0-fe4e0c-kserve-579bf4cc77-8pbq8" podUID="2b2ab722-d1bb-4376-bede-f5f4ee4562aa" containerName="main" probeResult="failure" output="Get \"https://10.134.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 02:55:46.129697 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:55:46.053805 4405 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-ebb7fd/distinct-w6-ebb7fd-kserve-55c47d8865-7z9pz" podUID="ccb8e181-5610-46da-bb8b-e0b07fa02ff1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 02:55:46.153492 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:55:46.112417 4405 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-a0bcbc/distinct-w2-a0bcbc-kserve-596f94c7cd-szl4g" podUID="bac32934-8a57-4919-b435-13f367cbea3e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 02:55:46.236260 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:55:46.236218 4405 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-ljhhw" podUID="f66b1861-d87b-4d57-ab27-f1a24544a534" containerName="main" probeResult="failure" output="Get \"https://10.134.0.26:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 02:55:46.345753 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:55:46.343452 4405 patch_prober.go:29] interesting pod/e2e-worker-w4-203c35-openshift-default-56497cdcc-ljwqc container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.71:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 04 02:55:46.345753 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:55:46.343532 4405 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-203c35-openshift-default-56497cdcc-ljwqc" podUID="fec507be-e742-4226-a6c4-6fb41e6f3479" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.71:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 04 02:55:46.345753 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:55:46.343626 4405 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-e516ff/distinct-w1-e516ff-kserve-f4474cb9c-glpzw" podUID="8842ad09-22ff-4492-ba5d-d4cacf4eac75" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 02:55:46.382695 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:55:46.347136 4405 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-e516ff/distinct-w1-e516ff-kserve-f4474cb9c-glpzw" Nov 04 02:55:46.382695 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:55:46.347197 4405 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-203c35/simulator-w4-203c35-kserve-f9c4bd695-999ns" Nov 04 02:55:46.382695 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:55:46.347231 4405 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-fe4e0c/distinct-2-w0-fe4e0c-kserve-579bf4cc77-8pbq8" Nov 04 02:55:46.382695 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:55:46.347266 4405 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-ebb7fd/distinct-w6-ebb7fd-kserve-55c47d8865-7z9pz" Nov 04 02:55:46.382695 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:55:46.347302 4405 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-a0bcbc/distinct-w2-a0bcbc-kserve-596f94c7cd-szl4g" Nov 04 02:55:46.382695 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:55:46.347339 4405 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-ljhhw" Nov 04 02:55:46.555694 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:55:46.555445 4405 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-a0bcbc/premium-w2-a0bcbc-kserve-78dfd6d78c-lwth2" podUID="33b620c1-389c-48b4-b277-90252018c4b6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/ready\": context deadline exceeded" Nov 04 02:55:46.580696 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:55:46.556137 4405 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-a0bcbc/premium-w2-a0bcbc-kserve-78dfd6d78c-lwth2" Nov 04 02:55:46.580696 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:55:46.556391 4405 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-ebb7fd/distinct-2-w6-ebb7fd-kserve-6f68c5db86-zrpjs" podUID="b7b786e0-9c94-4126-9949-cd354ebbfb2a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 02:55:46.686757 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:55:46.682997 4405 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-ebb7fd/distinct-2-w6-ebb7fd-kserve-6f68c5db86-zrpjs" Nov 04 02:55:46.789794 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:55:46.789746 4405 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-cdcf67/distinct-w3-cdcf67-kserve-667cc7f7d7-5m2lq" podUID="52ed22cc-96e4-4ef0-b34d-ed58c89a9183" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 02:55:47.141274 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:55:47.141212 4405 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-fe4e0c/distinct-2-w0-fe4e0c-kserve-579bf4cc77-8pbq8" podUID="2b2ab722-d1bb-4376-bede-f5f4ee4562aa" containerName="main" probeResult="failure" output="Get \"https://10.134.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 02:55:47.190867 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:55:47.190816 4405 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-203c35/simulator-w4-203c35-kserve-f9c4bd695-999ns" podUID="eb0d8bdd-6808-4fd0-b9b5-567dff3e7ca5" containerName="main" probeResult="failure" output="Get \"https://10.134.0.46:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 02:55:47.204723 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:55:47.196273 4405 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-ebb7fd/distinct-w6-ebb7fd-kserve-55c47d8865-7z9pz" podUID="ccb8e181-5610-46da-bb8b-e0b07fa02ff1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": context deadline exceeded" Nov 04 02:55:47.204723 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:55:47.197003 4405 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-e516ff/distinct-w1-e516ff-kserve-f4474cb9c-glpzw" podUID="8842ad09-22ff-4492-ba5d-d4cacf4eac75" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/ready\": context deadline exceeded" Nov 04 02:55:47.204723 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:55:47.197102 4405 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-a0bcbc/distinct-w2-a0bcbc-kserve-596f94c7cd-szl4g" podUID="bac32934-8a57-4919-b435-13f367cbea3e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": context deadline exceeded" Nov 04 02:55:47.242985 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:55:47.242942 4405 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-ljhhw" podUID="f66b1861-d87b-4d57-ab27-f1a24544a534" containerName="main" probeResult="failure" output="Get \"https://10.134.0.26:8000/ready\": context deadline exceeded" Nov 04 02:55:47.624692 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:55:47.609345 4405 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-ebb7fd/distinct-2-w6-ebb7fd-kserve-6f68c5db86-zrpjs" podUID="b7b786e0-9c94-4126-9949-cd354ebbfb2a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 02:55:47.624692 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:55:47.609869 4405 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-a0bcbc/premium-w2-a0bcbc-kserve-78dfd6d78c-lwth2" podUID="33b620c1-389c-48b4-b277-90252018c4b6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 02:55:47.837447 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:55:47.837396 4405 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-cdcf67/distinct-w3-cdcf67-kserve-667cc7f7d7-5m2lq" podUID="52ed22cc-96e4-4ef0-b34d-ed58c89a9183" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 02:55:48.177769 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:55:48.172709 4405 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-fe4e0c/distinct-2-w0-fe4e0c-kserve-579bf4cc77-8pbq8" podUID="2b2ab722-d1bb-4376-bede-f5f4ee4562aa" containerName="main" probeResult="failure" output="Get \"https://10.134.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 02:55:48.228695 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:55:48.198564 4405 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-a0bcbc/distinct-w2-a0bcbc-kserve-596f94c7cd-szl4g" podUID="bac32934-8a57-4919-b435-13f367cbea3e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": context deadline exceeded" Nov 04 02:55:48.236693 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:55:48.229328 4405 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-203c35/simulator-w4-203c35-kserve-f9c4bd695-999ns" podUID="eb0d8bdd-6808-4fd0-b9b5-567dff3e7ca5" containerName="main" probeResult="failure" output="Get \"https://10.134.0.46:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 02:55:48.297693 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:55:48.234630 4405 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-e516ff/distinct-w1-e516ff-kserve-f4474cb9c-glpzw" podUID="8842ad09-22ff-4492-ba5d-d4cacf4eac75" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/ready\": context deadline exceeded" Nov 04 02:55:48.319106 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:55:48.234691 4405 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-a0bcbc/distinct-w2-a0bcbc-kserve-596f94c7cd-szl4g" podUID="bac32934-8a57-4919-b435-13f367cbea3e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 02:55:48.330695 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:55:48.262204 4405 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-ljhhw" podUID="f66b1861-d87b-4d57-ab27-f1a24544a534" containerName="main" probeResult="failure" output="Get \"https://10.134.0.26:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 02:55:48.359695 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:55:48.266773 4405 patch_prober.go:29] interesting pod/e2e-worker-w6-ebb7fd-openshift-default-544f5768f-mngkl container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.67:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 04 02:55:48.360063 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:55:48.360026 4405 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w6-ebb7fd-openshift-default-544f5768f-mngkl" podUID="f55884d8-38e1-416b-853d-9c09082444bf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.67:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 04 02:55:48.360496 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:55:48.266821 4405 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-ebb7fd/distinct-w6-ebb7fd-kserve-55c47d8865-7z9pz" podUID="ccb8e181-5610-46da-bb8b-e0b07fa02ff1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 02:55:48.624562 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:55:48.624517 4405 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-a0bcbc/premium-w2-a0bcbc-kserve-78dfd6d78c-lwth2" podUID="33b620c1-389c-48b4-b277-90252018c4b6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 02:55:48.651693 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:55:48.625803 4405 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-ebb7fd/distinct-2-w6-ebb7fd-kserve-6f68c5db86-zrpjs" podUID="b7b786e0-9c94-4126-9949-cd354ebbfb2a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.65:8000/ready\": context deadline exceeded" Nov 04 02:55:48.706595 ip-10-0-134-125 kubenswrapper[4405]: E1104 02:55:48.706560 4405 kubelet.go:2711] "Housekeeping took longer than expected" err="housekeeping took too long" expected="1s" actual="1.011s" Nov 04 02:55:48.853493 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:55:48.853436 4405 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-cdcf67/distinct-w3-cdcf67-kserve-667cc7f7d7-5m2lq" podUID="52ed22cc-96e4-4ef0-b34d-ed58c89a9183" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 02:55:48.873731 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:55:48.873682 4405 prober.go:122] "Probe failed" probeType="Liveness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-ljhhw" podUID="f66b1861-d87b-4d57-ab27-f1a24544a534" containerName="main" probeResult="failure" output="Get \"https://10.134.0.26:8000/health\": context deadline exceeded" Nov 04 02:55:49.074445 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:55:49.074331 4405 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-ebb7fd/distinct-w6-ebb7fd-kserve-55c47d8865-7z9pz" podUID="ccb8e181-5610-46da-bb8b-e0b07fa02ff1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 02:55:49.266697 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:55:49.231949 4405 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-fe4e0c/distinct-2-w0-fe4e0c-kserve-579bf4cc77-8pbq8" podUID="2b2ab722-d1bb-4376-bede-f5f4ee4562aa" containerName="main" probeResult="failure" output="Get \"https://10.134.0.68:8000/ready\": context deadline exceeded" Nov 04 02:55:49.355754 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:55:49.326348 4405 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-e516ff/distinct-w1-e516ff-kserve-f4474cb9c-glpzw" podUID="8842ad09-22ff-4492-ba5d-d4cacf4eac75" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/ready\": context deadline exceeded" Nov 04 02:55:49.370984 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:55:49.370925 4405 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-203c35/simulator-w4-203c35-kserve-f9c4bd695-999ns" podUID="eb0d8bdd-6808-4fd0-b9b5-567dff3e7ca5" containerName="main" probeResult="failure" output="Get \"https://10.134.0.46:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 02:55:49.434928 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:55:49.434866 4405 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-a0bcbc/distinct-w2-a0bcbc-kserve-596f94c7cd-szl4g" podUID="bac32934-8a57-4919-b435-13f367cbea3e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 02:55:49.496695 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:55:49.438310 4405 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-ljhhw" podUID="f66b1861-d87b-4d57-ab27-f1a24544a534" containerName="main" probeResult="failure" output="Get \"https://10.134.0.26:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 02:55:49.555263 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:55:49.513467 4405 patch_prober.go:29] interesting pod/e2e-worker-w0-fe4e0c-openshift-default-c7f4fc84c-vh6fd container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.72:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 04 02:55:49.582221 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:55:49.582176 4405 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-fe4e0c-openshift-default-c7f4fc84c-vh6fd" podUID="70d9a666-2667-41e8-8425-4e2ba085599d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.72:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 04 02:55:49.582510 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:55:49.547072 4405 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-ebb7fd/distinct-w6-ebb7fd-kserve-55c47d8865-7z9pz" podUID="ccb8e181-5610-46da-bb8b-e0b07fa02ff1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": context deadline exceeded" Nov 04 02:55:49.682756 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:55:49.657876 4405 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-ebb7fd/distinct-2-w6-ebb7fd-kserve-6f68c5db86-zrpjs" podUID="b7b786e0-9c94-4126-9949-cd354ebbfb2a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 02:55:49.766121 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:55:49.706172 4405 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-a0bcbc/premium-w2-a0bcbc-kserve-78dfd6d78c-lwth2" podUID="33b620c1-389c-48b4-b277-90252018c4b6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/ready\": context deadline exceeded" Nov 04 02:55:49.981760 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:55:49.894740 4405 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-cdcf67/distinct-w3-cdcf67-kserve-667cc7f7d7-5m2lq" podUID="52ed22cc-96e4-4ef0-b34d-ed58c89a9183" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/ready\": context deadline exceeded" Nov 04 02:55:49.981760 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:55:49.922022 4405 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-a0bcbc/premium-w2-a0bcbc-kserve-78dfd6d78c-lwth2" podUID="33b620c1-389c-48b4-b277-90252018c4b6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 02:55:50.020872 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:55:50.020811 4405 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-cdcf67/distinct-w3-cdcf67-kserve-667cc7f7d7-5m2lq" podUID="52ed22cc-96e4-4ef0-b34d-ed58c89a9183" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/health\": context deadline exceeded" Nov 04 02:55:50.411708 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:55:50.392378 4405 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-203c35/simulator-w4-203c35-kserve-f9c4bd695-999ns" podUID="eb0d8bdd-6808-4fd0-b9b5-567dff3e7ca5" containerName="main" probeResult="failure" output="Get \"https://10.134.0.46:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 02:55:50.424704 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:55:50.420930 4405 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-e516ff/distinct-w1-e516ff-kserve-f4474cb9c-glpzw" podUID="8842ad09-22ff-4492-ba5d-d4cacf4eac75" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/ready\": context deadline exceeded" Nov 04 02:55:50.429709 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:55:50.420964 4405 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-fe4e0c/distinct-2-w0-fe4e0c-kserve-579bf4cc77-8pbq8" podUID="2b2ab722-d1bb-4376-bede-f5f4ee4562aa" containerName="main" probeResult="failure" output="Get \"https://10.134.0.68:8000/ready\": context deadline exceeded" Nov 04 02:55:50.495692 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:55:50.491805 4405 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-a0bcbc/distinct-w2-a0bcbc-kserve-596f94c7cd-szl4g" podUID="bac32934-8a57-4919-b435-13f367cbea3e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 02:55:50.555870 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:55:50.555820 4405 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-ljhhw" podUID="f66b1861-d87b-4d57-ab27-f1a24544a534" containerName="main" probeResult="failure" output="Get \"https://10.134.0.26:8000/ready\": context deadline exceeded" Nov 04 02:55:50.611729 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:55:50.587636 4405 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-ebb7fd/distinct-w6-ebb7fd-kserve-55c47d8865-7z9pz" podUID="ccb8e181-5610-46da-bb8b-e0b07fa02ff1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 02:55:50.684176 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:55:50.684081 4405 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-ebb7fd/distinct-2-w6-ebb7fd-kserve-6f68c5db86-zrpjs" podUID="b7b786e0-9c94-4126-9949-cd354ebbfb2a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 02:55:50.919696 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:55:50.906573 4405 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-a0bcbc/premium-w2-a0bcbc-kserve-78dfd6d78c-lwth2" podUID="33b620c1-389c-48b4-b277-90252018c4b6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 02:55:50.945756 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:55:50.910582 4405 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-cdcf67/distinct-w3-cdcf67-kserve-667cc7f7d7-5m2lq" podUID="52ed22cc-96e4-4ef0-b34d-ed58c89a9183" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 02:55:51.446692 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:55:51.445894 4405 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-fe4e0c/distinct-2-w0-fe4e0c-kserve-579bf4cc77-8pbq8" podUID="2b2ab722-d1bb-4376-bede-f5f4ee4562aa" containerName="main" probeResult="failure" output="Get \"https://10.134.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 02:55:51.446692 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:55:51.446361 4405 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-203c35/simulator-w4-203c35-kserve-f9c4bd695-999ns" podUID="eb0d8bdd-6808-4fd0-b9b5-567dff3e7ca5" containerName="main" probeResult="failure" output="Get \"https://10.134.0.46:8000/ready\": context deadline exceeded" Nov 04 02:55:51.447297 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:55:51.446740 4405 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-e516ff/distinct-w1-e516ff-kserve-f4474cb9c-glpzw" podUID="8842ad09-22ff-4492-ba5d-d4cacf4eac75" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/ready\": context deadline exceeded" Nov 04 02:55:51.530697 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:55:51.527866 4405 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-a0bcbc/distinct-w2-a0bcbc-kserve-596f94c7cd-szl4g" podUID="bac32934-8a57-4919-b435-13f367cbea3e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": context deadline exceeded" Nov 04 02:55:51.628979 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:55:51.628927 4405 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-ljhhw" podUID="f66b1861-d87b-4d57-ab27-f1a24544a534" containerName="main" probeResult="failure" output="Get \"https://10.134.0.26:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 02:55:51.638695 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:55:51.628881 4405 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-ebb7fd/distinct-w6-ebb7fd-kserve-55c47d8865-7z9pz" podUID="ccb8e181-5610-46da-bb8b-e0b07fa02ff1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 02:55:51.638695 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:55:51.631115 4405 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-ebb7fd/distinct-2-w6-ebb7fd-kserve-6f68c5db86-zrpjs" podUID="b7b786e0-9c94-4126-9949-cd354ebbfb2a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.65:8000/health\": context deadline exceeded" Nov 04 02:55:51.703927 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:55:51.703814 4405 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-ebb7fd/distinct-2-w6-ebb7fd-kserve-6f68c5db86-zrpjs" podUID="b7b786e0-9c94-4126-9949-cd354ebbfb2a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.65:8000/ready\": context deadline exceeded" Nov 04 02:55:51.931705 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:55:51.920146 4405 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-a0bcbc/premium-w2-a0bcbc-kserve-78dfd6d78c-lwth2" podUID="33b620c1-389c-48b4-b277-90252018c4b6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 02:55:51.963754 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:55:51.957782 4405 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-cdcf67/distinct-w3-cdcf67-kserve-667cc7f7d7-5m2lq" podUID="52ed22cc-96e4-4ef0-b34d-ed58c89a9183" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 02:55:52.457912 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:55:52.457829 4405 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-e516ff/distinct-w1-e516ff-kserve-f4474cb9c-glpzw" podUID="8842ad09-22ff-4492-ba5d-d4cacf4eac75" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 02:55:52.459687 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:55:52.459634 4405 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-fe4e0c/distinct-2-w0-fe4e0c-kserve-579bf4cc77-8pbq8" podUID="2b2ab722-d1bb-4376-bede-f5f4ee4562aa" containerName="main" probeResult="failure" output="Get \"https://10.134.0.68:8000/ready\": context deadline exceeded" Nov 04 02:55:52.460880 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:55:52.460856 4405 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-203c35/simulator-w4-203c35-kserve-f9c4bd695-999ns" podUID="eb0d8bdd-6808-4fd0-b9b5-567dff3e7ca5" containerName="main" probeResult="failure" output="Get \"https://10.134.0.46:8000/ready\": context deadline exceeded" Nov 04 02:55:52.549693 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:55:52.539486 4405 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-a0bcbc/distinct-w2-a0bcbc-kserve-596f94c7cd-szl4g" podUID="bac32934-8a57-4919-b435-13f367cbea3e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 02:55:52.598689 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:55:52.568005 4405 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-ebb7fd/distinct-2-w6-ebb7fd-kserve-6f68c5db86-zrpjs" Nov 04 02:55:52.649542 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:55:52.649489 4405 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-ljhhw" podUID="f66b1861-d87b-4d57-ab27-f1a24544a534" containerName="main" probeResult="failure" output="Get \"https://10.134.0.26:8000/ready\": context deadline exceeded" Nov 04 02:55:52.670138 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:55:52.669327 4405 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-ebb7fd/distinct-w6-ebb7fd-kserve-55c47d8865-7z9pz" Nov 04 02:55:52.670138 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:55:52.669733 4405 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-cdcf67/distinct-w3-cdcf67-kserve-667cc7f7d7-5m2lq" Nov 04 02:55:52.771703 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:55:52.771581 4405 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-fe4e0c/distinct-2-w0-fe4e0c-kserve-579bf4cc77-8pbq8" Nov 04 02:55:52.772037 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:55:52.772021 4405 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-e516ff/distinct-w1-e516ff-kserve-f4474cb9c-glpzw" Nov 04 02:55:52.780572 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:55:52.780544 4405 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-a0bcbc/distinct-w2-a0bcbc-kserve-596f94c7cd-szl4g" Nov 04 02:55:52.806688 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:55:52.801988 4405 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-ljhhw" Nov 04 02:55:52.813381 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:55:52.813357 4405 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-203c35/simulator-w4-203c35-kserve-f9c4bd695-999ns" Nov 04 02:55:52.893048 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:55:52.893018 4405 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-a0bcbc/premium-w2-a0bcbc-kserve-78dfd6d78c-lwth2" Nov 04 02:55:57.607321 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:55:57.607271 4405 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-87191e54-6ffd5f6b7-5qrkx"] Nov 04 02:55:57.625080 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:55:57.625048 4405 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-87191e54-6ffd5f6b7-5qrkx" Nov 04 02:55:57.683691 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:55:57.676802 4405 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-87191e54-6ffd5f6b7-5qrkx"] Nov 04 02:55:57.706640 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:55:57.706585 4405 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-87191e54\"" Nov 04 02:55:57.717154 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:55:57.716635 4405 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-87191e54-dockercfg-9w9zm\"" Nov 04 02:55:57.720495 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:55:57.720462 4405 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l2gln\" (UniqueName: \"kubernetes.io/projected/557eb86e-a5c3-4790-9dde-53619c054583-kube-api-access-l2gln\") pod \"payload-pre-processing-e2e-ait-87191e54-6ffd5f6b7-5qrkx\" (UID: \"557eb86e-a5c3-4790-9dde-53619c054583\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-87191e54-6ffd5f6b7-5qrkx" Nov 04 02:55:57.720781 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:55:57.720766 4405 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/557eb86e-a5c3-4790-9dde-53619c054583-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-87191e54-6ffd5f6b7-5qrkx\" (UID: \"557eb86e-a5c3-4790-9dde-53619c054583\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-87191e54-6ffd5f6b7-5qrkx" Nov 04 02:55:57.756266 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:55:57.756222 4405 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-87191e54-77b46c874-mvmn9"] Nov 04 02:55:57.761005 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:55:57.760978 4405 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-87191e54-77b46c874-mvmn9" Nov 04 02:55:57.830691 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:55:57.827197 4405 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/557eb86e-a5c3-4790-9dde-53619c054583-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-87191e54-6ffd5f6b7-5qrkx\" (UID: \"557eb86e-a5c3-4790-9dde-53619c054583\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-87191e54-6ffd5f6b7-5qrkx" Nov 04 02:55:57.830691 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:55:57.827432 4405 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fb036f6d-278f-4627-952b-c285dc65d364-plugins-config-volume\") pod \"payload-processing-e2e-ait-87191e54-77b46c874-mvmn9\" (UID: \"fb036f6d-278f-4627-952b-c285dc65d364\") " pod="openshift-ingress/payload-processing-e2e-ait-87191e54-77b46c874-mvmn9" Nov 04 02:55:57.830691 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:55:57.827525 4405 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kn4vj\" (UniqueName: \"kubernetes.io/projected/fb036f6d-278f-4627-952b-c285dc65d364-kube-api-access-kn4vj\") pod \"payload-processing-e2e-ait-87191e54-77b46c874-mvmn9\" (UID: \"fb036f6d-278f-4627-952b-c285dc65d364\") " pod="openshift-ingress/payload-processing-e2e-ait-87191e54-77b46c874-mvmn9" Nov 04 02:55:57.830691 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:55:57.827584 4405 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l2gln\" (UniqueName: \"kubernetes.io/projected/557eb86e-a5c3-4790-9dde-53619c054583-kube-api-access-l2gln\") pod \"payload-pre-processing-e2e-ait-87191e54-6ffd5f6b7-5qrkx\" (UID: \"557eb86e-a5c3-4790-9dde-53619c054583\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-87191e54-6ffd5f6b7-5qrkx" Nov 04 02:55:57.842330 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:55:57.839157 4405 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/557eb86e-a5c3-4790-9dde-53619c054583-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-87191e54-6ffd5f6b7-5qrkx\" (UID: \"557eb86e-a5c3-4790-9dde-53619c054583\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-87191e54-6ffd5f6b7-5qrkx" Nov 04 02:55:57.924092 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:55:57.911162 4405 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-87191e54-77b46c874-mvmn9"] Nov 04 02:55:57.981828 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:55:57.969715 4405 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fb036f6d-278f-4627-952b-c285dc65d364-plugins-config-volume\") pod \"payload-processing-e2e-ait-87191e54-77b46c874-mvmn9\" (UID: \"fb036f6d-278f-4627-952b-c285dc65d364\") " pod="openshift-ingress/payload-processing-e2e-ait-87191e54-77b46c874-mvmn9" Nov 04 02:55:57.982798 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:55:57.982755 4405 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kn4vj\" (UniqueName: \"kubernetes.io/projected/fb036f6d-278f-4627-952b-c285dc65d364-kube-api-access-kn4vj\") pod \"payload-processing-e2e-ait-87191e54-77b46c874-mvmn9\" (UID: \"fb036f6d-278f-4627-952b-c285dc65d364\") " pod="openshift-ingress/payload-processing-e2e-ait-87191e54-77b46c874-mvmn9" Nov 04 02:55:57.983720 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:55:57.974642 4405 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fb036f6d-278f-4627-952b-c285dc65d364-plugins-config-volume\") pod \"payload-processing-e2e-ait-87191e54-77b46c874-mvmn9\" (UID: \"fb036f6d-278f-4627-952b-c285dc65d364\") " pod="openshift-ingress/payload-processing-e2e-ait-87191e54-77b46c874-mvmn9" Nov 04 02:55:57.989141 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:55:57.988612 4405 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l2gln\" (UniqueName: \"kubernetes.io/projected/557eb86e-a5c3-4790-9dde-53619c054583-kube-api-access-l2gln\") pod \"payload-pre-processing-e2e-ait-87191e54-6ffd5f6b7-5qrkx\" (UID: \"557eb86e-a5c3-4790-9dde-53619c054583\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-87191e54-6ffd5f6b7-5qrkx" Nov 04 02:55:58.049954 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:55:58.049641 4405 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kn4vj\" (UniqueName: \"kubernetes.io/projected/fb036f6d-278f-4627-952b-c285dc65d364-kube-api-access-kn4vj\") pod \"payload-processing-e2e-ait-87191e54-77b46c874-mvmn9\" (UID: \"fb036f6d-278f-4627-952b-c285dc65d364\") " pod="openshift-ingress/payload-processing-e2e-ait-87191e54-77b46c874-mvmn9" Nov 04 02:55:58.085129 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:55:58.085072 4405 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-87191e54-77b46c874-mvmn9" Nov 04 02:55:58.280239 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:55:58.280202 4405 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-87191e54-6ffd5f6b7-5qrkx" Nov 04 02:55:58.507875 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:55:58.507847 4405 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-87191e54-77b46c874-mvmn9"] Nov 04 02:55:58.508160 ip-10-0-134-125 kubenswrapper[4405]: W1104 02:55:58.508136 4405 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfb036f6d_278f_4627_952b_c285dc65d364.slice/crio-a44c96d047fadf765e4112d953ad9b2e6ade09c8336a65e40013c61a77968956 WatchSource:0}: Error finding container a44c96d047fadf765e4112d953ad9b2e6ade09c8336a65e40013c61a77968956: Status 404 returned error can't find the container with id a44c96d047fadf765e4112d953ad9b2e6ade09c8336a65e40013c61a77968956 Nov 04 02:55:58.631706 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:55:58.631635 4405 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-87191e54-77b46c874-mvmn9" event={"ID":"fb036f6d-278f-4627-952b-c285dc65d364","Type":"ContainerStarted","Data":"f3afc32ff2cfb83e3a9c816f9d854f634823d3f06e98b6102f9d3e13f44ed385"} Nov 04 02:55:58.631706 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:55:58.631708 4405 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-87191e54-77b46c874-mvmn9" event={"ID":"fb036f6d-278f-4627-952b-c285dc65d364","Type":"ContainerStarted","Data":"a44c96d047fadf765e4112d953ad9b2e6ade09c8336a65e40013c61a77968956"} Nov 04 02:55:58.632185 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:55:58.631776 4405 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ait-87191e54-77b46c874-mvmn9" Nov 04 02:55:58.662784 ip-10-0-134-125 kubenswrapper[4405]: W1104 02:55:58.662753 4405 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod557eb86e_a5c3_4790_9dde_53619c054583.slice/crio-46978e9fe92924b42767110e74b5b024b6f56d150649a0c2fcc3929e40639bdb WatchSource:0}: Error finding container 46978e9fe92924b42767110e74b5b024b6f56d150649a0c2fcc3929e40639bdb: Status 404 returned error can't find the container with id 46978e9fe92924b42767110e74b5b024b6f56d150649a0c2fcc3929e40639bdb Nov 04 02:55:58.663386 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:55:58.663364 4405 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-87191e54-6ffd5f6b7-5qrkx"] Nov 04 02:55:58.749879 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:55:58.749823 4405 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ait-87191e54-77b46c874-mvmn9" podStartSLOduration=1.749808255 podStartE2EDuration="1.749808255s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 02:55:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 02:55:58.746701815 +0000 UTC m=+1405.177401936" watchObservedRunningTime="2026-11-04 02:55:58.749808255 +0000 UTC m=+1405.180508408" Nov 04 02:55:59.640964 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:55:59.640927 4405 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-87191e54-6ffd5f6b7-5qrkx" event={"ID":"557eb86e-a5c3-4790-9dde-53619c054583","Type":"ContainerStarted","Data":"e2db79d4b079b0aa8d088543c0adcc425c223516cc4b48e8cb83d3d240f56215"} Nov 04 02:55:59.641380 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:55:59.641026 4405 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-87191e54-6ffd5f6b7-5qrkx" event={"ID":"557eb86e-a5c3-4790-9dde-53619c054583","Type":"ContainerStarted","Data":"46978e9fe92924b42767110e74b5b024b6f56d150649a0c2fcc3929e40639bdb"} Nov 04 02:55:59.641380 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:55:59.641276 4405 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-87191e54-6ffd5f6b7-5qrkx" Nov 04 02:55:59.759065 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:55:59.759009 4405 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ait-87191e54-6ffd5f6b7-5qrkx" podStartSLOduration=2.758994287 podStartE2EDuration="2.758994287s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 02:55:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 02:55:59.757784605 +0000 UTC m=+1406.188484722" watchObservedRunningTime="2026-11-04 02:55:59.758994287 +0000 UTC m=+1406.189694406" Nov 04 02:56:06.891238 ip-10-0-134-125 kubenswrapper[4405]: E1104 02:56:06.891193 4405 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5f2vb" containerName="extractor" Nov 04 02:56:09.645280 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:56:09.645254 4405 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ait-87191e54-77b46c874-mvmn9" Nov 04 02:56:10.651577 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:56:10.651542 4405 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-87191e54-6ffd5f6b7-5qrkx" Nov 04 02:56:13.463818 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:56:13.463776 4405 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-f73e00ed-866fdf7694-m4mwj"] Nov 04 02:56:13.488702 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:56:13.482986 4405 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-f73e00ed-866fdf7694-m4mwj" Nov 04 02:56:13.584731 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:56:13.571491 4405 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-f73e00ed-866fdf7694-m4mwj"] Nov 04 02:56:13.721812 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:56:13.583001 4405 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-f73e00ed\"" Nov 04 02:56:13.721812 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:56:13.583160 4405 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-f73e00ed-dockercfg-srlfg\"" Nov 04 02:56:13.721812 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:56:13.669965 4405 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4ggn6\" (UniqueName: \"kubernetes.io/projected/55a6bf29-4dc5-4b77-9c91-7743a8b20b78-kube-api-access-4ggn6\") pod \"payload-processing-e2e-ait-f73e00ed-866fdf7694-m4mwj\" (UID: \"55a6bf29-4dc5-4b77-9c91-7743a8b20b78\") " pod="openshift-ingress/payload-processing-e2e-ait-f73e00ed-866fdf7694-m4mwj" Nov 04 02:56:13.722100 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:56:13.722015 4405 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/55a6bf29-4dc5-4b77-9c91-7743a8b20b78-plugins-config-volume\") pod \"payload-processing-e2e-ait-f73e00ed-866fdf7694-m4mwj\" (UID: \"55a6bf29-4dc5-4b77-9c91-7743a8b20b78\") " pod="openshift-ingress/payload-processing-e2e-ait-f73e00ed-866fdf7694-m4mwj" Nov 04 02:56:13.823495 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:56:13.823459 4405 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4ggn6\" (UniqueName: \"kubernetes.io/projected/55a6bf29-4dc5-4b77-9c91-7743a8b20b78-kube-api-access-4ggn6\") pod \"payload-processing-e2e-ait-f73e00ed-866fdf7694-m4mwj\" (UID: \"55a6bf29-4dc5-4b77-9c91-7743a8b20b78\") " pod="openshift-ingress/payload-processing-e2e-ait-f73e00ed-866fdf7694-m4mwj" Nov 04 02:56:13.823735 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:56:13.823621 4405 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/55a6bf29-4dc5-4b77-9c91-7743a8b20b78-plugins-config-volume\") pod \"payload-processing-e2e-ait-f73e00ed-866fdf7694-m4mwj\" (UID: \"55a6bf29-4dc5-4b77-9c91-7743a8b20b78\") " pod="openshift-ingress/payload-processing-e2e-ait-f73e00ed-866fdf7694-m4mwj" Nov 04 02:56:13.824391 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:56:13.824367 4405 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/55a6bf29-4dc5-4b77-9c91-7743a8b20b78-plugins-config-volume\") pod \"payload-processing-e2e-ait-f73e00ed-866fdf7694-m4mwj\" (UID: \"55a6bf29-4dc5-4b77-9c91-7743a8b20b78\") " pod="openshift-ingress/payload-processing-e2e-ait-f73e00ed-866fdf7694-m4mwj" Nov 04 02:56:13.890163 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:56:13.890123 4405 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4ggn6\" (UniqueName: \"kubernetes.io/projected/55a6bf29-4dc5-4b77-9c91-7743a8b20b78-kube-api-access-4ggn6\") pod \"payload-processing-e2e-ait-f73e00ed-866fdf7694-m4mwj\" (UID: \"55a6bf29-4dc5-4b77-9c91-7743a8b20b78\") " pod="openshift-ingress/payload-processing-e2e-ait-f73e00ed-866fdf7694-m4mwj" Nov 04 02:56:14.154382 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:56:14.154347 4405 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-f73e00ed-866fdf7694-m4mwj" Nov 04 02:56:14.537537 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:56:14.537507 4405 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-f73e00ed-866fdf7694-m4mwj"] Nov 04 02:56:14.538753 ip-10-0-134-125 kubenswrapper[4405]: W1104 02:56:14.538722 4405 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod55a6bf29_4dc5_4b77_9c91_7743a8b20b78.slice/crio-663a00c24734e225579dfc7b72bf42eafe0e004c27644d9c0ef9925b7526e6fd WatchSource:0}: Error finding container 663a00c24734e225579dfc7b72bf42eafe0e004c27644d9c0ef9925b7526e6fd: Status 404 returned error can't find the container with id 663a00c24734e225579dfc7b72bf42eafe0e004c27644d9c0ef9925b7526e6fd Nov 04 02:56:14.778184 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:56:14.778147 4405 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-f73e00ed-866fdf7694-m4mwj" event={"ID":"55a6bf29-4dc5-4b77-9c91-7743a8b20b78","Type":"ContainerStarted","Data":"143449c755d7a1fcb08167acd75b1e86836e1ed69029658e4bfb0c98de3ca2d7"} Nov 04 02:56:14.778367 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:56:14.778205 4405 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-f73e00ed-866fdf7694-m4mwj" event={"ID":"55a6bf29-4dc5-4b77-9c91-7743a8b20b78","Type":"ContainerStarted","Data":"663a00c24734e225579dfc7b72bf42eafe0e004c27644d9c0ef9925b7526e6fd"} Nov 04 02:56:14.778367 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:56:14.778225 4405 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ait-f73e00ed-866fdf7694-m4mwj" Nov 04 02:56:14.895774 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:56:14.895712 4405 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ait-f73e00ed-866fdf7694-m4mwj" podStartSLOduration=1.895692297 podStartE2EDuration="1.895692297s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 02:56:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 02:56:14.892239813 +0000 UTC m=+1421.322939935" watchObservedRunningTime="2026-11-04 02:56:14.895692297 +0000 UTC m=+1421.326392419" Nov 04 02:56:21.507638 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:56:21.507598 4405 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-bl5mg"] Nov 04 02:56:24.222471 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:56:24.222424 4405 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-fe4e0c/distinct-2-w0-fe4e0c-kserve-579bf4cc77-8pbq8" podUID="2b2ab722-d1bb-4376-bede-f5f4ee4562aa" containerName="main" probeResult="failure" output="Get \"https://10.134.0.68:8000/ready\": context deadline exceeded" Nov 04 02:56:24.270954 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:56:24.270883 4405 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-a0bcbc/distinct-w2-a0bcbc-kserve-596f94c7cd-szl4g" podUID="bac32934-8a57-4919-b435-13f367cbea3e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 02:56:24.309829 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:56:24.309780 4405 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-203c35/simulator-w4-203c35-kserve-f9c4bd695-999ns" podUID="eb0d8bdd-6808-4fd0-b9b5-567dff3e7ca5" containerName="main" probeResult="failure" output="Get \"https://10.134.0.46:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 02:56:24.399694 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:56:24.396057 4405 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-ljhhw" podUID="f66b1861-d87b-4d57-ab27-f1a24544a534" containerName="main" probeResult="failure" output="Get \"https://10.134.0.26:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 02:56:24.451694 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:56:24.449858 4405 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-ebb7fd/distinct-w6-ebb7fd-kserve-55c47d8865-7z9pz" podUID="ccb8e181-5610-46da-bb8b-e0b07fa02ff1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 02:56:24.455694 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:56:24.452473 4405 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-e516ff/distinct-w1-e516ff-kserve-f4474cb9c-glpzw" podUID="8842ad09-22ff-4492-ba5d-d4cacf4eac75" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 02:56:24.464855 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:56:24.464803 4405 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-fe4e0c/distinct-2-w0-fe4e0c-kserve-579bf4cc77-8pbq8" podUID="2b2ab722-d1bb-4376-bede-f5f4ee4562aa" containerName="main" probeResult="failure" output="Get \"https://10.134.0.68:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 02:56:24.614699 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:56:24.611056 4405 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-a0bcbc/premium-w2-a0bcbc-kserve-78dfd6d78c-lwth2" podUID="33b620c1-389c-48b4-b277-90252018c4b6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 02:56:24.614699 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:56:24.612240 4405 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-ebb7fd/distinct-2-w6-ebb7fd-kserve-6f68c5db86-zrpjs" podUID="b7b786e0-9c94-4126-9949-cd354ebbfb2a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.65:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 04 02:56:24.884753 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:56:24.881239 4405 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-cdcf67/distinct-w3-cdcf67-kserve-667cc7f7d7-5m2lq" podUID="52ed22cc-96e4-4ef0-b34d-ed58c89a9183" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 02:56:25.280752 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:56:25.275892 4405 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-a0bcbc/distinct-w2-a0bcbc-kserve-596f94c7cd-szl4g" podUID="bac32934-8a57-4919-b435-13f367cbea3e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 02:56:25.280752 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:56:25.277493 4405 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-a0bcbc/distinct-w2-a0bcbc-kserve-596f94c7cd-szl4g" Nov 04 02:56:25.306696 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:56:25.286402 4405 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-fe4e0c/distinct-2-w0-fe4e0c-kserve-579bf4cc77-8pbq8" podUID="2b2ab722-d1bb-4376-bede-f5f4ee4562aa" containerName="main" probeResult="failure" output="Get \"https://10.134.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 02:56:25.306696 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:56:25.287611 4405 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-fe4e0c/distinct-2-w0-fe4e0c-kserve-579bf4cc77-8pbq8" Nov 04 02:56:25.332694 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:56:25.315752 4405 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-203c35/simulator-w4-203c35-kserve-f9c4bd695-999ns" podUID="eb0d8bdd-6808-4fd0-b9b5-567dff3e7ca5" containerName="main" probeResult="failure" output="Get \"https://10.134.0.46:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 02:56:25.332694 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:56:25.316839 4405 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-203c35/simulator-w4-203c35-kserve-f9c4bd695-999ns" Nov 04 02:56:25.409703 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:56:25.408490 4405 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-ljhhw" podUID="f66b1861-d87b-4d57-ab27-f1a24544a534" containerName="main" probeResult="failure" output="Get \"https://10.134.0.26:8000/ready\": context deadline exceeded" Nov 04 02:56:25.447029 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:56:25.445476 4405 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-ljhhw" Nov 04 02:56:25.454865 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:56:25.454818 4405 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-ebb7fd/distinct-w6-ebb7fd-kserve-55c47d8865-7z9pz" podUID="ccb8e181-5610-46da-bb8b-e0b07fa02ff1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 02:56:25.577007 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:56:25.576909 4405 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-ebb7fd/distinct-w6-ebb7fd-kserve-55c47d8865-7z9pz" Nov 04 02:56:25.593800 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:56:25.593742 4405 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-e516ff/distinct-w1-e516ff-kserve-f4474cb9c-glpzw" podUID="8842ad09-22ff-4492-ba5d-d4cacf4eac75" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/ready\": context deadline exceeded" Nov 04 02:56:25.622715 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:56:25.611712 4405 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-e516ff/distinct-w1-e516ff-kserve-f4474cb9c-glpzw" Nov 04 02:56:25.622715 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:56:25.613388 4405 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-a0bcbc/premium-w2-a0bcbc-kserve-78dfd6d78c-lwth2" podUID="33b620c1-389c-48b4-b277-90252018c4b6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 02:56:25.622715 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:56:25.613855 4405 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-a0bcbc/premium-w2-a0bcbc-kserve-78dfd6d78c-lwth2" Nov 04 02:56:25.622715 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:56:25.614008 4405 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-ebb7fd/distinct-2-w6-ebb7fd-kserve-6f68c5db86-zrpjs" podUID="b7b786e0-9c94-4126-9949-cd354ebbfb2a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 02:56:25.622715 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:56:25.614250 4405 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-ebb7fd/distinct-2-w6-ebb7fd-kserve-6f68c5db86-zrpjs" Nov 04 02:56:25.851979 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:56:25.851941 4405 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-203c35/simulator-w4-203c35-kserve-f9c4bd695-999ns" podUID="eb0d8bdd-6808-4fd0-b9b5-567dff3e7ca5" containerName="main" probeResult="failure" output="Get \"https://10.134.0.46:8000/health\": context deadline exceeded" Nov 04 02:56:25.896757 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:56:25.893769 4405 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-cdcf67/distinct-w3-cdcf67-kserve-667cc7f7d7-5m2lq" podUID="52ed22cc-96e4-4ef0-b34d-ed58c89a9183" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 02:56:26.163751 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:56:26.147714 4405 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-cdcf67/distinct-w3-cdcf67-kserve-667cc7f7d7-5m2lq" Nov 04 02:56:26.163751 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:56:26.147788 4405 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ait-f73e00ed-866fdf7694-m4mwj" Nov 04 02:56:26.163751 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:56:26.147843 4405 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-fe4e0c/distinct-2-w0-fe4e0c-kserve-579bf4cc77-8pbq8" Nov 04 02:56:26.220047 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:56:26.220017 4405 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-ebb7fd/distinct-w6-ebb7fd-kserve-55c47d8865-7z9pz" Nov 04 02:56:26.265984 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:56:26.265956 4405 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-a0bcbc/distinct-w2-a0bcbc-kserve-596f94c7cd-szl4g" Nov 04 02:56:26.329337 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:56:26.329295 4405 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-203c35/simulator-w4-203c35-kserve-f9c4bd695-999ns" podUID="eb0d8bdd-6808-4fd0-b9b5-567dff3e7ca5" containerName="main" probeResult="failure" output="Get \"https://10.134.0.46:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 02:56:26.353308 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:56:26.351725 4405 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-e516ff/distinct-w1-e516ff-kserve-f4474cb9c-glpzw" podUID="8842ad09-22ff-4492-ba5d-d4cacf4eac75" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 02:56:26.409230 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:56:26.408731 4405 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-ljhhw" podUID="f66b1861-d87b-4d57-ab27-f1a24544a534" containerName="main" probeResult="failure" output="Get \"https://10.134.0.26:8000/ready\": context deadline exceeded" Nov 04 02:56:26.542460 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:56:26.542380 4405 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-ljhhw" Nov 04 02:56:26.569866 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:56:26.569836 4405 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-ebb7fd/distinct-2-w6-ebb7fd-kserve-6f68c5db86-zrpjs" Nov 04 02:56:26.607590 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:56:26.594917 4405 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-e516ff/distinct-w1-e516ff-kserve-f4474cb9c-glpzw" podUID="8842ad09-22ff-4492-ba5d-d4cacf4eac75" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 02:56:26.622902 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:56:26.622844 4405 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-a0bcbc/premium-w2-a0bcbc-kserve-78dfd6d78c-lwth2" podUID="33b620c1-389c-48b4-b277-90252018c4b6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/ready\": context deadline exceeded" Nov 04 02:56:26.639736 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:56:26.637265 4405 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-203c35/simulator-w4-203c35-kserve-f9c4bd695-999ns" Nov 04 02:56:26.671566 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:56:26.671538 4405 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-cdcf67/distinct-w3-cdcf67-kserve-667cc7f7d7-5m2lq" Nov 04 02:56:26.683799 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:56:26.683770 4405 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-e516ff/distinct-w1-e516ff-kserve-f4474cb9c-glpzw" Nov 04 02:56:26.707935 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:56:26.706825 4405 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-a0bcbc/premium-w2-a0bcbc-kserve-78dfd6d78c-lwth2" Nov 04 02:56:33.340700 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:56:33.340649 4405 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-87191e54-84bc674f5-6fjcp"] Nov 04 02:56:33.345280 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:56:33.345260 4405 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-87191e54-84bc674f5-6fjcp" Nov 04 02:56:33.449794 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:56:33.449761 4405 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-87191e54-serving-cert\"" Nov 04 02:56:33.453210 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:56:33.453168 4405 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/29be030d-637b-4256-b2b7-10d9abbf685b-openshift-service-ca\") pod \"maas-api-e2e-ait-87191e54-84bc674f5-6fjcp\" (UID: \"29be030d-637b-4256-b2b7-10d9abbf685b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-87191e54-84bc674f5-6fjcp" Nov 04 02:56:33.453390 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:56:33.453304 4405 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wjmsp\" (UniqueName: \"kubernetes.io/projected/29be030d-637b-4256-b2b7-10d9abbf685b-kube-api-access-wjmsp\") pod \"maas-api-e2e-ait-87191e54-84bc674f5-6fjcp\" (UID: \"29be030d-637b-4256-b2b7-10d9abbf685b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-87191e54-84bc674f5-6fjcp" Nov 04 02:56:33.453465 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:56:33.453392 4405 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/29be030d-637b-4256-b2b7-10d9abbf685b-metrics-certs\") pod \"maas-api-e2e-ait-87191e54-84bc674f5-6fjcp\" (UID: \"29be030d-637b-4256-b2b7-10d9abbf685b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-87191e54-84bc674f5-6fjcp" Nov 04 02:56:33.453530 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:56:33.453498 4405 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/29be030d-637b-4256-b2b7-10d9abbf685b-maas-api-tls\") pod \"maas-api-e2e-ait-87191e54-84bc674f5-6fjcp\" (UID: \"29be030d-637b-4256-b2b7-10d9abbf685b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-87191e54-84bc674f5-6fjcp" Nov 04 02:56:33.454349 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:56:33.454325 4405 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-87191e54-84bc674f5-6fjcp"] Nov 04 02:56:33.554505 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:56:33.554470 4405 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/29be030d-637b-4256-b2b7-10d9abbf685b-openshift-service-ca\") pod \"maas-api-e2e-ait-87191e54-84bc674f5-6fjcp\" (UID: \"29be030d-637b-4256-b2b7-10d9abbf685b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-87191e54-84bc674f5-6fjcp" Nov 04 02:56:33.554685 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:56:33.554600 4405 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wjmsp\" (UniqueName: \"kubernetes.io/projected/29be030d-637b-4256-b2b7-10d9abbf685b-kube-api-access-wjmsp\") pod \"maas-api-e2e-ait-87191e54-84bc674f5-6fjcp\" (UID: \"29be030d-637b-4256-b2b7-10d9abbf685b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-87191e54-84bc674f5-6fjcp" Nov 04 02:56:33.554758 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:56:33.554711 4405 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/29be030d-637b-4256-b2b7-10d9abbf685b-metrics-certs\") pod \"maas-api-e2e-ait-87191e54-84bc674f5-6fjcp\" (UID: \"29be030d-637b-4256-b2b7-10d9abbf685b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-87191e54-84bc674f5-6fjcp" Nov 04 02:56:33.555165 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:56:33.555123 4405 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/29be030d-637b-4256-b2b7-10d9abbf685b-maas-api-tls\") pod \"maas-api-e2e-ait-87191e54-84bc674f5-6fjcp\" (UID: \"29be030d-637b-4256-b2b7-10d9abbf685b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-87191e54-84bc674f5-6fjcp" Nov 04 02:56:33.555304 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:56:33.555266 4405 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/29be030d-637b-4256-b2b7-10d9abbf685b-openshift-service-ca\") pod \"maas-api-e2e-ait-87191e54-84bc674f5-6fjcp\" (UID: \"29be030d-637b-4256-b2b7-10d9abbf685b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-87191e54-84bc674f5-6fjcp" Nov 04 02:56:33.558234 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:56:33.558208 4405 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/29be030d-637b-4256-b2b7-10d9abbf685b-maas-api-tls\") pod \"maas-api-e2e-ait-87191e54-84bc674f5-6fjcp\" (UID: \"29be030d-637b-4256-b2b7-10d9abbf685b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-87191e54-84bc674f5-6fjcp" Nov 04 02:56:33.558440 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:56:33.558418 4405 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/29be030d-637b-4256-b2b7-10d9abbf685b-metrics-certs\") pod \"maas-api-e2e-ait-87191e54-84bc674f5-6fjcp\" (UID: \"29be030d-637b-4256-b2b7-10d9abbf685b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-87191e54-84bc674f5-6fjcp" Nov 04 02:56:33.615798 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:56:33.615723 4405 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wjmsp\" (UniqueName: \"kubernetes.io/projected/29be030d-637b-4256-b2b7-10d9abbf685b-kube-api-access-wjmsp\") pod \"maas-api-e2e-ait-87191e54-84bc674f5-6fjcp\" (UID: \"29be030d-637b-4256-b2b7-10d9abbf685b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-87191e54-84bc674f5-6fjcp" Nov 04 02:56:33.659517 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:56:33.659487 4405 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-87191e54-84bc674f5-6fjcp" Nov 04 02:56:34.052494 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:56:34.052462 4405 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-87191e54-84bc674f5-6fjcp"] Nov 04 02:56:34.052695 ip-10-0-134-125 kubenswrapper[4405]: W1104 02:56:34.052643 4405 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod29be030d_637b_4256_b2b7_10d9abbf685b.slice/crio-9386f5fae4c9eb16e5ba812397f36c46692ed0987651885a77d3859daa7fdbfc WatchSource:0}: Error finding container 9386f5fae4c9eb16e5ba812397f36c46692ed0987651885a77d3859daa7fdbfc: Status 404 returned error can't find the container with id 9386f5fae4c9eb16e5ba812397f36c46692ed0987651885a77d3859daa7fdbfc Nov 04 02:56:34.054223 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:56:34.054203 4405 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 04 02:56:34.407599 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:56:34.407556 4405 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-87191e54-84bc674f5-6fjcp" event={"ID":"29be030d-637b-4256-b2b7-10d9abbf685b","Type":"ContainerStarted","Data":"9386f5fae4c9eb16e5ba812397f36c46692ed0987651885a77d3859daa7fdbfc"} Nov 04 02:56:35.424132 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:56:35.424078 4405 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-87191e54-84bc674f5-6fjcp" event={"ID":"29be030d-637b-4256-b2b7-10d9abbf685b","Type":"ContainerStarted","Data":"3dd60c1988f3dabee4d56e619053954df58960bcc3ae1c7100afeb1a3c942302"} Nov 04 02:56:35.424561 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:56:35.424325 4405 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-87191e54-84bc674f5-6fjcp" Nov 04 02:56:35.545905 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:56:35.545839 4405 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-87191e54-84bc674f5-6fjcp" podStartSLOduration=1.7060068670000001 podStartE2EDuration="2.545818637s" totalImagesPullingTime="839.81177ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 02:56:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 02:56:35.541056129 +0000 UTC m=+1441.971756249" watchObservedRunningTime="2026-11-04 02:56:35.545818637 +0000 UTC m=+1441.976518761" Nov 04 02:56:36.660045 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:56:36.660011 4405 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-76597d6855-7c95j"] Nov 04 02:56:36.664373 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:56:36.664353 4405 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-7c95j" Nov 04 02:56:36.716403 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:56:36.716340 4405 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-pm6wh\"" Nov 04 02:56:36.801447 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:56:36.801397 4405 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-76597d6855-7c95j"] Nov 04 02:56:36.833270 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:56:36.833226 4405 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/76df1601-21fb-40ab-a5ee-433f367453c7-istio-podinfo\") pod \"maas-default-gateway-openshift-default-76597d6855-7c95j\" (UID: \"76df1601-21fb-40ab-a5ee-433f367453c7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-7c95j" Nov 04 02:56:36.833511 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:56:36.833331 4405 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/76df1601-21fb-40ab-a5ee-433f367453c7-workload-socket\") pod \"maas-default-gateway-openshift-default-76597d6855-7c95j\" (UID: \"76df1601-21fb-40ab-a5ee-433f367453c7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-7c95j" Nov 04 02:56:36.833511 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:56:36.833373 4405 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/76df1601-21fb-40ab-a5ee-433f367453c7-workload-certs\") pod \"maas-default-gateway-openshift-default-76597d6855-7c95j\" (UID: \"76df1601-21fb-40ab-a5ee-433f367453c7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-7c95j" Nov 04 02:56:36.833511 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:56:36.833398 4405 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x9ln6\" (UniqueName: \"kubernetes.io/projected/76df1601-21fb-40ab-a5ee-433f367453c7-kube-api-access-x9ln6\") pod \"maas-default-gateway-openshift-default-76597d6855-7c95j\" (UID: \"76df1601-21fb-40ab-a5ee-433f367453c7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-7c95j" Nov 04 02:56:36.833711 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:56:36.833512 4405 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/76df1601-21fb-40ab-a5ee-433f367453c7-istio-data\") pod \"maas-default-gateway-openshift-default-76597d6855-7c95j\" (UID: \"76df1601-21fb-40ab-a5ee-433f367453c7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-7c95j" Nov 04 02:56:36.833711 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:56:36.833546 4405 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/76df1601-21fb-40ab-a5ee-433f367453c7-istio-token\") pod \"maas-default-gateway-openshift-default-76597d6855-7c95j\" (UID: \"76df1601-21fb-40ab-a5ee-433f367453c7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-7c95j" Nov 04 02:56:36.833711 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:56:36.833590 4405 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/76df1601-21fb-40ab-a5ee-433f367453c7-credential-socket\") pod \"maas-default-gateway-openshift-default-76597d6855-7c95j\" (UID: \"76df1601-21fb-40ab-a5ee-433f367453c7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-7c95j" Nov 04 02:56:36.833891 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:56:36.833722 4405 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/76df1601-21fb-40ab-a5ee-433f367453c7-istio-envoy\") pod \"maas-default-gateway-openshift-default-76597d6855-7c95j\" (UID: \"76df1601-21fb-40ab-a5ee-433f367453c7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-7c95j" Nov 04 02:56:36.833891 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:56:36.833874 4405 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/76df1601-21fb-40ab-a5ee-433f367453c7-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-76597d6855-7c95j\" (UID: \"76df1601-21fb-40ab-a5ee-433f367453c7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-7c95j" Nov 04 02:56:36.885473 ip-10-0-134-125 kubenswrapper[4405]: E1104 02:56:36.885441 4405 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5f2vb" containerName="extractor" Nov 04 02:56:36.935548 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:56:36.935457 4405 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/76df1601-21fb-40ab-a5ee-433f367453c7-istio-podinfo\") pod \"maas-default-gateway-openshift-default-76597d6855-7c95j\" (UID: \"76df1601-21fb-40ab-a5ee-433f367453c7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-7c95j" Nov 04 02:56:36.935548 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:56:36.935522 4405 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/76df1601-21fb-40ab-a5ee-433f367453c7-workload-socket\") pod \"maas-default-gateway-openshift-default-76597d6855-7c95j\" (UID: \"76df1601-21fb-40ab-a5ee-433f367453c7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-7c95j" Nov 04 02:56:36.935548 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:56:36.935548 4405 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/76df1601-21fb-40ab-a5ee-433f367453c7-workload-certs\") pod \"maas-default-gateway-openshift-default-76597d6855-7c95j\" (UID: \"76df1601-21fb-40ab-a5ee-433f367453c7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-7c95j" Nov 04 02:56:36.935902 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:56:36.935565 4405 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x9ln6\" (UniqueName: \"kubernetes.io/projected/76df1601-21fb-40ab-a5ee-433f367453c7-kube-api-access-x9ln6\") pod \"maas-default-gateway-openshift-default-76597d6855-7c95j\" (UID: \"76df1601-21fb-40ab-a5ee-433f367453c7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-7c95j" Nov 04 02:56:36.935902 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:56:36.935653 4405 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/76df1601-21fb-40ab-a5ee-433f367453c7-istio-data\") pod \"maas-default-gateway-openshift-default-76597d6855-7c95j\" (UID: \"76df1601-21fb-40ab-a5ee-433f367453c7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-7c95j" Nov 04 02:56:36.935902 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:56:36.935724 4405 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/76df1601-21fb-40ab-a5ee-433f367453c7-istio-token\") pod \"maas-default-gateway-openshift-default-76597d6855-7c95j\" (UID: \"76df1601-21fb-40ab-a5ee-433f367453c7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-7c95j" Nov 04 02:56:36.935902 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:56:36.935769 4405 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/76df1601-21fb-40ab-a5ee-433f367453c7-credential-socket\") pod \"maas-default-gateway-openshift-default-76597d6855-7c95j\" (UID: \"76df1601-21fb-40ab-a5ee-433f367453c7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-7c95j" Nov 04 02:56:36.935902 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:56:36.935870 4405 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/76df1601-21fb-40ab-a5ee-433f367453c7-istio-envoy\") pod \"maas-default-gateway-openshift-default-76597d6855-7c95j\" (UID: \"76df1601-21fb-40ab-a5ee-433f367453c7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-7c95j" Nov 04 02:56:36.936146 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:56:36.935975 4405 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/76df1601-21fb-40ab-a5ee-433f367453c7-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-76597d6855-7c95j\" (UID: \"76df1601-21fb-40ab-a5ee-433f367453c7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-7c95j" Nov 04 02:56:36.936207 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:56:36.936177 4405 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/76df1601-21fb-40ab-a5ee-433f367453c7-workload-socket\") pod \"maas-default-gateway-openshift-default-76597d6855-7c95j\" (UID: \"76df1601-21fb-40ab-a5ee-433f367453c7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-7c95j" Nov 04 02:56:36.936263 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:56:36.936199 4405 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/76df1601-21fb-40ab-a5ee-433f367453c7-workload-certs\") pod \"maas-default-gateway-openshift-default-76597d6855-7c95j\" (UID: \"76df1601-21fb-40ab-a5ee-433f367453c7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-7c95j" Nov 04 02:56:36.936308 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:56:36.936260 4405 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/76df1601-21fb-40ab-a5ee-433f367453c7-credential-socket\") pod \"maas-default-gateway-openshift-default-76597d6855-7c95j\" (UID: \"76df1601-21fb-40ab-a5ee-433f367453c7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-7c95j" Nov 04 02:56:36.936515 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:56:36.936456 4405 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/76df1601-21fb-40ab-a5ee-433f367453c7-istio-data\") pod \"maas-default-gateway-openshift-default-76597d6855-7c95j\" (UID: \"76df1601-21fb-40ab-a5ee-433f367453c7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-7c95j" Nov 04 02:56:36.936828 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:56:36.936804 4405 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/76df1601-21fb-40ab-a5ee-433f367453c7-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-76597d6855-7c95j\" (UID: \"76df1601-21fb-40ab-a5ee-433f367453c7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-7c95j" Nov 04 02:56:36.938544 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:56:36.938519 4405 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/76df1601-21fb-40ab-a5ee-433f367453c7-istio-envoy\") pod \"maas-default-gateway-openshift-default-76597d6855-7c95j\" (UID: \"76df1601-21fb-40ab-a5ee-433f367453c7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-7c95j" Nov 04 02:56:36.939093 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:56:36.939072 4405 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/76df1601-21fb-40ab-a5ee-433f367453c7-istio-podinfo\") pod \"maas-default-gateway-openshift-default-76597d6855-7c95j\" (UID: \"76df1601-21fb-40ab-a5ee-433f367453c7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-7c95j" Nov 04 02:56:36.999121 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:56:36.999090 4405 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x9ln6\" (UniqueName: \"kubernetes.io/projected/76df1601-21fb-40ab-a5ee-433f367453c7-kube-api-access-x9ln6\") pod \"maas-default-gateway-openshift-default-76597d6855-7c95j\" (UID: \"76df1601-21fb-40ab-a5ee-433f367453c7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-7c95j" Nov 04 02:56:36.999287 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:56:36.999100 4405 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/76df1601-21fb-40ab-a5ee-433f367453c7-istio-token\") pod \"maas-default-gateway-openshift-default-76597d6855-7c95j\" (UID: \"76df1601-21fb-40ab-a5ee-433f367453c7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-7c95j" Nov 04 02:56:37.279788 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:56:37.279688 4405 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-7c95j" Nov 04 02:56:37.720336 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:56:37.720292 4405 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-76597d6855-7c95j"] Nov 04 02:56:37.720764 ip-10-0-134-125 kubenswrapper[4405]: W1104 02:56:37.720651 4405 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod76df1601_21fb_40ab_a5ee_433f367453c7.slice/crio-fa12276092eec9b16ff606bf8f73b324ec401a0826e0630c0c9d762ca8c84494 WatchSource:0}: Error finding container fa12276092eec9b16ff606bf8f73b324ec401a0826e0630c0c9d762ca8c84494: Status 404 returned error can't find the container with id fa12276092eec9b16ff606bf8f73b324ec401a0826e0630c0c9d762ca8c84494 Nov 04 02:56:37.724462 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:56:37.724422 4405 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 04 02:56:37.724560 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:56:37.724509 4405 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 04 02:56:37.724560 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:56:37.724542 4405 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 04 02:56:38.451162 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:56:38.451123 4405 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-7c95j" event={"ID":"76df1601-21fb-40ab-a5ee-433f367453c7","Type":"ContainerStarted","Data":"38c9e467cd3ef893c13c74047e5564d7503d03bbaff95daab2bf37842d10b775"} Nov 04 02:56:38.451392 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:56:38.451191 4405 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-7c95j" event={"ID":"76df1601-21fb-40ab-a5ee-433f367453c7","Type":"ContainerStarted","Data":"fa12276092eec9b16ff606bf8f73b324ec401a0826e0630c0c9d762ca8c84494"} Nov 04 02:56:38.662758 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:56:38.662702 4405 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-7c95j" podStartSLOduration=2.662688814 podStartE2EDuration="2.662688814s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 02:56:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 02:56:38.620494467 +0000 UTC m=+1445.051194589" watchObservedRunningTime="2026-11-04 02:56:38.662688814 +0000 UTC m=+1445.093388928" Nov 04 02:56:39.280720 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:56:39.280643 4405 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-7c95j" Nov 04 02:56:39.282439 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:56:39.282412 4405 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76597d6855-7c95j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused" start-of-body= Nov 04 02:56:39.282556 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:56:39.282472 4405 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-7c95j" podUID="76df1601-21fb-40ab-a5ee-433f367453c7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused" Nov 04 02:56:40.280939 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:56:40.280897 4405 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76597d6855-7c95j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused" start-of-body= Nov 04 02:56:40.281335 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:56:40.280960 4405 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-7c95j" podUID="76df1601-21fb-40ab-a5ee-433f367453c7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused" Nov 04 02:56:41.280301 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:56:41.280269 4405 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76597d6855-7c95j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused" start-of-body= Nov 04 02:56:41.280646 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:56:41.280326 4405 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-7c95j" podUID="76df1601-21fb-40ab-a5ee-433f367453c7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused" Nov 04 02:56:41.439335 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:56:41.439303 4405 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-87191e54-84bc674f5-6fjcp" Nov 04 02:56:42.280916 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:56:42.280883 4405 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76597d6855-7c95j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused" start-of-body= Nov 04 02:56:42.281099 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:56:42.280959 4405 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-7c95j" podUID="76df1601-21fb-40ab-a5ee-433f367453c7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused" Nov 04 02:56:43.280260 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:56:43.280224 4405 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76597d6855-7c95j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused" start-of-body= Nov 04 02:56:43.280808 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:56:43.280286 4405 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-7c95j" podUID="76df1601-21fb-40ab-a5ee-433f367453c7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused" Nov 04 02:56:44.280352 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:56:44.280317 4405 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76597d6855-7c95j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused" start-of-body= Nov 04 02:56:44.280809 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:56:44.280377 4405 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-7c95j" podUID="76df1601-21fb-40ab-a5ee-433f367453c7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused" Nov 04 02:56:45.280750 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:56:45.280709 4405 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76597d6855-7c95j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused" start-of-body= Nov 04 02:56:45.281165 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:56:45.280785 4405 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-7c95j" podUID="76df1601-21fb-40ab-a5ee-433f367453c7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused" Nov 04 02:56:46.953463 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:56:46.953425 4405 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-7c95j" Nov 04 02:56:46.954004 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:56:46.953897 4405 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-7c95j" Nov 04 02:56:46.954912 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:56:46.954892 4405 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-7c95j" Nov 04 02:56:51.761079 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:56:51.761042 4405 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-76597d6855-brntt"] Nov 04 02:56:51.767368 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:56:51.767342 4405 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-brntt" Nov 04 02:56:51.812820 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:56:51.812780 4405 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/423b62dd-792f-442c-b1ac-3248072565e8-istio-data\") pod \"maas-default-gateway-openshift-default-76597d6855-brntt\" (UID: \"423b62dd-792f-442c-b1ac-3248072565e8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-brntt" Nov 04 02:56:51.812820 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:56:51.812822 4405 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/423b62dd-792f-442c-b1ac-3248072565e8-workload-certs\") pod \"maas-default-gateway-openshift-default-76597d6855-brntt\" (UID: \"423b62dd-792f-442c-b1ac-3248072565e8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-brntt" Nov 04 02:56:51.813080 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:56:51.812944 4405 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/423b62dd-792f-442c-b1ac-3248072565e8-istio-token\") pod \"maas-default-gateway-openshift-default-76597d6855-brntt\" (UID: \"423b62dd-792f-442c-b1ac-3248072565e8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-brntt" Nov 04 02:56:51.813080 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:56:51.813040 4405 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/423b62dd-792f-442c-b1ac-3248072565e8-credential-socket\") pod \"maas-default-gateway-openshift-default-76597d6855-brntt\" (UID: \"423b62dd-792f-442c-b1ac-3248072565e8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-brntt" Nov 04 02:56:51.813184 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:56:51.813104 4405 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/423b62dd-792f-442c-b1ac-3248072565e8-istio-envoy\") pod \"maas-default-gateway-openshift-default-76597d6855-brntt\" (UID: \"423b62dd-792f-442c-b1ac-3248072565e8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-brntt" Nov 04 02:56:51.813374 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:56:51.813354 4405 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/423b62dd-792f-442c-b1ac-3248072565e8-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-76597d6855-brntt\" (UID: \"423b62dd-792f-442c-b1ac-3248072565e8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-brntt" Nov 04 02:56:51.813539 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:56:51.813510 4405 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/423b62dd-792f-442c-b1ac-3248072565e8-istio-podinfo\") pod \"maas-default-gateway-openshift-default-76597d6855-brntt\" (UID: \"423b62dd-792f-442c-b1ac-3248072565e8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-brntt" Nov 04 02:56:51.813693 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:56:51.813596 4405 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/423b62dd-792f-442c-b1ac-3248072565e8-workload-socket\") pod \"maas-default-gateway-openshift-default-76597d6855-brntt\" (UID: \"423b62dd-792f-442c-b1ac-3248072565e8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-brntt" Nov 04 02:56:51.813770 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:56:51.813710 4405 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lw6pt\" (UniqueName: \"kubernetes.io/projected/423b62dd-792f-442c-b1ac-3248072565e8-kube-api-access-lw6pt\") pod \"maas-default-gateway-openshift-default-76597d6855-brntt\" (UID: \"423b62dd-792f-442c-b1ac-3248072565e8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-brntt" Nov 04 02:56:51.874466 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:56:51.874426 4405 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-76597d6855-brntt"] Nov 04 02:56:51.915629 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:56:51.915593 4405 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/423b62dd-792f-442c-b1ac-3248072565e8-credential-socket\") pod \"maas-default-gateway-openshift-default-76597d6855-brntt\" (UID: \"423b62dd-792f-442c-b1ac-3248072565e8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-brntt" Nov 04 02:56:51.915863 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:56:51.915722 4405 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/423b62dd-792f-442c-b1ac-3248072565e8-istio-envoy\") pod \"maas-default-gateway-openshift-default-76597d6855-brntt\" (UID: \"423b62dd-792f-442c-b1ac-3248072565e8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-brntt" Nov 04 02:56:51.915931 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:56:51.915886 4405 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/423b62dd-792f-442c-b1ac-3248072565e8-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-76597d6855-brntt\" (UID: \"423b62dd-792f-442c-b1ac-3248072565e8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-brntt" Nov 04 02:56:51.915998 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:56:51.915945 4405 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/423b62dd-792f-442c-b1ac-3248072565e8-istio-podinfo\") pod \"maas-default-gateway-openshift-default-76597d6855-brntt\" (UID: \"423b62dd-792f-442c-b1ac-3248072565e8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-brntt" Nov 04 02:56:51.916055 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:56:51.916023 4405 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/423b62dd-792f-442c-b1ac-3248072565e8-workload-socket\") pod \"maas-default-gateway-openshift-default-76597d6855-brntt\" (UID: \"423b62dd-792f-442c-b1ac-3248072565e8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-brntt" Nov 04 02:56:51.916110 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:56:51.916065 4405 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/423b62dd-792f-442c-b1ac-3248072565e8-credential-socket\") pod \"maas-default-gateway-openshift-default-76597d6855-brntt\" (UID: \"423b62dd-792f-442c-b1ac-3248072565e8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-brntt" Nov 04 02:56:51.916110 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:56:51.916084 4405 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lw6pt\" (UniqueName: \"kubernetes.io/projected/423b62dd-792f-442c-b1ac-3248072565e8-kube-api-access-lw6pt\") pod \"maas-default-gateway-openshift-default-76597d6855-brntt\" (UID: \"423b62dd-792f-442c-b1ac-3248072565e8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-brntt" Nov 04 02:56:51.916220 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:56:51.916170 4405 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/423b62dd-792f-442c-b1ac-3248072565e8-istio-data\") pod \"maas-default-gateway-openshift-default-76597d6855-brntt\" (UID: \"423b62dd-792f-442c-b1ac-3248072565e8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-brntt" Nov 04 02:56:51.916220 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:56:51.916202 4405 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/423b62dd-792f-442c-b1ac-3248072565e8-workload-certs\") pod \"maas-default-gateway-openshift-default-76597d6855-brntt\" (UID: \"423b62dd-792f-442c-b1ac-3248072565e8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-brntt" Nov 04 02:56:51.916378 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:56:51.916345 4405 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/423b62dd-792f-442c-b1ac-3248072565e8-istio-token\") pod \"maas-default-gateway-openshift-default-76597d6855-brntt\" (UID: \"423b62dd-792f-442c-b1ac-3248072565e8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-brntt" Nov 04 02:56:51.916914 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:56:51.916885 4405 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/423b62dd-792f-442c-b1ac-3248072565e8-istio-data\") pod \"maas-default-gateway-openshift-default-76597d6855-brntt\" (UID: \"423b62dd-792f-442c-b1ac-3248072565e8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-brntt" Nov 04 02:56:51.917069 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:56:51.917008 4405 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/423b62dd-792f-442c-b1ac-3248072565e8-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-76597d6855-brntt\" (UID: \"423b62dd-792f-442c-b1ac-3248072565e8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-brntt" Nov 04 02:56:51.917266 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:56:51.917240 4405 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/423b62dd-792f-442c-b1ac-3248072565e8-workload-certs\") pod \"maas-default-gateway-openshift-default-76597d6855-brntt\" (UID: \"423b62dd-792f-442c-b1ac-3248072565e8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-brntt" Nov 04 02:56:51.917528 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:56:51.917501 4405 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/423b62dd-792f-442c-b1ac-3248072565e8-workload-socket\") pod \"maas-default-gateway-openshift-default-76597d6855-brntt\" (UID: \"423b62dd-792f-442c-b1ac-3248072565e8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-brntt" Nov 04 02:56:51.918816 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:56:51.918793 4405 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/423b62dd-792f-442c-b1ac-3248072565e8-istio-podinfo\") pod \"maas-default-gateway-openshift-default-76597d6855-brntt\" (UID: \"423b62dd-792f-442c-b1ac-3248072565e8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-brntt" Nov 04 02:56:51.919482 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:56:51.919465 4405 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/423b62dd-792f-442c-b1ac-3248072565e8-istio-envoy\") pod \"maas-default-gateway-openshift-default-76597d6855-brntt\" (UID: \"423b62dd-792f-442c-b1ac-3248072565e8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-brntt" Nov 04 02:56:51.977442 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:56:51.977410 4405 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/423b62dd-792f-442c-b1ac-3248072565e8-istio-token\") pod \"maas-default-gateway-openshift-default-76597d6855-brntt\" (UID: \"423b62dd-792f-442c-b1ac-3248072565e8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-brntt" Nov 04 02:56:51.977628 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:56:51.977607 4405 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lw6pt\" (UniqueName: \"kubernetes.io/projected/423b62dd-792f-442c-b1ac-3248072565e8-kube-api-access-lw6pt\") pod \"maas-default-gateway-openshift-default-76597d6855-brntt\" (UID: \"423b62dd-792f-442c-b1ac-3248072565e8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-brntt" Nov 04 02:56:52.082790 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:56:52.082758 4405 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-brntt" Nov 04 02:56:52.516979 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:56:52.516895 4405 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-76597d6855-brntt"] Nov 04 02:56:52.552617 ip-10-0-134-125 kubenswrapper[4405]: W1104 02:56:52.552581 4405 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod423b62dd_792f_442c_b1ac_3248072565e8.slice/crio-578adc0e4480fcca480ac0029d68baeb2c3e1f719b0330dd5387c9d245860c9d WatchSource:0}: Error finding container 578adc0e4480fcca480ac0029d68baeb2c3e1f719b0330dd5387c9d245860c9d: Status 404 returned error can't find the container with id 578adc0e4480fcca480ac0029d68baeb2c3e1f719b0330dd5387c9d245860c9d Nov 04 02:56:52.555532 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:56:52.555495 4405 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 04 02:56:52.555659 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:56:52.555583 4405 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 04 02:56:52.555659 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:56:52.555629 4405 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 04 02:56:52.562282 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:56:52.562253 4405 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-brntt" event={"ID":"423b62dd-792f-442c-b1ac-3248072565e8","Type":"ContainerStarted","Data":"578adc0e4480fcca480ac0029d68baeb2c3e1f719b0330dd5387c9d245860c9d"} Nov 04 02:56:53.573722 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:56:53.573656 4405 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-brntt" event={"ID":"423b62dd-792f-442c-b1ac-3248072565e8","Type":"ContainerStarted","Data":"e24a32ba8e02e59a6d4901acdaf8b888fd3d35821730e056c2e4aec27ad72fe5"} Nov 04 02:56:53.743775 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:56:53.743723 4405 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-brntt" podStartSLOduration=2.743708256 podStartE2EDuration="2.743708256s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 02:56:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 02:56:53.742710765 +0000 UTC m=+1460.173410886" watchObservedRunningTime="2026-11-04 02:56:53.743708256 +0000 UTC m=+1460.174408439" Nov 04 02:56:54.082939 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:56:54.082901 4405 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-brntt" Nov 04 02:56:54.084532 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:56:54.084505 4405 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76597d6855-brntt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused" start-of-body= Nov 04 02:56:54.084698 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:56:54.084555 4405 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-brntt" podUID="423b62dd-792f-442c-b1ac-3248072565e8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused" Nov 04 02:56:55.083570 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:56:55.083533 4405 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76597d6855-brntt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused" start-of-body= Nov 04 02:56:55.084090 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:56:55.083598 4405 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-brntt" podUID="423b62dd-792f-442c-b1ac-3248072565e8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused" Nov 04 02:56:56.083362 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:56:56.083324 4405 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76597d6855-brntt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused" start-of-body= Nov 04 02:56:56.083525 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:56:56.083394 4405 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-brntt" podUID="423b62dd-792f-442c-b1ac-3248072565e8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused" Nov 04 02:56:57.083190 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:56:57.083155 4405 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76597d6855-brntt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused" start-of-body= Nov 04 02:56:57.083593 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:56:57.083213 4405 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-brntt" podUID="423b62dd-792f-442c-b1ac-3248072565e8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused" Nov 04 02:56:58.083653 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:56:58.083620 4405 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76597d6855-brntt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused" start-of-body= Nov 04 02:56:58.084062 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:56:58.083693 4405 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-brntt" podUID="423b62dd-792f-442c-b1ac-3248072565e8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused" Nov 04 02:56:59.083927 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:56:59.083891 4405 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76597d6855-brntt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused" start-of-body= Nov 04 02:56:59.084339 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:56:59.083957 4405 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-brntt" podUID="423b62dd-792f-442c-b1ac-3248072565e8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused" Nov 04 02:57:00.083255 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:57:00.083213 4405 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76597d6855-brntt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused" start-of-body= Nov 04 02:57:00.083435 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:57:00.083282 4405 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-brntt" podUID="423b62dd-792f-442c-b1ac-3248072565e8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused" Nov 04 02:57:01.722263 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:57:01.722215 4405 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-brntt" Nov 04 02:57:01.722810 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:57:01.722725 4405 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-brntt" Nov 04 02:57:01.724007 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:57:01.723986 4405 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-brntt" Nov 04 02:57:07.068828 ip-10-0-134-125 kubenswrapper[4405]: E1104 02:57:07.068781 4405 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5f2vb" containerName="extractor" Nov 04 02:57:17.869945 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:57:17.869908 4405 generic.go:356] "Generic (PLEG): container finished" podID="65bede99-6a95-4ac2-84a9-2ffdaef6375d" containerID="4ecb3c5f8ee794f0d2eb13e06c3b61c5179f14d4a2be11269ab328d86637081a" exitCode=0 Nov 04 02:57:17.870383 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:57:17.869967 4405 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-87191e54-openshift-default-7c59fcc9cc-6s8gq" event={"ID":"65bede99-6a95-4ac2-84a9-2ffdaef6375d","Type":"ContainerDied","Data":"4ecb3c5f8ee794f0d2eb13e06c3b61c5179f14d4a2be11269ab328d86637081a"} Nov 04 02:57:17.870472 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:57:17.870456 4405 scope.go:122] "RemoveContainer" containerID="4ecb3c5f8ee794f0d2eb13e06c3b61c5179f14d4a2be11269ab328d86637081a" Nov 04 02:57:17.872223 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:57:17.872194 4405 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 04 02:57:17.872295 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:57:17.872264 4405 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 04 02:57:17.872340 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:57:17.872296 4405 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 04 02:57:18.878284 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:57:18.878252 4405 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-87191e54-openshift-default-7c59fcc9cc-6s8gq" event={"ID":"65bede99-6a95-4ac2-84a9-2ffdaef6375d","Type":"ContainerStarted","Data":"c57b122e894a511162cb2c6094ea0fc5181c9e3df5e5f67ee5dde373f5ec9b06"} Nov 04 02:57:19.653421 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:57:19.653390 4405 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-87191e54-openshift-default-7c59fcc9cc-6s8gq" Nov 04 02:57:19.653612 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:57:19.653586 4405 patch_prober.go:29] interesting pod/e2e-ait-87191e54-openshift-default-7c59fcc9cc-6s8gq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused" start-of-body= Nov 04 02:57:19.653658 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:57:19.653640 4405 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-87191e54-openshift-default-7c59fcc9cc-6s8gq" podUID="65bede99-6a95-4ac2-84a9-2ffdaef6375d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused" Nov 04 02:57:20.653210 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:57:20.653178 4405 patch_prober.go:29] interesting pod/e2e-ait-87191e54-openshift-default-7c59fcc9cc-6s8gq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused" start-of-body= Nov 04 02:57:20.653615 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:57:20.653232 4405 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-87191e54-openshift-default-7c59fcc9cc-6s8gq" podUID="65bede99-6a95-4ac2-84a9-2ffdaef6375d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused" Nov 04 02:57:21.653396 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:57:21.653361 4405 patch_prober.go:29] interesting pod/e2e-ait-87191e54-openshift-default-7c59fcc9cc-6s8gq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused" start-of-body= Nov 04 02:57:21.653879 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:57:21.653422 4405 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-87191e54-openshift-default-7c59fcc9cc-6s8gq" podUID="65bede99-6a95-4ac2-84a9-2ffdaef6375d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused" Nov 04 02:57:22.653793 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:57:22.653751 4405 patch_prober.go:29] interesting pod/e2e-ait-87191e54-openshift-default-7c59fcc9cc-6s8gq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused" start-of-body= Nov 04 02:57:22.654208 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:57:22.653803 4405 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-87191e54-openshift-default-7c59fcc9cc-6s8gq" podUID="65bede99-6a95-4ac2-84a9-2ffdaef6375d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused" Nov 04 02:57:23.653215 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:57:23.653181 4405 patch_prober.go:29] interesting pod/e2e-ait-87191e54-openshift-default-7c59fcc9cc-6s8gq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused" start-of-body= Nov 04 02:57:23.653404 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:57:23.653230 4405 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-87191e54-openshift-default-7c59fcc9cc-6s8gq" podUID="65bede99-6a95-4ac2-84a9-2ffdaef6375d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused" Nov 04 02:57:24.659170 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:57:24.659139 4405 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-87191e54-openshift-default-7c59fcc9cc-6s8gq" Nov 04 02:57:24.659622 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:57:24.659603 4405 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-87191e54-openshift-default-7c59fcc9cc-6s8gq" Nov 04 02:57:24.660522 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:57:24.660504 4405 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-87191e54-openshift-default-7c59fcc9cc-6s8gq" Nov 04 02:57:33.868121 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:57:33.868093 4405 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-vwcx8_ffbb2d83-5a93-4a5b-bce3-e40afb8f58ea/ovn-acl-logging/0.log" Nov 04 02:57:33.870427 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:57:33.870406 4405 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-vwcx8_ffbb2d83-5a93-4a5b-bce3-e40afb8f58ea/ovn-acl-logging/0.log" Nov 04 02:57:36.885336 ip-10-0-134-125 kubenswrapper[4405]: E1104 02:57:36.885296 4405 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5f2vb" containerName="extractor" Nov 04 02:57:40.705417 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:57:40.705382 4405 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-bl5mg"] Nov 04 02:57:51.424415 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:57:51.424380 4405 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-87191e54-77b46c874-mvmn9"] Nov 04 02:57:51.547583 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:57:51.547545 4405 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-87191e54-6ffd5f6b7-5qrkx"] Nov 04 02:57:56.424848 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:57:56.424796 4405 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ait-87191e54-77b46c874-mvmn9" podUID="fb036f6d-278f-4627-952b-c285dc65d364" containerName="payload-processing" containerID="cri-o://f3afc32ff2cfb83e3a9c816f9d854f634823d3f06e98b6102f9d3e13f44ed385" gracePeriod=25 Nov 04 02:57:56.548445 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:57:56.548408 4405 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ait-87191e54-6ffd5f6b7-5qrkx" podUID="557eb86e-a5c3-4790-9dde-53619c054583" containerName="payload-pre-processing" containerID="cri-o://e2db79d4b079b0aa8d088543c0adcc425c223516cc4b48e8cb83d3d240f56215" gracePeriod=25 Nov 04 02:57:56.734075 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:57:56.734043 4405 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-87191e54-77b46c874-mvmn9" Nov 04 02:57:56.870544 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:57:56.870508 4405 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-87191e54-6ffd5f6b7-5qrkx" Nov 04 02:57:56.893582 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:57:56.893551 4405 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fb036f6d-278f-4627-952b-c285dc65d364-plugins-config-volume" pod "fb036f6d-278f-4627-952b-c285dc65d364" (UID: "fb036f6d-278f-4627-952b-c285dc65d364"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 02:57:56.893771 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:57:56.893234 4405 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fb036f6d-278f-4627-952b-c285dc65d364-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fb036f6d-278f-4627-952b-c285dc65d364-plugins-config-volume\") pod \"fb036f6d-278f-4627-952b-c285dc65d364\" (UID: \"fb036f6d-278f-4627-952b-c285dc65d364\") " Nov 04 02:57:56.893914 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:57:56.893896 4405 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fb036f6d-278f-4627-952b-c285dc65d364-kube-api-access-kn4vj\" (UniqueName: \"kubernetes.io/projected/fb036f6d-278f-4627-952b-c285dc65d364-kube-api-access-kn4vj\") pod \"fb036f6d-278f-4627-952b-c285dc65d364\" (UID: \"fb036f6d-278f-4627-952b-c285dc65d364\") " Nov 04 02:57:56.894827 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:57:56.894802 4405 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fb036f6d-278f-4627-952b-c285dc65d364-plugins-config-volume\") on node \"ip-10-0-134-125.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 02:57:56.896408 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:57:56.896383 4405 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fb036f6d-278f-4627-952b-c285dc65d364-kube-api-access-kn4vj" pod "fb036f6d-278f-4627-952b-c285dc65d364" (UID: "fb036f6d-278f-4627-952b-c285dc65d364"). InnerVolumeSpecName "kube-api-access-kn4vj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 02:57:56.995500 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:57:56.995410 4405 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/557eb86e-a5c3-4790-9dde-53619c054583-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/557eb86e-a5c3-4790-9dde-53619c054583-plugins-config-volume\") pod \"557eb86e-a5c3-4790-9dde-53619c054583\" (UID: \"557eb86e-a5c3-4790-9dde-53619c054583\") " Nov 04 02:57:56.995715 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:57:56.995684 4405 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/557eb86e-a5c3-4790-9dde-53619c054583-kube-api-access-l2gln\" (UniqueName: \"kubernetes.io/projected/557eb86e-a5c3-4790-9dde-53619c054583-kube-api-access-l2gln\") pod \"557eb86e-a5c3-4790-9dde-53619c054583\" (UID: \"557eb86e-a5c3-4790-9dde-53619c054583\") " Nov 04 02:57:56.996486 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:57:56.996458 4405 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kn4vj\" (UniqueName: \"kubernetes.io/projected/fb036f6d-278f-4627-952b-c285dc65d364-kube-api-access-kn4vj\") on node \"ip-10-0-134-125.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 02:57:56.996628 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:57:56.996606 4405 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/557eb86e-a5c3-4790-9dde-53619c054583-plugins-config-volume" pod "557eb86e-a5c3-4790-9dde-53619c054583" (UID: "557eb86e-a5c3-4790-9dde-53619c054583"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 02:57:56.998769 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:57:56.998743 4405 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/557eb86e-a5c3-4790-9dde-53619c054583-kube-api-access-l2gln" pod "557eb86e-a5c3-4790-9dde-53619c054583" (UID: "557eb86e-a5c3-4790-9dde-53619c054583"). InnerVolumeSpecName "kube-api-access-l2gln". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 02:57:57.097811 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:57:57.097771 4405 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/557eb86e-a5c3-4790-9dde-53619c054583-plugins-config-volume\") on node \"ip-10-0-134-125.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 02:57:57.097811 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:57:57.097799 4405 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l2gln\" (UniqueName: \"kubernetes.io/projected/557eb86e-a5c3-4790-9dde-53619c054583-kube-api-access-l2gln\") on node \"ip-10-0-134-125.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 02:57:57.277849 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:57:57.277752 4405 generic.go:356] "Generic (PLEG): container finished" podID="557eb86e-a5c3-4790-9dde-53619c054583" containerID="e2db79d4b079b0aa8d088543c0adcc425c223516cc4b48e8cb83d3d240f56215" exitCode=0 Nov 04 02:57:57.277849 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:57:57.277816 4405 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-87191e54-6ffd5f6b7-5qrkx" Nov 04 02:57:57.278076 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:57:57.277843 4405 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-87191e54-6ffd5f6b7-5qrkx" event={"ID":"557eb86e-a5c3-4790-9dde-53619c054583","Type":"ContainerDied","Data":"e2db79d4b079b0aa8d088543c0adcc425c223516cc4b48e8cb83d3d240f56215"} Nov 04 02:57:57.278076 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:57:57.277905 4405 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-87191e54-6ffd5f6b7-5qrkx" event={"ID":"557eb86e-a5c3-4790-9dde-53619c054583","Type":"ContainerDied","Data":"46978e9fe92924b42767110e74b5b024b6f56d150649a0c2fcc3929e40639bdb"} Nov 04 02:57:57.278076 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:57:57.277928 4405 scope.go:122] "RemoveContainer" containerID="e2db79d4b079b0aa8d088543c0adcc425c223516cc4b48e8cb83d3d240f56215" Nov 04 02:57:57.279773 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:57:57.279742 4405 generic.go:356] "Generic (PLEG): container finished" podID="fb036f6d-278f-4627-952b-c285dc65d364" containerID="f3afc32ff2cfb83e3a9c816f9d854f634823d3f06e98b6102f9d3e13f44ed385" exitCode=0 Nov 04 02:57:57.279883 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:57:57.279776 4405 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-87191e54-77b46c874-mvmn9" event={"ID":"fb036f6d-278f-4627-952b-c285dc65d364","Type":"ContainerDied","Data":"f3afc32ff2cfb83e3a9c816f9d854f634823d3f06e98b6102f9d3e13f44ed385"} Nov 04 02:57:57.279883 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:57:57.279799 4405 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-87191e54-77b46c874-mvmn9" Nov 04 02:57:57.279883 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:57:57.279825 4405 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-87191e54-77b46c874-mvmn9" event={"ID":"fb036f6d-278f-4627-952b-c285dc65d364","Type":"ContainerDied","Data":"a44c96d047fadf765e4112d953ad9b2e6ade09c8336a65e40013c61a77968956"} Nov 04 02:57:57.289009 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:57:57.288992 4405 scope.go:122] "RemoveContainer" containerID="e2db79d4b079b0aa8d088543c0adcc425c223516cc4b48e8cb83d3d240f56215" Nov 04 02:57:57.289296 ip-10-0-134-125 kubenswrapper[4405]: E1104 02:57:57.289277 4405 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e2db79d4b079b0aa8d088543c0adcc425c223516cc4b48e8cb83d3d240f56215\": container with ID starting with e2db79d4b079b0aa8d088543c0adcc425c223516cc4b48e8cb83d3d240f56215 not found: ID does not exist" containerID="e2db79d4b079b0aa8d088543c0adcc425c223516cc4b48e8cb83d3d240f56215" Nov 04 02:57:57.289380 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:57:57.289304 4405 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e2db79d4b079b0aa8d088543c0adcc425c223516cc4b48e8cb83d3d240f56215"} err="failed to get container status \"e2db79d4b079b0aa8d088543c0adcc425c223516cc4b48e8cb83d3d240f56215\": rpc error: code = NotFound desc = could not find container \"e2db79d4b079b0aa8d088543c0adcc425c223516cc4b48e8cb83d3d240f56215\": container with ID starting with e2db79d4b079b0aa8d088543c0adcc425c223516cc4b48e8cb83d3d240f56215 not found: ID does not exist" Nov 04 02:57:57.289380 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:57:57.289321 4405 scope.go:122] "RemoveContainer" containerID="f3afc32ff2cfb83e3a9c816f9d854f634823d3f06e98b6102f9d3e13f44ed385" Nov 04 02:57:57.299347 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:57:57.299324 4405 scope.go:122] "RemoveContainer" containerID="f3afc32ff2cfb83e3a9c816f9d854f634823d3f06e98b6102f9d3e13f44ed385" Nov 04 02:57:57.299604 ip-10-0-134-125 kubenswrapper[4405]: E1104 02:57:57.299585 4405 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f3afc32ff2cfb83e3a9c816f9d854f634823d3f06e98b6102f9d3e13f44ed385\": container with ID starting with f3afc32ff2cfb83e3a9c816f9d854f634823d3f06e98b6102f9d3e13f44ed385 not found: ID does not exist" containerID="f3afc32ff2cfb83e3a9c816f9d854f634823d3f06e98b6102f9d3e13f44ed385" Nov 04 02:57:57.299704 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:57:57.299611 4405 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f3afc32ff2cfb83e3a9c816f9d854f634823d3f06e98b6102f9d3e13f44ed385"} err="failed to get container status \"f3afc32ff2cfb83e3a9c816f9d854f634823d3f06e98b6102f9d3e13f44ed385\": rpc error: code = NotFound desc = could not find container \"f3afc32ff2cfb83e3a9c816f9d854f634823d3f06e98b6102f9d3e13f44ed385\": container with ID starting with f3afc32ff2cfb83e3a9c816f9d854f634823d3f06e98b6102f9d3e13f44ed385 not found: ID does not exist" Nov 04 02:57:57.446119 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:57:57.446091 4405 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-87191e54-6ffd5f6b7-5qrkx"] Nov 04 02:57:57.493303 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:57:57.493268 4405 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-87191e54-6ffd5f6b7-5qrkx"] Nov 04 02:57:57.602518 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:57:57.602475 4405 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-87191e54-77b46c874-mvmn9"] Nov 04 02:57:57.606455 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:57:57.606434 4405 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-87191e54-77b46c874-mvmn9"] Nov 04 02:57:57.703237 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:57:57.703206 4405 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="557eb86e-a5c3-4790-9dde-53619c054583" path="/var/lib/kubelet/pods/557eb86e-a5c3-4790-9dde-53619c054583/volumes" Nov 04 02:57:57.703695 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:57:57.703662 4405 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fb036f6d-278f-4627-952b-c285dc65d364" path="/var/lib/kubelet/pods/fb036f6d-278f-4627-952b-c285dc65d364/volumes" Nov 04 02:58:06.884318 ip-10-0-134-125 kubenswrapper[4405]: E1104 02:58:06.884279 4405 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5f2vb" containerName="extractor" Nov 04 02:58:20.047695 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:58:20.046652 4405 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-f73e00ed-866fdf7694-m4mwj"] Nov 04 02:58:20.784484 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:58:20.784444 4405 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-87191e54-84bc674f5-6fjcp"] Nov 04 02:58:20.784777 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:58:20.784747 4405 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-87191e54-84bc674f5-6fjcp" podUID="29be030d-637b-4256-b2b7-10d9abbf685b" containerName="maas-api" containerID="cri-o://3dd60c1988f3dabee4d56e619053954df58960bcc3ae1c7100afeb1a3c942302" gracePeriod=30 Nov 04 02:58:21.153614 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:58:21.153584 4405 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-87191e54-84bc674f5-6fjcp" Nov 04 02:58:21.267986 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:58:21.267942 4405 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/29be030d-637b-4256-b2b7-10d9abbf685b-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/29be030d-637b-4256-b2b7-10d9abbf685b-maas-api-tls\") pod \"29be030d-637b-4256-b2b7-10d9abbf685b\" (UID: \"29be030d-637b-4256-b2b7-10d9abbf685b\") " Nov 04 02:58:21.268818 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:58:21.268781 4405 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/29be030d-637b-4256-b2b7-10d9abbf685b-kube-api-access-wjmsp\" (UniqueName: \"kubernetes.io/projected/29be030d-637b-4256-b2b7-10d9abbf685b-kube-api-access-wjmsp\") pod \"29be030d-637b-4256-b2b7-10d9abbf685b\" (UID: \"29be030d-637b-4256-b2b7-10d9abbf685b\") " Nov 04 02:58:21.269403 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:58:21.269383 4405 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/29be030d-637b-4256-b2b7-10d9abbf685b-metrics-certs\" (UniqueName: \"kubernetes.io/secret/29be030d-637b-4256-b2b7-10d9abbf685b-metrics-certs\") pod \"29be030d-637b-4256-b2b7-10d9abbf685b\" (UID: \"29be030d-637b-4256-b2b7-10d9abbf685b\") " Nov 04 02:58:21.269811 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:58:21.269789 4405 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/29be030d-637b-4256-b2b7-10d9abbf685b-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/29be030d-637b-4256-b2b7-10d9abbf685b-openshift-service-ca\") pod \"29be030d-637b-4256-b2b7-10d9abbf685b\" (UID: \"29be030d-637b-4256-b2b7-10d9abbf685b\") " Nov 04 02:58:21.270277 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:58:21.270250 4405 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/29be030d-637b-4256-b2b7-10d9abbf685b-openshift-service-ca" pod "29be030d-637b-4256-b2b7-10d9abbf685b" (UID: "29be030d-637b-4256-b2b7-10d9abbf685b"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 02:58:21.280053 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:58:21.275194 4405 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/29be030d-637b-4256-b2b7-10d9abbf685b-openshift-service-ca\") on node \"ip-10-0-134-125.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 02:58:21.318986 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:58:21.291146 4405 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/29be030d-637b-4256-b2b7-10d9abbf685b-metrics-certs" pod "29be030d-637b-4256-b2b7-10d9abbf685b" (UID: "29be030d-637b-4256-b2b7-10d9abbf685b"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 04 02:58:21.318986 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:58:21.292585 4405 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/29be030d-637b-4256-b2b7-10d9abbf685b-maas-api-tls" pod "29be030d-637b-4256-b2b7-10d9abbf685b" (UID: "29be030d-637b-4256-b2b7-10d9abbf685b"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 04 02:58:21.318986 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:58:21.293507 4405 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/29be030d-637b-4256-b2b7-10d9abbf685b-kube-api-access-wjmsp" pod "29be030d-637b-4256-b2b7-10d9abbf685b" (UID: "29be030d-637b-4256-b2b7-10d9abbf685b"). InnerVolumeSpecName "kube-api-access-wjmsp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 02:58:21.377278 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:58:21.377154 4405 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/29be030d-637b-4256-b2b7-10d9abbf685b-maas-api-tls\") on node \"ip-10-0-134-125.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 02:58:21.377278 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:58:21.377202 4405 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wjmsp\" (UniqueName: \"kubernetes.io/projected/29be030d-637b-4256-b2b7-10d9abbf685b-kube-api-access-wjmsp\") on node \"ip-10-0-134-125.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 02:58:21.377278 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:58:21.377214 4405 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/29be030d-637b-4256-b2b7-10d9abbf685b-metrics-certs\") on node \"ip-10-0-134-125.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 02:58:21.516690 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:58:21.516482 4405 generic.go:356] "Generic (PLEG): container finished" podID="29be030d-637b-4256-b2b7-10d9abbf685b" containerID="3dd60c1988f3dabee4d56e619053954df58960bcc3ae1c7100afeb1a3c942302" exitCode=0 Nov 04 02:58:21.516888 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:58:21.516707 4405 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-87191e54-84bc674f5-6fjcp" event={"ID":"29be030d-637b-4256-b2b7-10d9abbf685b","Type":"ContainerDied","Data":"3dd60c1988f3dabee4d56e619053954df58960bcc3ae1c7100afeb1a3c942302"} Nov 04 02:58:21.516888 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:58:21.516839 4405 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-87191e54-84bc674f5-6fjcp" event={"ID":"29be030d-637b-4256-b2b7-10d9abbf685b","Type":"ContainerDied","Data":"9386f5fae4c9eb16e5ba812397f36c46692ed0987651885a77d3859daa7fdbfc"} Nov 04 02:58:21.516888 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:58:21.516873 4405 scope.go:122] "RemoveContainer" containerID="3dd60c1988f3dabee4d56e619053954df58960bcc3ae1c7100afeb1a3c942302" Nov 04 02:58:21.517052 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:58:21.517033 4405 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-87191e54-84bc674f5-6fjcp" Nov 04 02:58:21.557639 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:58:21.557605 4405 scope.go:122] "RemoveContainer" containerID="3dd60c1988f3dabee4d56e619053954df58960bcc3ae1c7100afeb1a3c942302" Nov 04 02:58:21.558095 ip-10-0-134-125 kubenswrapper[4405]: E1104 02:58:21.558069 4405 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3dd60c1988f3dabee4d56e619053954df58960bcc3ae1c7100afeb1a3c942302\": container with ID starting with 3dd60c1988f3dabee4d56e619053954df58960bcc3ae1c7100afeb1a3c942302 not found: ID does not exist" containerID="3dd60c1988f3dabee4d56e619053954df58960bcc3ae1c7100afeb1a3c942302" Nov 04 02:58:21.558207 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:58:21.558104 4405 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3dd60c1988f3dabee4d56e619053954df58960bcc3ae1c7100afeb1a3c942302"} err="failed to get container status \"3dd60c1988f3dabee4d56e619053954df58960bcc3ae1c7100afeb1a3c942302\": rpc error: code = NotFound desc = could not find container \"3dd60c1988f3dabee4d56e619053954df58960bcc3ae1c7100afeb1a3c942302\": container with ID starting with 3dd60c1988f3dabee4d56e619053954df58960bcc3ae1c7100afeb1a3c942302 not found: ID does not exist" Nov 04 02:58:21.727752 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:58:21.722975 4405 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-87191e54-84bc674f5-6fjcp"] Nov 04 02:58:21.749456 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:58:21.749413 4405 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-87191e54-84bc674f5-6fjcp"] Nov 04 02:58:22.547421 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:58:22.547387 4405 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-76597d6855-jjxxp"] Nov 04 02:58:22.548322 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:58:22.548300 4405 cpu_manager.go:414] "containerMap: removing container" podUID="29be030d-637b-4256-b2b7-10d9abbf685b" containerName="maas-api" Nov 04 02:58:22.548432 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:58:22.548325 4405 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="29be030d-637b-4256-b2b7-10d9abbf685b" containerName="maas-api" Nov 04 02:58:22.548432 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:58:22.548377 4405 cpu_manager.go:414] "containerMap: removing container" podUID="fb036f6d-278f-4627-952b-c285dc65d364" containerName="payload-processing" Nov 04 02:58:22.548432 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:58:22.548385 4405 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fb036f6d-278f-4627-952b-c285dc65d364" containerName="payload-processing" Nov 04 02:58:22.548432 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:58:22.548413 4405 cpu_manager.go:414] "containerMap: removing container" podUID="557eb86e-a5c3-4790-9dde-53619c054583" containerName="payload-pre-processing" Nov 04 02:58:22.548432 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:58:22.548422 4405 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="557eb86e-a5c3-4790-9dde-53619c054583" containerName="payload-pre-processing" Nov 04 02:58:22.548714 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:58:22.548605 4405 memory_manager.go:365] "RemoveStaleState removing state" podUID="29be030d-637b-4256-b2b7-10d9abbf685b" containerName="maas-api" Nov 04 02:58:22.548714 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:58:22.548627 4405 memory_manager.go:365] "RemoveStaleState removing state" podUID="fb036f6d-278f-4627-952b-c285dc65d364" containerName="payload-processing" Nov 04 02:58:22.548714 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:58:22.548651 4405 memory_manager.go:365] "RemoveStaleState removing state" podUID="557eb86e-a5c3-4790-9dde-53619c054583" containerName="payload-pre-processing" Nov 04 02:58:22.553516 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:58:22.553493 4405 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-jjxxp" Nov 04 02:58:22.609830 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:58:22.609798 4405 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-76597d6855-jjxxp"] Nov 04 02:58:22.617712 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:58:22.617497 4405 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/121fbac1-908f-4c37-8efa-f17d449ed6b9-workload-socket\") pod \"maas-default-gateway-openshift-default-76597d6855-jjxxp\" (UID: \"121fbac1-908f-4c37-8efa-f17d449ed6b9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-jjxxp" Nov 04 02:58:22.617712 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:58:22.617562 4405 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f5zkl\" (UniqueName: \"kubernetes.io/projected/121fbac1-908f-4c37-8efa-f17d449ed6b9-kube-api-access-f5zkl\") pod \"maas-default-gateway-openshift-default-76597d6855-jjxxp\" (UID: \"121fbac1-908f-4c37-8efa-f17d449ed6b9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-jjxxp" Nov 04 02:58:22.617712 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:58:22.617650 4405 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/121fbac1-908f-4c37-8efa-f17d449ed6b9-istio-token\") pod \"maas-default-gateway-openshift-default-76597d6855-jjxxp\" (UID: \"121fbac1-908f-4c37-8efa-f17d449ed6b9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-jjxxp" Nov 04 02:58:22.618006 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:58:22.617728 4405 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/121fbac1-908f-4c37-8efa-f17d449ed6b9-istio-envoy\") pod \"maas-default-gateway-openshift-default-76597d6855-jjxxp\" (UID: \"121fbac1-908f-4c37-8efa-f17d449ed6b9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-jjxxp" Nov 04 02:58:22.618006 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:58:22.617752 4405 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/121fbac1-908f-4c37-8efa-f17d449ed6b9-istio-podinfo\") pod \"maas-default-gateway-openshift-default-76597d6855-jjxxp\" (UID: \"121fbac1-908f-4c37-8efa-f17d449ed6b9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-jjxxp" Nov 04 02:58:22.618006 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:58:22.617821 4405 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/121fbac1-908f-4c37-8efa-f17d449ed6b9-credential-socket\") pod \"maas-default-gateway-openshift-default-76597d6855-jjxxp\" (UID: \"121fbac1-908f-4c37-8efa-f17d449ed6b9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-jjxxp" Nov 04 02:58:22.618006 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:58:22.617874 4405 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/121fbac1-908f-4c37-8efa-f17d449ed6b9-workload-certs\") pod \"maas-default-gateway-openshift-default-76597d6855-jjxxp\" (UID: \"121fbac1-908f-4c37-8efa-f17d449ed6b9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-jjxxp" Nov 04 02:58:22.618006 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:58:22.617901 4405 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/121fbac1-908f-4c37-8efa-f17d449ed6b9-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-76597d6855-jjxxp\" (UID: \"121fbac1-908f-4c37-8efa-f17d449ed6b9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-jjxxp" Nov 04 02:58:22.618006 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:58:22.617965 4405 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/121fbac1-908f-4c37-8efa-f17d449ed6b9-istio-data\") pod \"maas-default-gateway-openshift-default-76597d6855-jjxxp\" (UID: \"121fbac1-908f-4c37-8efa-f17d449ed6b9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-jjxxp" Nov 04 02:58:22.719442 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:58:22.719399 4405 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/121fbac1-908f-4c37-8efa-f17d449ed6b9-istio-token\") pod \"maas-default-gateway-openshift-default-76597d6855-jjxxp\" (UID: \"121fbac1-908f-4c37-8efa-f17d449ed6b9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-jjxxp" Nov 04 02:58:22.719645 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:58:22.719482 4405 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/121fbac1-908f-4c37-8efa-f17d449ed6b9-istio-envoy\") pod \"maas-default-gateway-openshift-default-76597d6855-jjxxp\" (UID: \"121fbac1-908f-4c37-8efa-f17d449ed6b9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-jjxxp" Nov 04 02:58:22.719645 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:58:22.719516 4405 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/121fbac1-908f-4c37-8efa-f17d449ed6b9-istio-podinfo\") pod \"maas-default-gateway-openshift-default-76597d6855-jjxxp\" (UID: \"121fbac1-908f-4c37-8efa-f17d449ed6b9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-jjxxp" Nov 04 02:58:22.719645 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:58:22.719588 4405 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/121fbac1-908f-4c37-8efa-f17d449ed6b9-credential-socket\") pod \"maas-default-gateway-openshift-default-76597d6855-jjxxp\" (UID: \"121fbac1-908f-4c37-8efa-f17d449ed6b9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-jjxxp" Nov 04 02:58:22.719645 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:58:22.719642 4405 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/121fbac1-908f-4c37-8efa-f17d449ed6b9-workload-certs\") pod \"maas-default-gateway-openshift-default-76597d6855-jjxxp\" (UID: \"121fbac1-908f-4c37-8efa-f17d449ed6b9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-jjxxp" Nov 04 02:58:22.719918 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:58:22.719706 4405 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/121fbac1-908f-4c37-8efa-f17d449ed6b9-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-76597d6855-jjxxp\" (UID: \"121fbac1-908f-4c37-8efa-f17d449ed6b9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-jjxxp" Nov 04 02:58:22.719918 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:58:22.719775 4405 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/121fbac1-908f-4c37-8efa-f17d449ed6b9-istio-data\") pod \"maas-default-gateway-openshift-default-76597d6855-jjxxp\" (UID: \"121fbac1-908f-4c37-8efa-f17d449ed6b9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-jjxxp" Nov 04 02:58:22.720028 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:58:22.720010 4405 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/121fbac1-908f-4c37-8efa-f17d449ed6b9-workload-socket\") pod \"maas-default-gateway-openshift-default-76597d6855-jjxxp\" (UID: \"121fbac1-908f-4c37-8efa-f17d449ed6b9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-jjxxp" Nov 04 02:58:22.720087 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:58:22.720063 4405 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f5zkl\" (UniqueName: \"kubernetes.io/projected/121fbac1-908f-4c37-8efa-f17d449ed6b9-kube-api-access-f5zkl\") pod \"maas-default-gateway-openshift-default-76597d6855-jjxxp\" (UID: \"121fbac1-908f-4c37-8efa-f17d449ed6b9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-jjxxp" Nov 04 02:58:22.720942 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:58:22.720913 4405 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/121fbac1-908f-4c37-8efa-f17d449ed6b9-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-76597d6855-jjxxp\" (UID: \"121fbac1-908f-4c37-8efa-f17d449ed6b9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-jjxxp" Nov 04 02:58:22.721090 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:58:22.721066 4405 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/121fbac1-908f-4c37-8efa-f17d449ed6b9-istio-data\") pod \"maas-default-gateway-openshift-default-76597d6855-jjxxp\" (UID: \"121fbac1-908f-4c37-8efa-f17d449ed6b9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-jjxxp" Nov 04 02:58:22.721500 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:58:22.721455 4405 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/121fbac1-908f-4c37-8efa-f17d449ed6b9-credential-socket\") pod \"maas-default-gateway-openshift-default-76597d6855-jjxxp\" (UID: \"121fbac1-908f-4c37-8efa-f17d449ed6b9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-jjxxp" Nov 04 02:58:22.721500 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:58:22.721478 4405 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/121fbac1-908f-4c37-8efa-f17d449ed6b9-workload-certs\") pod \"maas-default-gateway-openshift-default-76597d6855-jjxxp\" (UID: \"121fbac1-908f-4c37-8efa-f17d449ed6b9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-jjxxp" Nov 04 02:58:22.721997 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:58:22.721975 4405 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/121fbac1-908f-4c37-8efa-f17d449ed6b9-workload-socket\") pod \"maas-default-gateway-openshift-default-76597d6855-jjxxp\" (UID: \"121fbac1-908f-4c37-8efa-f17d449ed6b9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-jjxxp" Nov 04 02:58:22.723068 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:58:22.723033 4405 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/121fbac1-908f-4c37-8efa-f17d449ed6b9-istio-podinfo\") pod \"maas-default-gateway-openshift-default-76597d6855-jjxxp\" (UID: \"121fbac1-908f-4c37-8efa-f17d449ed6b9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-jjxxp" Nov 04 02:58:22.723646 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:58:22.723626 4405 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/121fbac1-908f-4c37-8efa-f17d449ed6b9-istio-envoy\") pod \"maas-default-gateway-openshift-default-76597d6855-jjxxp\" (UID: \"121fbac1-908f-4c37-8efa-f17d449ed6b9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-jjxxp" Nov 04 02:58:22.782301 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:58:22.782259 4405 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/121fbac1-908f-4c37-8efa-f17d449ed6b9-istio-token\") pod \"maas-default-gateway-openshift-default-76597d6855-jjxxp\" (UID: \"121fbac1-908f-4c37-8efa-f17d449ed6b9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-jjxxp" Nov 04 02:58:22.782453 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:58:22.782325 4405 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f5zkl\" (UniqueName: \"kubernetes.io/projected/121fbac1-908f-4c37-8efa-f17d449ed6b9-kube-api-access-f5zkl\") pod \"maas-default-gateway-openshift-default-76597d6855-jjxxp\" (UID: \"121fbac1-908f-4c37-8efa-f17d449ed6b9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-jjxxp" Nov 04 02:58:22.869064 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:58:22.869029 4405 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-jjxxp" Nov 04 02:58:23.253021 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:58:23.250492 4405 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-87191e54-openshift-default-7c59fcc9cc-6s8gq"] Nov 04 02:58:23.253021 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:58:23.250768 4405 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-87191e54-openshift-default-7c59fcc9cc-6s8gq" podUID="65bede99-6a95-4ac2-84a9-2ffdaef6375d" containerName="istio-proxy" containerID="cri-o://c57b122e894a511162cb2c6094ea0fc5181c9e3df5e5f67ee5dde373f5ec9b06" gracePeriod=30 Nov 04 02:58:23.356572 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:58:23.355972 4405 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 04 02:58:23.356572 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:58:23.356042 4405 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 04 02:58:23.356572 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:58:23.356071 4405 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 04 02:58:23.360333 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:58:23.360301 4405 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-76597d6855-jjxxp"] Nov 04 02:58:23.571335 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:58:23.571292 4405 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-jjxxp" event={"ID":"121fbac1-908f-4c37-8efa-f17d449ed6b9","Type":"ContainerStarted","Data":"a2a9f16593a0b563618f94c943923072e9f1c5a1d145cd8c5735c9974dba0cc8"} Nov 04 02:58:23.571763 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:58:23.571353 4405 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-jjxxp" event={"ID":"121fbac1-908f-4c37-8efa-f17d449ed6b9","Type":"ContainerStarted","Data":"54e4270711abffd3cd536a7336bc85a507d4d19ee8cd641a890d3c2ab167d1e3"} Nov 04 02:58:23.712934 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:58:23.712874 4405 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-jjxxp" podStartSLOduration=1.712856134 podStartE2EDuration="1.712856134s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 02:58:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 02:58:23.709968817 +0000 UTC m=+1550.140668937" watchObservedRunningTime="2026-11-04 02:58:23.712856134 +0000 UTC m=+1550.143556255" Nov 04 02:58:23.715977 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:58:23.715953 4405 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="29be030d-637b-4256-b2b7-10d9abbf685b" path="/var/lib/kubelet/pods/29be030d-637b-4256-b2b7-10d9abbf685b/volumes" Nov 04 02:58:23.869218 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:58:23.869180 4405 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-jjxxp" Nov 04 02:58:24.677970 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:58:24.677932 4405 patch_prober.go:29] interesting pod/e2e-ait-87191e54-openshift-default-7c59fcc9cc-6s8gq container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 04 02:58:24.678538 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:58:24.678510 4405 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ait-87191e54-openshift-default-7c59fcc9cc-6s8gq" podUID="65bede99-6a95-4ac2-84a9-2ffdaef6375d" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 04 02:58:24.871807 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:58:24.871772 4405 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76597d6855-jjxxp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.83:15021/healthz/ready\": dial tcp 10.134.0.83:15021: connect: connection refused" start-of-body= Nov 04 02:58:24.871987 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:58:24.871827 4405 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-jjxxp" podUID="121fbac1-908f-4c37-8efa-f17d449ed6b9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.83:15021/healthz/ready\": dial tcp 10.134.0.83:15021: connect: connection refused" Nov 04 02:58:25.047217 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:58:25.047126 4405 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ait-f73e00ed-866fdf7694-m4mwj" podUID="55a6bf29-4dc5-4b77-9c91-7743a8b20b78" containerName="payload-processing" containerID="cri-o://143449c755d7a1fcb08167acd75b1e86836e1ed69029658e4bfb0c98de3ca2d7" gracePeriod=25 Nov 04 02:58:25.579241 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:58:25.579208 4405 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-f73e00ed-866fdf7694-m4mwj" Nov 04 02:58:25.767274 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:58:25.767185 4405 generic.go:356] "Generic (PLEG): container finished" podID="55a6bf29-4dc5-4b77-9c91-7743a8b20b78" containerID="143449c755d7a1fcb08167acd75b1e86836e1ed69029658e4bfb0c98de3ca2d7" exitCode=0 Nov 04 02:58:25.767801 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:58:25.767577 4405 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-f73e00ed-866fdf7694-m4mwj" Nov 04 02:58:25.770318 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:58:25.769134 4405 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/55a6bf29-4dc5-4b77-9c91-7743a8b20b78-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/55a6bf29-4dc5-4b77-9c91-7743a8b20b78-plugins-config-volume\") pod \"55a6bf29-4dc5-4b77-9c91-7743a8b20b78\" (UID: \"55a6bf29-4dc5-4b77-9c91-7743a8b20b78\") " Nov 04 02:58:25.770318 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:58:25.769490 4405 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/55a6bf29-4dc5-4b77-9c91-7743a8b20b78-kube-api-access-4ggn6\" (UniqueName: \"kubernetes.io/projected/55a6bf29-4dc5-4b77-9c91-7743a8b20b78-kube-api-access-4ggn6\") pod \"55a6bf29-4dc5-4b77-9c91-7743a8b20b78\" (UID: \"55a6bf29-4dc5-4b77-9c91-7743a8b20b78\") " Nov 04 02:58:25.772568 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:58:25.772530 4405 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/55a6bf29-4dc5-4b77-9c91-7743a8b20b78-plugins-config-volume" pod "55a6bf29-4dc5-4b77-9c91-7743a8b20b78" (UID: "55a6bf29-4dc5-4b77-9c91-7743a8b20b78"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 02:58:25.872699 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:58:25.854877 4405 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/55a6bf29-4dc5-4b77-9c91-7743a8b20b78-kube-api-access-4ggn6" pod "55a6bf29-4dc5-4b77-9c91-7743a8b20b78" (UID: "55a6bf29-4dc5-4b77-9c91-7743a8b20b78"). InnerVolumeSpecName "kube-api-access-4ggn6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 02:58:25.872699 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:58:25.856525 4405 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-4d5a0200-5959b9699-k2dpp"] Nov 04 02:58:25.872699 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:58:25.868745 4405 cpu_manager.go:414] "containerMap: removing container" podUID="55a6bf29-4dc5-4b77-9c91-7743a8b20b78" containerName="payload-processing" Nov 04 02:58:25.872699 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:58:25.868773 4405 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="55a6bf29-4dc5-4b77-9c91-7743a8b20b78" containerName="payload-processing" Nov 04 02:58:25.872699 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:58:25.869007 4405 memory_manager.go:365] "RemoveStaleState removing state" podUID="55a6bf29-4dc5-4b77-9c91-7743a8b20b78" containerName="payload-processing" Nov 04 02:58:25.872699 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:58:25.872366 4405 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4ggn6\" (UniqueName: \"kubernetes.io/projected/55a6bf29-4dc5-4b77-9c91-7743a8b20b78-kube-api-access-4ggn6\") on node \"ip-10-0-134-125.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 02:58:25.872699 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:58:25.872394 4405 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/55a6bf29-4dc5-4b77-9c91-7743a8b20b78-plugins-config-volume\") on node \"ip-10-0-134-125.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 02:58:25.873356 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:58:25.872780 4405 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76597d6855-jjxxp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.83:15021/healthz/ready\": dial tcp 10.134.0.83:15021: connect: connection refused" start-of-body= Nov 04 02:58:25.873356 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:58:25.872827 4405 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-jjxxp" podUID="121fbac1-908f-4c37-8efa-f17d449ed6b9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.83:15021/healthz/ready\": dial tcp 10.134.0.83:15021: connect: connection refused" Nov 04 02:58:25.873356 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:58:25.873243 4405 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-4d5a0200-5959b9699-k2dpp"] Nov 04 02:58:25.873356 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:58:25.873286 4405 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-f73e00ed-866fdf7694-m4mwj" event={"ID":"55a6bf29-4dc5-4b77-9c91-7743a8b20b78","Type":"ContainerDied","Data":"143449c755d7a1fcb08167acd75b1e86836e1ed69029658e4bfb0c98de3ca2d7"} Nov 04 02:58:25.873356 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:58:25.873321 4405 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-f73e00ed-866fdf7694-m4mwj" event={"ID":"55a6bf29-4dc5-4b77-9c91-7743a8b20b78","Type":"ContainerDied","Data":"663a00c24734e225579dfc7b72bf42eafe0e004c27644d9c0ef9925b7526e6fd"} Nov 04 02:58:25.873356 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:58:25.873353 4405 scope.go:122] "RemoveContainer" containerID="143449c755d7a1fcb08167acd75b1e86836e1ed69029658e4bfb0c98de3ca2d7" Nov 04 02:58:25.873657 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:58:25.873415 4405 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-4d5a0200-5959b9699-k2dpp" Nov 04 02:58:25.922314 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:58:25.922285 4405 scope.go:122] "RemoveContainer" containerID="143449c755d7a1fcb08167acd75b1e86836e1ed69029658e4bfb0c98de3ca2d7" Nov 04 02:58:25.934178 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:58:25.934140 4405 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-4d5a0200\"" Nov 04 02:58:25.945105 ip-10-0-134-125 kubenswrapper[4405]: E1104 02:58:25.945058 4405 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"143449c755d7a1fcb08167acd75b1e86836e1ed69029658e4bfb0c98de3ca2d7\": container with ID starting with 143449c755d7a1fcb08167acd75b1e86836e1ed69029658e4bfb0c98de3ca2d7 not found: ID does not exist" containerID="143449c755d7a1fcb08167acd75b1e86836e1ed69029658e4bfb0c98de3ca2d7" Nov 04 02:58:25.945302 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:58:25.945113 4405 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"143449c755d7a1fcb08167acd75b1e86836e1ed69029658e4bfb0c98de3ca2d7"} err="failed to get container status \"143449c755d7a1fcb08167acd75b1e86836e1ed69029658e4bfb0c98de3ca2d7\": rpc error: code = NotFound desc = could not find container \"143449c755d7a1fcb08167acd75b1e86836e1ed69029658e4bfb0c98de3ca2d7\": container with ID starting with 143449c755d7a1fcb08167acd75b1e86836e1ed69029658e4bfb0c98de3ca2d7 not found: ID does not exist" Nov 04 02:58:25.945745 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:58:25.945720 4405 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-4d5a0200-5cdf8ddbcd-wdl8z"] Nov 04 02:58:25.960997 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:58:25.960965 4405 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-4d5a0200-5cdf8ddbcd-wdl8z" Nov 04 02:58:25.977703 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:58:25.975904 4405 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3e760665-863f-41a0-b2fb-7b33a1768fc6-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-4d5a0200-5959b9699-k2dpp\" (UID: \"3e760665-863f-41a0-b2fb-7b33a1768fc6\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-4d5a0200-5959b9699-k2dpp" Nov 04 02:58:25.977703 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:58:25.976094 4405 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mz87d\" (UniqueName: \"kubernetes.io/projected/3e760665-863f-41a0-b2fb-7b33a1768fc6-kube-api-access-mz87d\") pod \"payload-pre-processing-e2e-ait-4d5a0200-5959b9699-k2dpp\" (UID: \"3e760665-863f-41a0-b2fb-7b33a1768fc6\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-4d5a0200-5959b9699-k2dpp" Nov 04 02:58:26.023946 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:58:26.023867 4405 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-4d5a0200-5cdf8ddbcd-wdl8z"] Nov 04 02:58:26.024432 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:58:26.024408 4405 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-4d5a0200-dockercfg-bn4w5\"" Nov 04 02:58:26.083029 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:58:26.081384 4405 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mz87d\" (UniqueName: \"kubernetes.io/projected/3e760665-863f-41a0-b2fb-7b33a1768fc6-kube-api-access-mz87d\") pod \"payload-pre-processing-e2e-ait-4d5a0200-5959b9699-k2dpp\" (UID: \"3e760665-863f-41a0-b2fb-7b33a1768fc6\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-4d5a0200-5959b9699-k2dpp" Nov 04 02:58:26.087689 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:58:26.086869 4405 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k7bkq\" (UniqueName: \"kubernetes.io/projected/e0691251-c255-4643-a6c4-4dacae0f15b3-kube-api-access-k7bkq\") pod \"payload-processing-e2e-ait-4d5a0200-5cdf8ddbcd-wdl8z\" (UID: \"e0691251-c255-4643-a6c4-4dacae0f15b3\") " pod="openshift-ingress/payload-processing-e2e-ait-4d5a0200-5cdf8ddbcd-wdl8z" Nov 04 02:58:26.088190 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:58:26.088160 4405 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e0691251-c255-4643-a6c4-4dacae0f15b3-plugins-config-volume\") pod \"payload-processing-e2e-ait-4d5a0200-5cdf8ddbcd-wdl8z\" (UID: \"e0691251-c255-4643-a6c4-4dacae0f15b3\") " pod="openshift-ingress/payload-processing-e2e-ait-4d5a0200-5cdf8ddbcd-wdl8z" Nov 04 02:58:26.088560 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:58:26.088527 4405 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3e760665-863f-41a0-b2fb-7b33a1768fc6-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-4d5a0200-5959b9699-k2dpp\" (UID: \"3e760665-863f-41a0-b2fb-7b33a1768fc6\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-4d5a0200-5959b9699-k2dpp" Nov 04 02:58:26.089828 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:58:26.089526 4405 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3e760665-863f-41a0-b2fb-7b33a1768fc6-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-4d5a0200-5959b9699-k2dpp\" (UID: \"3e760665-863f-41a0-b2fb-7b33a1768fc6\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-4d5a0200-5959b9699-k2dpp" Nov 04 02:58:26.150524 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:58:26.150398 4405 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mz87d\" (UniqueName: \"kubernetes.io/projected/3e760665-863f-41a0-b2fb-7b33a1768fc6-kube-api-access-mz87d\") pod \"payload-pre-processing-e2e-ait-4d5a0200-5959b9699-k2dpp\" (UID: \"3e760665-863f-41a0-b2fb-7b33a1768fc6\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-4d5a0200-5959b9699-k2dpp" Nov 04 02:58:26.197692 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:58:26.191717 4405 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k7bkq\" (UniqueName: \"kubernetes.io/projected/e0691251-c255-4643-a6c4-4dacae0f15b3-kube-api-access-k7bkq\") pod \"payload-processing-e2e-ait-4d5a0200-5cdf8ddbcd-wdl8z\" (UID: \"e0691251-c255-4643-a6c4-4dacae0f15b3\") " pod="openshift-ingress/payload-processing-e2e-ait-4d5a0200-5cdf8ddbcd-wdl8z" Nov 04 02:58:26.197692 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:58:26.191933 4405 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e0691251-c255-4643-a6c4-4dacae0f15b3-plugins-config-volume\") pod \"payload-processing-e2e-ait-4d5a0200-5cdf8ddbcd-wdl8z\" (UID: \"e0691251-c255-4643-a6c4-4dacae0f15b3\") " pod="openshift-ingress/payload-processing-e2e-ait-4d5a0200-5cdf8ddbcd-wdl8z" Nov 04 02:58:26.197692 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:58:26.192925 4405 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e0691251-c255-4643-a6c4-4dacae0f15b3-plugins-config-volume\") pod \"payload-processing-e2e-ait-4d5a0200-5cdf8ddbcd-wdl8z\" (UID: \"e0691251-c255-4643-a6c4-4dacae0f15b3\") " pod="openshift-ingress/payload-processing-e2e-ait-4d5a0200-5cdf8ddbcd-wdl8z" Nov 04 02:58:26.215713 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:58:26.215650 4405 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-4d5a0200-5959b9699-k2dpp" Nov 04 02:58:26.280497 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:58:26.280382 4405 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k7bkq\" (UniqueName: \"kubernetes.io/projected/e0691251-c255-4643-a6c4-4dacae0f15b3-kube-api-access-k7bkq\") pod \"payload-processing-e2e-ait-4d5a0200-5cdf8ddbcd-wdl8z\" (UID: \"e0691251-c255-4643-a6c4-4dacae0f15b3\") " pod="openshift-ingress/payload-processing-e2e-ait-4d5a0200-5cdf8ddbcd-wdl8z" Nov 04 02:58:26.382546 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:58:26.382317 4405 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-4d5a0200-5cdf8ddbcd-wdl8z" Nov 04 02:58:26.444865 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:58:26.444761 4405 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-f73e00ed-866fdf7694-m4mwj"] Nov 04 02:58:26.474847 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:58:26.474819 4405 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-f73e00ed-866fdf7694-m4mwj"] Nov 04 02:58:26.869640 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:58:26.869606 4405 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76597d6855-jjxxp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.83:15021/healthz/ready\": dial tcp 10.134.0.83:15021: connect: connection refused" start-of-body= Nov 04 02:58:26.870043 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:58:26.869707 4405 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-jjxxp" podUID="121fbac1-908f-4c37-8efa-f17d449ed6b9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.83:15021/healthz/ready\": dial tcp 10.134.0.83:15021: connect: connection refused" Nov 04 02:58:26.955104 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:58:26.955076 4405 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-4d5a0200-5959b9699-k2dpp"] Nov 04 02:58:26.955937 ip-10-0-134-125 kubenswrapper[4405]: W1104 02:58:26.955912 4405 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3e760665_863f_41a0_b2fb_7b33a1768fc6.slice/crio-35e93fcd052b6c43ff9684e508be59b81e5be782f56a0119447497342cfbd58f WatchSource:0}: Error finding container 35e93fcd052b6c43ff9684e508be59b81e5be782f56a0119447497342cfbd58f: Status 404 returned error can't find the container with id 35e93fcd052b6c43ff9684e508be59b81e5be782f56a0119447497342cfbd58f Nov 04 02:58:27.066355 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:58:27.066326 4405 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-4d5a0200-5cdf8ddbcd-wdl8z"] Nov 04 02:58:27.068025 ip-10-0-134-125 kubenswrapper[4405]: W1104 02:58:27.067998 4405 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode0691251_c255_4643_a6c4_4dacae0f15b3.slice/crio-0513aaa17b1d480652b36b285731d2139f531f052bfc913cd02f4116dc68d099 WatchSource:0}: Error finding container 0513aaa17b1d480652b36b285731d2139f531f052bfc913cd02f4116dc68d099: Status 404 returned error can't find the container with id 0513aaa17b1d480652b36b285731d2139f531f052bfc913cd02f4116dc68d099 Nov 04 02:58:27.733982 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:58:27.733929 4405 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="55a6bf29-4dc5-4b77-9c91-7743a8b20b78" path="/var/lib/kubelet/pods/55a6bf29-4dc5-4b77-9c91-7743a8b20b78/volumes" Nov 04 02:58:27.790977 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:58:27.790943 4405 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-4d5a0200-5cdf8ddbcd-wdl8z" event={"ID":"e0691251-c255-4643-a6c4-4dacae0f15b3","Type":"ContainerStarted","Data":"a1567458568b76116a8757e8b5c54f3a6211b21f30835edc8dc1b759b23f4db8"} Nov 04 02:58:27.791179 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:58:27.791006 4405 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-4d5a0200-5cdf8ddbcd-wdl8z" event={"ID":"e0691251-c255-4643-a6c4-4dacae0f15b3","Type":"ContainerStarted","Data":"0513aaa17b1d480652b36b285731d2139f531f052bfc913cd02f4116dc68d099"} Nov 04 02:58:27.791179 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:58:27.791136 4405 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ait-4d5a0200-5cdf8ddbcd-wdl8z" Nov 04 02:58:27.832199 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:58:27.825759 4405 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-4d5a0200-5959b9699-k2dpp" event={"ID":"3e760665-863f-41a0-b2fb-7b33a1768fc6","Type":"ContainerStarted","Data":"ca362cfaa24e49540dd86bfd538d1fb23eec5fda60f69950cf92551176119618"} Nov 04 02:58:27.832199 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:58:27.825831 4405 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-4d5a0200-5959b9699-k2dpp" event={"ID":"3e760665-863f-41a0-b2fb-7b33a1768fc6","Type":"ContainerStarted","Data":"35e93fcd052b6c43ff9684e508be59b81e5be782f56a0119447497342cfbd58f"} Nov 04 02:58:27.832199 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:58:27.826812 4405 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-4d5a0200-5959b9699-k2dpp" Nov 04 02:58:27.869972 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:58:27.869934 4405 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76597d6855-jjxxp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.83:15021/healthz/ready\": dial tcp 10.134.0.83:15021: connect: connection refused" start-of-body= Nov 04 02:58:27.870369 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:58:27.870014 4405 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-jjxxp" podUID="121fbac1-908f-4c37-8efa-f17d449ed6b9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.83:15021/healthz/ready\": dial tcp 10.134.0.83:15021: connect: connection refused" Nov 04 02:58:27.923693 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:58:27.923436 4405 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ait-4d5a0200-5cdf8ddbcd-wdl8z" podStartSLOduration=2.92341478 podStartE2EDuration="2.92341478s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 02:58:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 02:58:27.910110805 +0000 UTC m=+1554.340810927" watchObservedRunningTime="2026-11-04 02:58:27.92341478 +0000 UTC m=+1554.354114901" Nov 04 02:58:28.040741 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:58:28.030041 4405 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ait-4d5a0200-5959b9699-k2dpp" podStartSLOduration=3.030020506 podStartE2EDuration="3.030020506s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 02:58:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 02:58:28.029282684 +0000 UTC m=+1554.459982816" watchObservedRunningTime="2026-11-04 02:58:28.030020506 +0000 UTC m=+1554.460720627" Nov 04 02:58:28.680957 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:58:28.680934 4405 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-87191e54-openshift-default-7c59fcc9cc-6s8gq" Nov 04 02:58:28.789708 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:58:28.789660 4405 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/65bede99-6a95-4ac2-84a9-2ffdaef6375d-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/65bede99-6a95-4ac2-84a9-2ffdaef6375d-credential-socket\") pod \"65bede99-6a95-4ac2-84a9-2ffdaef6375d\" (UID: \"65bede99-6a95-4ac2-84a9-2ffdaef6375d\") " Nov 04 02:58:28.789923 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:58:28.789728 4405 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/65bede99-6a95-4ac2-84a9-2ffdaef6375d-istio-token\" (UniqueName: \"kubernetes.io/projected/65bede99-6a95-4ac2-84a9-2ffdaef6375d-istio-token\") pod \"65bede99-6a95-4ac2-84a9-2ffdaef6375d\" (UID: \"65bede99-6a95-4ac2-84a9-2ffdaef6375d\") " Nov 04 02:58:28.789923 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:58:28.789867 4405 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/65bede99-6a95-4ac2-84a9-2ffdaef6375d-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/65bede99-6a95-4ac2-84a9-2ffdaef6375d-istiod-ca-cert\") pod \"65bede99-6a95-4ac2-84a9-2ffdaef6375d\" (UID: \"65bede99-6a95-4ac2-84a9-2ffdaef6375d\") " Nov 04 02:58:28.790046 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:58:28.789927 4405 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/65bede99-6a95-4ac2-84a9-2ffdaef6375d-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/65bede99-6a95-4ac2-84a9-2ffdaef6375d-workload-certs\") pod \"65bede99-6a95-4ac2-84a9-2ffdaef6375d\" (UID: \"65bede99-6a95-4ac2-84a9-2ffdaef6375d\") " Nov 04 02:58:28.790046 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:58:28.789933 4405 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/65bede99-6a95-4ac2-84a9-2ffdaef6375d-credential-socket" pod "65bede99-6a95-4ac2-84a9-2ffdaef6375d" (UID: "65bede99-6a95-4ac2-84a9-2ffdaef6375d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 02:58:28.790046 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:58:28.789982 4405 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/65bede99-6a95-4ac2-84a9-2ffdaef6375d-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/65bede99-6a95-4ac2-84a9-2ffdaef6375d-istio-podinfo\") pod \"65bede99-6a95-4ac2-84a9-2ffdaef6375d\" (UID: \"65bede99-6a95-4ac2-84a9-2ffdaef6375d\") " Nov 04 02:58:28.790046 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:58:28.790023 4405 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/65bede99-6a95-4ac2-84a9-2ffdaef6375d-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/65bede99-6a95-4ac2-84a9-2ffdaef6375d-workload-socket\") pod \"65bede99-6a95-4ac2-84a9-2ffdaef6375d\" (UID: \"65bede99-6a95-4ac2-84a9-2ffdaef6375d\") " Nov 04 02:58:28.790233 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:58:28.790053 4405 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/65bede99-6a95-4ac2-84a9-2ffdaef6375d-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/65bede99-6a95-4ac2-84a9-2ffdaef6375d-istio-envoy\") pod \"65bede99-6a95-4ac2-84a9-2ffdaef6375d\" (UID: \"65bede99-6a95-4ac2-84a9-2ffdaef6375d\") " Nov 04 02:58:28.790233 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:58:28.790145 4405 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/65bede99-6a95-4ac2-84a9-2ffdaef6375d-kube-api-access-mrj4j\" (UniqueName: \"kubernetes.io/projected/65bede99-6a95-4ac2-84a9-2ffdaef6375d-kube-api-access-mrj4j\") pod \"65bede99-6a95-4ac2-84a9-2ffdaef6375d\" (UID: \"65bede99-6a95-4ac2-84a9-2ffdaef6375d\") " Nov 04 02:58:28.790233 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:58:28.790183 4405 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/65bede99-6a95-4ac2-84a9-2ffdaef6375d-workload-certs" pod "65bede99-6a95-4ac2-84a9-2ffdaef6375d" (UID: "65bede99-6a95-4ac2-84a9-2ffdaef6375d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 02:58:28.790379 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:58:28.790269 4405 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/65bede99-6a95-4ac2-84a9-2ffdaef6375d-istiod-ca-cert" pod "65bede99-6a95-4ac2-84a9-2ffdaef6375d" (UID: "65bede99-6a95-4ac2-84a9-2ffdaef6375d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 02:58:28.790379 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:58:28.790364 4405 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/65bede99-6a95-4ac2-84a9-2ffdaef6375d-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/65bede99-6a95-4ac2-84a9-2ffdaef6375d-istio-data\") pod \"65bede99-6a95-4ac2-84a9-2ffdaef6375d\" (UID: \"65bede99-6a95-4ac2-84a9-2ffdaef6375d\") " Nov 04 02:58:28.790636 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:58:28.790613 4405 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/65bede99-6a95-4ac2-84a9-2ffdaef6375d-workload-socket" pod "65bede99-6a95-4ac2-84a9-2ffdaef6375d" (UID: "65bede99-6a95-4ac2-84a9-2ffdaef6375d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 02:58:28.790922 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:58:28.790868 4405 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/65bede99-6a95-4ac2-84a9-2ffdaef6375d-istio-data" pod "65bede99-6a95-4ac2-84a9-2ffdaef6375d" (UID: "65bede99-6a95-4ac2-84a9-2ffdaef6375d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 02:58:28.791227 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:58:28.791206 4405 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/65bede99-6a95-4ac2-84a9-2ffdaef6375d-credential-socket\") on node \"ip-10-0-134-125.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 02:58:28.791227 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:58:28.791229 4405 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/65bede99-6a95-4ac2-84a9-2ffdaef6375d-istiod-ca-cert\") on node \"ip-10-0-134-125.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 02:58:28.791413 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:58:28.791243 4405 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/65bede99-6a95-4ac2-84a9-2ffdaef6375d-workload-certs\") on node \"ip-10-0-134-125.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 02:58:28.791413 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:58:28.791257 4405 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/65bede99-6a95-4ac2-84a9-2ffdaef6375d-workload-socket\") on node \"ip-10-0-134-125.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 02:58:28.791413 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:58:28.791270 4405 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/65bede99-6a95-4ac2-84a9-2ffdaef6375d-istio-data\") on node \"ip-10-0-134-125.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 02:58:28.802345 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:58:28.802212 4405 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/65bede99-6a95-4ac2-84a9-2ffdaef6375d-kube-api-access-mrj4j" pod "65bede99-6a95-4ac2-84a9-2ffdaef6375d" (UID: "65bede99-6a95-4ac2-84a9-2ffdaef6375d"). InnerVolumeSpecName "kube-api-access-mrj4j". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 02:58:28.802345 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:58:28.802302 4405 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/65bede99-6a95-4ac2-84a9-2ffdaef6375d-istio-podinfo" pod "65bede99-6a95-4ac2-84a9-2ffdaef6375d" (UID: "65bede99-6a95-4ac2-84a9-2ffdaef6375d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 04 02:58:28.807870 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:58:28.807809 4405 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/65bede99-6a95-4ac2-84a9-2ffdaef6375d-istio-envoy" pod "65bede99-6a95-4ac2-84a9-2ffdaef6375d" (UID: "65bede99-6a95-4ac2-84a9-2ffdaef6375d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 02:58:28.808198 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:58:28.808156 4405 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/65bede99-6a95-4ac2-84a9-2ffdaef6375d-istio-token" pod "65bede99-6a95-4ac2-84a9-2ffdaef6375d" (UID: "65bede99-6a95-4ac2-84a9-2ffdaef6375d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 02:58:28.838049 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:58:28.838014 4405 generic.go:356] "Generic (PLEG): container finished" podID="65bede99-6a95-4ac2-84a9-2ffdaef6375d" containerID="c57b122e894a511162cb2c6094ea0fc5181c9e3df5e5f67ee5dde373f5ec9b06" exitCode=0 Nov 04 02:58:28.838207 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:58:28.838064 4405 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-87191e54-openshift-default-7c59fcc9cc-6s8gq" Nov 04 02:58:28.838207 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:58:28.838097 4405 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-87191e54-openshift-default-7c59fcc9cc-6s8gq" event={"ID":"65bede99-6a95-4ac2-84a9-2ffdaef6375d","Type":"ContainerDied","Data":"c57b122e894a511162cb2c6094ea0fc5181c9e3df5e5f67ee5dde373f5ec9b06"} Nov 04 02:58:28.838207 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:58:28.838158 4405 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-87191e54-openshift-default-7c59fcc9cc-6s8gq" event={"ID":"65bede99-6a95-4ac2-84a9-2ffdaef6375d","Type":"ContainerDied","Data":"2add8c4ed047724876e03323c864274c6e37d02559efe443ae15a22ca2a0d753"} Nov 04 02:58:28.838207 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:58:28.838185 4405 scope.go:122] "RemoveContainer" containerID="c57b122e894a511162cb2c6094ea0fc5181c9e3df5e5f67ee5dde373f5ec9b06" Nov 04 02:58:28.854139 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:58:28.854117 4405 scope.go:122] "RemoveContainer" containerID="4ecb3c5f8ee794f0d2eb13e06c3b61c5179f14d4a2be11269ab328d86637081a" Nov 04 02:58:28.866722 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:58:28.865868 4405 scope.go:122] "RemoveContainer" containerID="c57b122e894a511162cb2c6094ea0fc5181c9e3df5e5f67ee5dde373f5ec9b06" Nov 04 02:58:28.866722 ip-10-0-134-125 kubenswrapper[4405]: E1104 02:58:28.866237 4405 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c57b122e894a511162cb2c6094ea0fc5181c9e3df5e5f67ee5dde373f5ec9b06\": container with ID starting with c57b122e894a511162cb2c6094ea0fc5181c9e3df5e5f67ee5dde373f5ec9b06 not found: ID does not exist" containerID="c57b122e894a511162cb2c6094ea0fc5181c9e3df5e5f67ee5dde373f5ec9b06" Nov 04 02:58:28.866722 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:58:28.866271 4405 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c57b122e894a511162cb2c6094ea0fc5181c9e3df5e5f67ee5dde373f5ec9b06"} err="failed to get container status \"c57b122e894a511162cb2c6094ea0fc5181c9e3df5e5f67ee5dde373f5ec9b06\": rpc error: code = NotFound desc = could not find container \"c57b122e894a511162cb2c6094ea0fc5181c9e3df5e5f67ee5dde373f5ec9b06\": container with ID starting with c57b122e894a511162cb2c6094ea0fc5181c9e3df5e5f67ee5dde373f5ec9b06 not found: ID does not exist" Nov 04 02:58:28.866722 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:58:28.866292 4405 scope.go:122] "RemoveContainer" containerID="4ecb3c5f8ee794f0d2eb13e06c3b61c5179f14d4a2be11269ab328d86637081a" Nov 04 02:58:28.866722 ip-10-0-134-125 kubenswrapper[4405]: E1104 02:58:28.866571 4405 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4ecb3c5f8ee794f0d2eb13e06c3b61c5179f14d4a2be11269ab328d86637081a\": container with ID starting with 4ecb3c5f8ee794f0d2eb13e06c3b61c5179f14d4a2be11269ab328d86637081a not found: ID does not exist" containerID="4ecb3c5f8ee794f0d2eb13e06c3b61c5179f14d4a2be11269ab328d86637081a" Nov 04 02:58:28.866722 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:58:28.866602 4405 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4ecb3c5f8ee794f0d2eb13e06c3b61c5179f14d4a2be11269ab328d86637081a"} err="failed to get container status \"4ecb3c5f8ee794f0d2eb13e06c3b61c5179f14d4a2be11269ab328d86637081a\": rpc error: code = NotFound desc = could not find container \"4ecb3c5f8ee794f0d2eb13e06c3b61c5179f14d4a2be11269ab328d86637081a\": container with ID starting with 4ecb3c5f8ee794f0d2eb13e06c3b61c5179f14d4a2be11269ab328d86637081a not found: ID does not exist" Nov 04 02:58:28.871397 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:58:28.869986 4405 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76597d6855-jjxxp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.83:15021/healthz/ready\": dial tcp 10.134.0.83:15021: connect: connection refused" start-of-body= Nov 04 02:58:28.871397 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:58:28.870027 4405 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-jjxxp" podUID="121fbac1-908f-4c37-8efa-f17d449ed6b9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.83:15021/healthz/ready\": dial tcp 10.134.0.83:15021: connect: connection refused" Nov 04 02:58:28.932094 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:58:28.924725 4405 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/65bede99-6a95-4ac2-84a9-2ffdaef6375d-istio-podinfo\") on node \"ip-10-0-134-125.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 02:58:28.932094 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:58:28.924763 4405 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/65bede99-6a95-4ac2-84a9-2ffdaef6375d-istio-envoy\") on node \"ip-10-0-134-125.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 02:58:28.932094 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:58:28.924779 4405 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mrj4j\" (UniqueName: \"kubernetes.io/projected/65bede99-6a95-4ac2-84a9-2ffdaef6375d-kube-api-access-mrj4j\") on node \"ip-10-0-134-125.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 02:58:28.932094 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:58:28.924793 4405 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/65bede99-6a95-4ac2-84a9-2ffdaef6375d-istio-token\") on node \"ip-10-0-134-125.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 02:58:28.968501 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:58:28.968462 4405 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-87191e54-openshift-default-7c59fcc9cc-6s8gq"] Nov 04 02:58:28.974415 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:58:28.974380 4405 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-87191e54-openshift-default-7c59fcc9cc-6s8gq"] Nov 04 02:58:29.731620 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:58:29.731585 4405 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="65bede99-6a95-4ac2-84a9-2ffdaef6375d" path="/var/lib/kubelet/pods/65bede99-6a95-4ac2-84a9-2ffdaef6375d/volumes" Nov 04 02:58:29.869927 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:58:29.869893 4405 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76597d6855-jjxxp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.83:15021/healthz/ready\": dial tcp 10.134.0.83:15021: connect: connection refused" start-of-body= Nov 04 02:58:29.870081 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:58:29.869952 4405 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-jjxxp" podUID="121fbac1-908f-4c37-8efa-f17d449ed6b9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.83:15021/healthz/ready\": dial tcp 10.134.0.83:15021: connect: connection refused" Nov 04 02:58:30.870216 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:58:30.870174 4405 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76597d6855-jjxxp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.83:15021/healthz/ready\": dial tcp 10.134.0.83:15021: connect: connection refused" start-of-body= Nov 04 02:58:30.870638 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:58:30.870235 4405 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-jjxxp" podUID="121fbac1-908f-4c37-8efa-f17d449ed6b9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.83:15021/healthz/ready\": dial tcp 10.134.0.83:15021: connect: connection refused" Nov 04 02:58:31.869512 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:58:31.869477 4405 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76597d6855-jjxxp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.83:15021/healthz/ready\": dial tcp 10.134.0.83:15021: connect: connection refused" start-of-body= Nov 04 02:58:31.869707 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:58:31.869527 4405 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-jjxxp" podUID="121fbac1-908f-4c37-8efa-f17d449ed6b9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.83:15021/healthz/ready\": dial tcp 10.134.0.83:15021: connect: connection refused" Nov 04 02:58:32.782691 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:58:32.778761 4405 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-5fde04ee-59df47b5d7-gnjpp"] Nov 04 02:58:32.782691 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:58:32.779693 4405 cpu_manager.go:414] "containerMap: removing container" podUID="65bede99-6a95-4ac2-84a9-2ffdaef6375d" containerName="istio-proxy" Nov 04 02:58:32.782691 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:58:32.779715 4405 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="65bede99-6a95-4ac2-84a9-2ffdaef6375d" containerName="istio-proxy" Nov 04 02:58:32.782691 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:58:32.779775 4405 cpu_manager.go:414] "containerMap: removing container" podUID="65bede99-6a95-4ac2-84a9-2ffdaef6375d" containerName="istio-proxy" Nov 04 02:58:32.782691 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:58:32.779786 4405 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="65bede99-6a95-4ac2-84a9-2ffdaef6375d" containerName="istio-proxy" Nov 04 02:58:32.782691 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:58:32.779973 4405 memory_manager.go:365] "RemoveStaleState removing state" podUID="65bede99-6a95-4ac2-84a9-2ffdaef6375d" containerName="istio-proxy" Nov 04 02:58:32.791846 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:58:32.788986 4405 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-5fde04ee-59df47b5d7-gnjpp" Nov 04 02:58:32.871011 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:58:32.870972 4405 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-5fde04ee-59df47b5d7-gnjpp"] Nov 04 02:58:32.895697 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:58:32.895468 4405 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-derive-5fde04ee\"" Nov 04 02:58:32.895697 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:58:32.895641 4405 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-derive-5fde04ee-dockercfg-j8kkk\"" Nov 04 02:58:32.929523 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:58:32.929492 4405 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-5fde04ee-5fdc7f866d-lk25n"] Nov 04 02:58:32.931018 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:58:32.930992 4405 memory_manager.go:365] "RemoveStaleState removing state" podUID="65bede99-6a95-4ac2-84a9-2ffdaef6375d" containerName="istio-proxy" Nov 04 02:58:32.943783 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:58:32.935535 4405 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-5fde04ee-5fdc7f866d-lk25n" Nov 04 02:58:32.996272 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:58:32.996141 4405 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-5fde04ee-5fdc7f866d-lk25n"] Nov 04 02:58:33.023558 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:58:33.023508 4405 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/100fce2c-765a-4370-ac93-8dcbca038872-plugins-config-volume\") pod \"payload-processing-e2e-derive-5fde04ee-5fdc7f866d-lk25n\" (UID: \"100fce2c-765a-4370-ac93-8dcbca038872\") " pod="openshift-ingress/payload-processing-e2e-derive-5fde04ee-5fdc7f866d-lk25n" Nov 04 02:58:33.024050 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:58:33.024003 4405 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z7whb\" (UniqueName: \"kubernetes.io/projected/82ad73c2-7f63-43cb-a8af-5f17c4621c67-kube-api-access-z7whb\") pod \"payload-pre-processing-e2e-derive-5fde04ee-59df47b5d7-gnjpp\" (UID: \"82ad73c2-7f63-43cb-a8af-5f17c4621c67\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-5fde04ee-59df47b5d7-gnjpp" Nov 04 02:58:33.024409 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:58:33.024388 4405 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/82ad73c2-7f63-43cb-a8af-5f17c4621c67-plugins-config-volume\") pod \"payload-pre-processing-e2e-derive-5fde04ee-59df47b5d7-gnjpp\" (UID: \"82ad73c2-7f63-43cb-a8af-5f17c4621c67\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-5fde04ee-59df47b5d7-gnjpp" Nov 04 02:58:33.024627 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:58:33.024608 4405 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-64q7n\" (UniqueName: \"kubernetes.io/projected/100fce2c-765a-4370-ac93-8dcbca038872-kube-api-access-64q7n\") pod \"payload-processing-e2e-derive-5fde04ee-5fdc7f866d-lk25n\" (UID: \"100fce2c-765a-4370-ac93-8dcbca038872\") " pod="openshift-ingress/payload-processing-e2e-derive-5fde04ee-5fdc7f866d-lk25n" Nov 04 02:58:33.135567 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:58:33.135527 4405 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/100fce2c-765a-4370-ac93-8dcbca038872-plugins-config-volume\") pod \"payload-processing-e2e-derive-5fde04ee-5fdc7f866d-lk25n\" (UID: \"100fce2c-765a-4370-ac93-8dcbca038872\") " pod="openshift-ingress/payload-processing-e2e-derive-5fde04ee-5fdc7f866d-lk25n" Nov 04 02:58:33.239479 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:58:33.239437 4405 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z7whb\" (UniqueName: \"kubernetes.io/projected/82ad73c2-7f63-43cb-a8af-5f17c4621c67-kube-api-access-z7whb\") pod \"payload-pre-processing-e2e-derive-5fde04ee-59df47b5d7-gnjpp\" (UID: \"82ad73c2-7f63-43cb-a8af-5f17c4621c67\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-5fde04ee-59df47b5d7-gnjpp" Nov 04 02:58:33.240375 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:58:33.240346 4405 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/82ad73c2-7f63-43cb-a8af-5f17c4621c67-plugins-config-volume\") pod \"payload-pre-processing-e2e-derive-5fde04ee-59df47b5d7-gnjpp\" (UID: \"82ad73c2-7f63-43cb-a8af-5f17c4621c67\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-5fde04ee-59df47b5d7-gnjpp" Nov 04 02:58:33.241706 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:58:33.241578 4405 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/82ad73c2-7f63-43cb-a8af-5f17c4621c67-plugins-config-volume\") pod \"payload-pre-processing-e2e-derive-5fde04ee-59df47b5d7-gnjpp\" (UID: \"82ad73c2-7f63-43cb-a8af-5f17c4621c67\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-5fde04ee-59df47b5d7-gnjpp" Nov 04 02:58:33.242305 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:58:33.136734 4405 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/100fce2c-765a-4370-ac93-8dcbca038872-plugins-config-volume\") pod \"payload-processing-e2e-derive-5fde04ee-5fdc7f866d-lk25n\" (UID: \"100fce2c-765a-4370-ac93-8dcbca038872\") " pod="openshift-ingress/payload-processing-e2e-derive-5fde04ee-5fdc7f866d-lk25n" Nov 04 02:58:33.242453 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:58:33.241937 4405 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-64q7n\" (UniqueName: \"kubernetes.io/projected/100fce2c-765a-4370-ac93-8dcbca038872-kube-api-access-64q7n\") pod \"payload-processing-e2e-derive-5fde04ee-5fdc7f866d-lk25n\" (UID: \"100fce2c-765a-4370-ac93-8dcbca038872\") " pod="openshift-ingress/payload-processing-e2e-derive-5fde04ee-5fdc7f866d-lk25n" Nov 04 02:58:33.301449 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:58:33.301407 4405 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z7whb\" (UniqueName: \"kubernetes.io/projected/82ad73c2-7f63-43cb-a8af-5f17c4621c67-kube-api-access-z7whb\") pod \"payload-pre-processing-e2e-derive-5fde04ee-59df47b5d7-gnjpp\" (UID: \"82ad73c2-7f63-43cb-a8af-5f17c4621c67\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-5fde04ee-59df47b5d7-gnjpp" Nov 04 02:58:33.303446 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:58:33.303420 4405 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-64q7n\" (UniqueName: \"kubernetes.io/projected/100fce2c-765a-4370-ac93-8dcbca038872-kube-api-access-64q7n\") pod \"payload-processing-e2e-derive-5fde04ee-5fdc7f866d-lk25n\" (UID: \"100fce2c-765a-4370-ac93-8dcbca038872\") " pod="openshift-ingress/payload-processing-e2e-derive-5fde04ee-5fdc7f866d-lk25n" Nov 04 02:58:33.410785 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:58:33.410694 4405 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-5fde04ee-59df47b5d7-gnjpp" Nov 04 02:58:33.595624 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:58:33.595359 4405 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-5fde04ee-5fdc7f866d-lk25n" Nov 04 02:58:33.818405 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:58:33.808860 4405 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-5fde04ee-59df47b5d7-gnjpp"] Nov 04 02:58:33.881266 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:58:33.880784 4405 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76597d6855-jjxxp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.83:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 04 02:58:33.881266 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:58:33.880849 4405 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-jjxxp" podUID="121fbac1-908f-4c37-8efa-f17d449ed6b9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.83:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 04 02:58:33.918580 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:58:33.918067 4405 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-jjxxp" Nov 04 02:58:33.919426 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:58:33.919405 4405 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-jjxxp" Nov 04 02:58:33.921068 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:58:33.921040 4405 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-5fde04ee-59df47b5d7-gnjpp" event={"ID":"82ad73c2-7f63-43cb-a8af-5f17c4621c67","Type":"ContainerStarted","Data":"01d758935d7e23355b862eeb5c67ba322a04e4bc07d28b1501a6b2caa388c069"} Nov 04 02:58:33.922417 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:58:33.922395 4405 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-jjxxp" Nov 04 02:58:34.011658 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:58:34.011631 4405 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-5fde04ee-5fdc7f866d-lk25n"] Nov 04 02:58:34.012548 ip-10-0-134-125 kubenswrapper[4405]: W1104 02:58:34.012522 4405 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod100fce2c_765a_4370_ac93_8dcbca038872.slice/crio-84256fdcffd99f447f7ecf8909df13801038db51d41513e1095433fff74d5002 WatchSource:0}: Error finding container 84256fdcffd99f447f7ecf8909df13801038db51d41513e1095433fff74d5002: Status 404 returned error can't find the container with id 84256fdcffd99f447f7ecf8909df13801038db51d41513e1095433fff74d5002 Nov 04 02:58:34.930735 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:58:34.930698 4405 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-5fde04ee-5fdc7f866d-lk25n" event={"ID":"100fce2c-765a-4370-ac93-8dcbca038872","Type":"ContainerStarted","Data":"37dfe03c937c3c23aad39a11e25a7dbca9e449ae2b01a890d6664f9bbc3e96f5"} Nov 04 02:58:34.931207 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:58:34.930755 4405 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-5fde04ee-5fdc7f866d-lk25n" event={"ID":"100fce2c-765a-4370-ac93-8dcbca038872","Type":"ContainerStarted","Data":"84256fdcffd99f447f7ecf8909df13801038db51d41513e1095433fff74d5002"} Nov 04 02:58:34.931207 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:58:34.930920 4405 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-derive-5fde04ee-5fdc7f866d-lk25n" Nov 04 02:58:34.932546 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:58:34.932523 4405 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-5fde04ee-59df47b5d7-gnjpp" event={"ID":"82ad73c2-7f63-43cb-a8af-5f17c4621c67","Type":"ContainerStarted","Data":"c88e436d48f06c7058f5cfef43ca5f7d5826b286ef36b16bbc264934bab87401"} Nov 04 02:58:35.049560 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:58:35.049499 4405 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-derive-5fde04ee-5fdc7f866d-lk25n" podStartSLOduration=3.049481574 podStartE2EDuration="3.049481574s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 02:58:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 02:58:35.04617153 +0000 UTC m=+1561.476871650" watchObservedRunningTime="2026-11-04 02:58:35.049481574 +0000 UTC m=+1561.480181745" Nov 04 02:58:35.175701 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:58:35.175607 4405 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-derive-5fde04ee-59df47b5d7-gnjpp" podStartSLOduration=3.175588005 podStartE2EDuration="3.175588005s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 02:58:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 02:58:35.161919104 +0000 UTC m=+1561.592619266" watchObservedRunningTime="2026-11-04 02:58:35.175588005 +0000 UTC m=+1561.606288125" Nov 04 02:58:35.941494 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:58:35.941461 4405 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-derive-5fde04ee-59df47b5d7-gnjpp" Nov 04 02:58:36.886538 ip-10-0-134-125 kubenswrapper[4405]: E1104 02:58:36.886502 4405 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5f2vb" containerName="extractor" Nov 04 02:58:37.683377 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:58:37.683342 4405 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-76597d6855-dtlt4"] Nov 04 02:58:37.688017 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:58:37.687998 4405 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-dtlt4" Nov 04 02:58:37.742005 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:58:37.741966 4405 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3918cd06-f979-4c62-a75f-916151257784-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-76597d6855-dtlt4\" (UID: \"3918cd06-f979-4c62-a75f-916151257784\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-dtlt4" Nov 04 02:58:37.742227 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:58:37.742203 4405 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3918cd06-f979-4c62-a75f-916151257784-workload-socket\") pod \"maas-default-gateway-openshift-default-76597d6855-dtlt4\" (UID: \"3918cd06-f979-4c62-a75f-916151257784\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-dtlt4" Nov 04 02:58:37.742293 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:58:37.742272 4405 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3918cd06-f979-4c62-a75f-916151257784-credential-socket\") pod \"maas-default-gateway-openshift-default-76597d6855-dtlt4\" (UID: \"3918cd06-f979-4c62-a75f-916151257784\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-dtlt4" Nov 04 02:58:37.742432 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:58:37.742417 4405 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3918cd06-f979-4c62-a75f-916151257784-istio-podinfo\") pod \"maas-default-gateway-openshift-default-76597d6855-dtlt4\" (UID: \"3918cd06-f979-4c62-a75f-916151257784\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-dtlt4" Nov 04 02:58:37.742471 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:58:37.742448 4405 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3918cd06-f979-4c62-a75f-916151257784-workload-certs\") pod \"maas-default-gateway-openshift-default-76597d6855-dtlt4\" (UID: \"3918cd06-f979-4c62-a75f-916151257784\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-dtlt4" Nov 04 02:58:37.742514 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:58:37.742477 4405 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3918cd06-f979-4c62-a75f-916151257784-istio-envoy\") pod \"maas-default-gateway-openshift-default-76597d6855-dtlt4\" (UID: \"3918cd06-f979-4c62-a75f-916151257784\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-dtlt4" Nov 04 02:58:37.742514 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:58:37.742495 4405 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3918cd06-f979-4c62-a75f-916151257784-istio-token\") pod \"maas-default-gateway-openshift-default-76597d6855-dtlt4\" (UID: \"3918cd06-f979-4c62-a75f-916151257784\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-dtlt4" Nov 04 02:58:37.742650 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:58:37.742634 4405 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vd2mb\" (UniqueName: \"kubernetes.io/projected/3918cd06-f979-4c62-a75f-916151257784-kube-api-access-vd2mb\") pod \"maas-default-gateway-openshift-default-76597d6855-dtlt4\" (UID: \"3918cd06-f979-4c62-a75f-916151257784\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-dtlt4" Nov 04 02:58:37.742708 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:58:37.742689 4405 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3918cd06-f979-4c62-a75f-916151257784-istio-data\") pod \"maas-default-gateway-openshift-default-76597d6855-dtlt4\" (UID: \"3918cd06-f979-4c62-a75f-916151257784\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-dtlt4" Nov 04 02:58:37.843844 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:58:37.843811 4405 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3918cd06-f979-4c62-a75f-916151257784-credential-socket\") pod \"maas-default-gateway-openshift-default-76597d6855-dtlt4\" (UID: \"3918cd06-f979-4c62-a75f-916151257784\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-dtlt4" Nov 04 02:58:37.844022 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:58:37.843900 4405 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3918cd06-f979-4c62-a75f-916151257784-istio-podinfo\") pod \"maas-default-gateway-openshift-default-76597d6855-dtlt4\" (UID: \"3918cd06-f979-4c62-a75f-916151257784\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-dtlt4" Nov 04 02:58:37.844022 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:58:37.843940 4405 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3918cd06-f979-4c62-a75f-916151257784-workload-certs\") pod \"maas-default-gateway-openshift-default-76597d6855-dtlt4\" (UID: \"3918cd06-f979-4c62-a75f-916151257784\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-dtlt4" Nov 04 02:58:37.844022 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:58:37.843981 4405 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3918cd06-f979-4c62-a75f-916151257784-istio-envoy\") pod \"maas-default-gateway-openshift-default-76597d6855-dtlt4\" (UID: \"3918cd06-f979-4c62-a75f-916151257784\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-dtlt4" Nov 04 02:58:37.844022 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:58:37.844010 4405 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3918cd06-f979-4c62-a75f-916151257784-istio-token\") pod \"maas-default-gateway-openshift-default-76597d6855-dtlt4\" (UID: \"3918cd06-f979-4c62-a75f-916151257784\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-dtlt4" Nov 04 02:58:37.844207 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:58:37.844066 4405 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vd2mb\" (UniqueName: \"kubernetes.io/projected/3918cd06-f979-4c62-a75f-916151257784-kube-api-access-vd2mb\") pod \"maas-default-gateway-openshift-default-76597d6855-dtlt4\" (UID: \"3918cd06-f979-4c62-a75f-916151257784\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-dtlt4" Nov 04 02:58:37.844207 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:58:37.844118 4405 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3918cd06-f979-4c62-a75f-916151257784-istio-data\") pod \"maas-default-gateway-openshift-default-76597d6855-dtlt4\" (UID: \"3918cd06-f979-4c62-a75f-916151257784\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-dtlt4" Nov 04 02:58:37.844311 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:58:37.844210 4405 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3918cd06-f979-4c62-a75f-916151257784-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-76597d6855-dtlt4\" (UID: \"3918cd06-f979-4c62-a75f-916151257784\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-dtlt4" Nov 04 02:58:37.844311 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:58:37.844261 4405 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3918cd06-f979-4c62-a75f-916151257784-credential-socket\") pod \"maas-default-gateway-openshift-default-76597d6855-dtlt4\" (UID: \"3918cd06-f979-4c62-a75f-916151257784\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-dtlt4" Nov 04 02:58:37.844419 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:58:37.844338 4405 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3918cd06-f979-4c62-a75f-916151257784-workload-certs\") pod \"maas-default-gateway-openshift-default-76597d6855-dtlt4\" (UID: \"3918cd06-f979-4c62-a75f-916151257784\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-dtlt4" Nov 04 02:58:37.844474 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:58:37.844449 4405 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3918cd06-f979-4c62-a75f-916151257784-workload-socket\") pod \"maas-default-gateway-openshift-default-76597d6855-dtlt4\" (UID: \"3918cd06-f979-4c62-a75f-916151257784\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-dtlt4" Nov 04 02:58:37.844538 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:58:37.844504 4405 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3918cd06-f979-4c62-a75f-916151257784-istio-data\") pod \"maas-default-gateway-openshift-default-76597d6855-dtlt4\" (UID: \"3918cd06-f979-4c62-a75f-916151257784\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-dtlt4" Nov 04 02:58:37.844845 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:58:37.844765 4405 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3918cd06-f979-4c62-a75f-916151257784-workload-socket\") pod \"maas-default-gateway-openshift-default-76597d6855-dtlt4\" (UID: \"3918cd06-f979-4c62-a75f-916151257784\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-dtlt4" Nov 04 02:58:37.845032 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:58:37.844936 4405 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3918cd06-f979-4c62-a75f-916151257784-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-76597d6855-dtlt4\" (UID: \"3918cd06-f979-4c62-a75f-916151257784\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-dtlt4" Nov 04 02:58:37.846424 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:58:37.846403 4405 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3918cd06-f979-4c62-a75f-916151257784-istio-envoy\") pod \"maas-default-gateway-openshift-default-76597d6855-dtlt4\" (UID: \"3918cd06-f979-4c62-a75f-916151257784\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-dtlt4" Nov 04 02:58:37.846625 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:58:37.846606 4405 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3918cd06-f979-4c62-a75f-916151257784-istio-podinfo\") pod \"maas-default-gateway-openshift-default-76597d6855-dtlt4\" (UID: \"3918cd06-f979-4c62-a75f-916151257784\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-dtlt4" Nov 04 02:58:37.900059 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:58:37.900019 4405 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-76597d6855-dtlt4"] Nov 04 02:58:37.910315 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:58:37.910259 4405 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3918cd06-f979-4c62-a75f-916151257784-istio-token\") pod \"maas-default-gateway-openshift-default-76597d6855-dtlt4\" (UID: \"3918cd06-f979-4c62-a75f-916151257784\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-dtlt4" Nov 04 02:58:37.910501 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:58:37.910415 4405 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vd2mb\" (UniqueName: \"kubernetes.io/projected/3918cd06-f979-4c62-a75f-916151257784-kube-api-access-vd2mb\") pod \"maas-default-gateway-openshift-default-76597d6855-dtlt4\" (UID: \"3918cd06-f979-4c62-a75f-916151257784\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-dtlt4" Nov 04 02:58:38.003713 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:58:38.003595 4405 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-dtlt4" Nov 04 02:58:38.441191 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:58:38.441156 4405 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-76597d6855-dtlt4"] Nov 04 02:58:38.476715 ip-10-0-134-125 kubenswrapper[4405]: W1104 02:58:38.476653 4405 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3918cd06_f979_4c62_a75f_916151257784.slice/crio-00baa01c68564641aa37129e2fab5a1bd5e532cfb57e3ceacd79c81b7afb09fe WatchSource:0}: Error finding container 00baa01c68564641aa37129e2fab5a1bd5e532cfb57e3ceacd79c81b7afb09fe: Status 404 returned error can't find the container with id 00baa01c68564641aa37129e2fab5a1bd5e532cfb57e3ceacd79c81b7afb09fe Nov 04 02:58:38.479647 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:58:38.479613 4405 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 04 02:58:38.479769 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:58:38.479724 4405 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 04 02:58:38.479769 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:58:38.479758 4405 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 04 02:58:38.842379 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:58:38.842352 4405 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ait-4d5a0200-5cdf8ddbcd-wdl8z" Nov 04 02:58:38.970872 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:58:38.970839 4405 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-dtlt4" event={"ID":"3918cd06-f979-4c62-a75f-916151257784","Type":"ContainerStarted","Data":"fc53a9a9008d3810a8389307a3b49d36237f3d36cf06b43c2f838d30676feb91"} Nov 04 02:58:38.971070 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:58:38.970895 4405 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-dtlt4" event={"ID":"3918cd06-f979-4c62-a75f-916151257784","Type":"ContainerStarted","Data":"00baa01c68564641aa37129e2fab5a1bd5e532cfb57e3ceacd79c81b7afb09fe"} Nov 04 02:58:39.008132 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:58:39.008091 4405 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-dtlt4" Nov 04 02:58:39.011627 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:58:39.011603 4405 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76597d6855-dtlt4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.88:15021/healthz/ready\": dial tcp 10.134.0.88:15021: connect: connection refused" start-of-body= Nov 04 02:58:39.011809 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:58:39.011701 4405 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-dtlt4" podUID="3918cd06-f979-4c62-a75f-916151257784" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.88:15021/healthz/ready\": dial tcp 10.134.0.88:15021: connect: connection refused" Nov 04 02:58:39.852868 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:58:39.852834 4405 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-4d5a0200-5959b9699-k2dpp" Nov 04 02:58:39.975917 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:58:39.975844 4405 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-dtlt4" podStartSLOduration=2.97582564 podStartE2EDuration="2.97582564s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 02:58:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 02:58:39.112273487 +0000 UTC m=+1565.542973620" watchObservedRunningTime="2026-11-04 02:58:39.97582564 +0000 UTC m=+1566.406525763" Nov 04 02:58:40.035581 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:58:40.035539 4405 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76597d6855-dtlt4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.88:15021/healthz/ready\": dial tcp 10.134.0.88:15021: connect: connection refused" start-of-body= Nov 04 02:58:40.036775 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:58:40.036741 4405 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-dtlt4" podUID="3918cd06-f979-4c62-a75f-916151257784" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.88:15021/healthz/ready\": dial tcp 10.134.0.88:15021: connect: connection refused" Nov 04 02:58:41.016921 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:58:41.016878 4405 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76597d6855-dtlt4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.88:15021/healthz/ready\": dial tcp 10.134.0.88:15021: connect: connection refused" start-of-body= Nov 04 02:58:41.017397 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:58:41.016946 4405 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-dtlt4" podUID="3918cd06-f979-4c62-a75f-916151257784" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.88:15021/healthz/ready\": dial tcp 10.134.0.88:15021: connect: connection refused" Nov 04 02:58:42.017735 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:58:42.016489 4405 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76597d6855-dtlt4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.88:15021/healthz/ready\": dial tcp 10.134.0.88:15021: connect: connection refused" start-of-body= Nov 04 02:58:42.018355 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:58:42.018320 4405 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-dtlt4" podUID="3918cd06-f979-4c62-a75f-916151257784" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.88:15021/healthz/ready\": dial tcp 10.134.0.88:15021: connect: connection refused" Nov 04 02:58:43.019052 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:58:43.019004 4405 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76597d6855-dtlt4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.88:15021/healthz/ready\": dial tcp 10.134.0.88:15021: connect: connection refused" start-of-body= Nov 04 02:58:43.019710 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:58:43.019622 4405 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-dtlt4" podUID="3918cd06-f979-4c62-a75f-916151257784" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.88:15021/healthz/ready\": dial tcp 10.134.0.88:15021: connect: connection refused" Nov 04 02:58:44.009582 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:58:44.009117 4405 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76597d6855-dtlt4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.88:15021/healthz/ready\": dial tcp 10.134.0.88:15021: connect: connection refused" start-of-body= Nov 04 02:58:44.009582 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:58:44.009186 4405 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-dtlt4" podUID="3918cd06-f979-4c62-a75f-916151257784" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.88:15021/healthz/ready\": dial tcp 10.134.0.88:15021: connect: connection refused" Nov 04 02:58:45.004260 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:58:45.004227 4405 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76597d6855-dtlt4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.88:15021/healthz/ready\": dial tcp 10.134.0.88:15021: connect: connection refused" start-of-body= Nov 04 02:58:45.004680 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:58:45.004286 4405 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-dtlt4" podUID="3918cd06-f979-4c62-a75f-916151257784" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.88:15021/healthz/ready\": dial tcp 10.134.0.88:15021: connect: connection refused" Nov 04 02:58:45.945829 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:58:45.945795 4405 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-derive-5fde04ee-5fdc7f866d-lk25n" Nov 04 02:58:46.004378 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:58:46.004337 4405 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76597d6855-dtlt4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.88:15021/healthz/ready\": dial tcp 10.134.0.88:15021: connect: connection refused" start-of-body= Nov 04 02:58:46.004877 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:58:46.004391 4405 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-dtlt4" podUID="3918cd06-f979-4c62-a75f-916151257784" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.88:15021/healthz/ready\": dial tcp 10.134.0.88:15021: connect: connection refused" Nov 04 02:58:46.952273 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:58:46.952243 4405 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-derive-5fde04ee-59df47b5d7-gnjpp" Nov 04 02:58:47.708352 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:58:47.708319 4405 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-dtlt4" Nov 04 02:58:47.708879 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:58:47.708729 4405 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-dtlt4" Nov 04 02:58:47.709797 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:58:47.709776 4405 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-dtlt4" Nov 04 02:58:51.909129 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:58:51.909090 4405 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w4-072ece-openshift-default-78b784fd8c-k2vfq"] Nov 04 02:58:51.913718 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:58:51.913693 4405 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-072ece-openshift-default-78b784fd8c-k2vfq" Nov 04 02:58:51.944823 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:58:51.944787 4405 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/652d018b-8d07-49ae-b9fb-a87cfc70b0c8-workload-socket\") pod \"e2e-shared-a-w4-072ece-openshift-default-78b784fd8c-k2vfq\" (UID: \"652d018b-8d07-49ae-b9fb-a87cfc70b0c8\") " pod="openshift-ingress/e2e-shared-a-w4-072ece-openshift-default-78b784fd8c-k2vfq" Nov 04 02:58:51.945031 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:58:51.944860 4405 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/652d018b-8d07-49ae-b9fb-a87cfc70b0c8-istio-podinfo\") pod \"e2e-shared-a-w4-072ece-openshift-default-78b784fd8c-k2vfq\" (UID: \"652d018b-8d07-49ae-b9fb-a87cfc70b0c8\") " pod="openshift-ingress/e2e-shared-a-w4-072ece-openshift-default-78b784fd8c-k2vfq" Nov 04 02:58:51.945031 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:58:51.944964 4405 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/652d018b-8d07-49ae-b9fb-a87cfc70b0c8-istio-data\") pod \"e2e-shared-a-w4-072ece-openshift-default-78b784fd8c-k2vfq\" (UID: \"652d018b-8d07-49ae-b9fb-a87cfc70b0c8\") " pod="openshift-ingress/e2e-shared-a-w4-072ece-openshift-default-78b784fd8c-k2vfq" Nov 04 02:58:51.945148 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:58:51.945058 4405 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/652d018b-8d07-49ae-b9fb-a87cfc70b0c8-workload-certs\") pod \"e2e-shared-a-w4-072ece-openshift-default-78b784fd8c-k2vfq\" (UID: \"652d018b-8d07-49ae-b9fb-a87cfc70b0c8\") " pod="openshift-ingress/e2e-shared-a-w4-072ece-openshift-default-78b784fd8c-k2vfq" Nov 04 02:58:51.945148 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:58:51.945096 4405 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/652d018b-8d07-49ae-b9fb-a87cfc70b0c8-istio-envoy\") pod \"e2e-shared-a-w4-072ece-openshift-default-78b784fd8c-k2vfq\" (UID: \"652d018b-8d07-49ae-b9fb-a87cfc70b0c8\") " pod="openshift-ingress/e2e-shared-a-w4-072ece-openshift-default-78b784fd8c-k2vfq" Nov 04 02:58:51.945246 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:58:51.945190 4405 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2h5lc\" (UniqueName: \"kubernetes.io/projected/652d018b-8d07-49ae-b9fb-a87cfc70b0c8-kube-api-access-2h5lc\") pod \"e2e-shared-a-w4-072ece-openshift-default-78b784fd8c-k2vfq\" (UID: \"652d018b-8d07-49ae-b9fb-a87cfc70b0c8\") " pod="openshift-ingress/e2e-shared-a-w4-072ece-openshift-default-78b784fd8c-k2vfq" Nov 04 02:58:51.945246 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:58:51.945237 4405 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/652d018b-8d07-49ae-b9fb-a87cfc70b0c8-credential-socket\") pod \"e2e-shared-a-w4-072ece-openshift-default-78b784fd8c-k2vfq\" (UID: \"652d018b-8d07-49ae-b9fb-a87cfc70b0c8\") " pod="openshift-ingress/e2e-shared-a-w4-072ece-openshift-default-78b784fd8c-k2vfq" Nov 04 02:58:51.945334 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:58:51.945282 4405 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/652d018b-8d07-49ae-b9fb-a87cfc70b0c8-istiod-ca-cert\") pod \"e2e-shared-a-w4-072ece-openshift-default-78b784fd8c-k2vfq\" (UID: \"652d018b-8d07-49ae-b9fb-a87cfc70b0c8\") " pod="openshift-ingress/e2e-shared-a-w4-072ece-openshift-default-78b784fd8c-k2vfq" Nov 04 02:58:51.945868 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:58:51.945371 4405 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/652d018b-8d07-49ae-b9fb-a87cfc70b0c8-istio-token\") pod \"e2e-shared-a-w4-072ece-openshift-default-78b784fd8c-k2vfq\" (UID: \"652d018b-8d07-49ae-b9fb-a87cfc70b0c8\") " pod="openshift-ingress/e2e-shared-a-w4-072ece-openshift-default-78b784fd8c-k2vfq" Nov 04 02:58:51.965865 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:58:51.965832 4405 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w4-072ece-openshift-default-dockercfg-58sg6\"" Nov 04 02:58:52.021591 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:58:52.021554 4405 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-072ece-openshift-default-78b784fd8c-k2vfq"] Nov 04 02:58:52.047179 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:58:52.047140 4405 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/652d018b-8d07-49ae-b9fb-a87cfc70b0c8-istio-token\") pod \"e2e-shared-a-w4-072ece-openshift-default-78b784fd8c-k2vfq\" (UID: \"652d018b-8d07-49ae-b9fb-a87cfc70b0c8\") " pod="openshift-ingress/e2e-shared-a-w4-072ece-openshift-default-78b784fd8c-k2vfq" Nov 04 02:58:52.047427 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:58:52.047299 4405 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/652d018b-8d07-49ae-b9fb-a87cfc70b0c8-workload-socket\") pod \"e2e-shared-a-w4-072ece-openshift-default-78b784fd8c-k2vfq\" (UID: \"652d018b-8d07-49ae-b9fb-a87cfc70b0c8\") " pod="openshift-ingress/e2e-shared-a-w4-072ece-openshift-default-78b784fd8c-k2vfq" Nov 04 02:58:52.047557 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:58:52.047456 4405 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/652d018b-8d07-49ae-b9fb-a87cfc70b0c8-istio-podinfo\") pod \"e2e-shared-a-w4-072ece-openshift-default-78b784fd8c-k2vfq\" (UID: \"652d018b-8d07-49ae-b9fb-a87cfc70b0c8\") " pod="openshift-ingress/e2e-shared-a-w4-072ece-openshift-default-78b784fd8c-k2vfq" Nov 04 02:58:52.047557 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:58:52.047510 4405 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/652d018b-8d07-49ae-b9fb-a87cfc70b0c8-istio-data\") pod \"e2e-shared-a-w4-072ece-openshift-default-78b784fd8c-k2vfq\" (UID: \"652d018b-8d07-49ae-b9fb-a87cfc70b0c8\") " pod="openshift-ingress/e2e-shared-a-w4-072ece-openshift-default-78b784fd8c-k2vfq" Nov 04 02:58:52.047783 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:58:52.047606 4405 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/652d018b-8d07-49ae-b9fb-a87cfc70b0c8-workload-certs\") pod \"e2e-shared-a-w4-072ece-openshift-default-78b784fd8c-k2vfq\" (UID: \"652d018b-8d07-49ae-b9fb-a87cfc70b0c8\") " pod="openshift-ingress/e2e-shared-a-w4-072ece-openshift-default-78b784fd8c-k2vfq" Nov 04 02:58:52.047783 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:58:52.047637 4405 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/652d018b-8d07-49ae-b9fb-a87cfc70b0c8-istio-envoy\") pod \"e2e-shared-a-w4-072ece-openshift-default-78b784fd8c-k2vfq\" (UID: \"652d018b-8d07-49ae-b9fb-a87cfc70b0c8\") " pod="openshift-ingress/e2e-shared-a-w4-072ece-openshift-default-78b784fd8c-k2vfq" Nov 04 02:58:52.047783 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:58:52.047722 4405 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2h5lc\" (UniqueName: \"kubernetes.io/projected/652d018b-8d07-49ae-b9fb-a87cfc70b0c8-kube-api-access-2h5lc\") pod \"e2e-shared-a-w4-072ece-openshift-default-78b784fd8c-k2vfq\" (UID: \"652d018b-8d07-49ae-b9fb-a87cfc70b0c8\") " pod="openshift-ingress/e2e-shared-a-w4-072ece-openshift-default-78b784fd8c-k2vfq" Nov 04 02:58:52.047953 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:58:52.047792 4405 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/652d018b-8d07-49ae-b9fb-a87cfc70b0c8-credential-socket\") pod \"e2e-shared-a-w4-072ece-openshift-default-78b784fd8c-k2vfq\" (UID: \"652d018b-8d07-49ae-b9fb-a87cfc70b0c8\") " pod="openshift-ingress/e2e-shared-a-w4-072ece-openshift-default-78b784fd8c-k2vfq" Nov 04 02:58:52.047953 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:58:52.047848 4405 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/652d018b-8d07-49ae-b9fb-a87cfc70b0c8-istiod-ca-cert\") pod \"e2e-shared-a-w4-072ece-openshift-default-78b784fd8c-k2vfq\" (UID: \"652d018b-8d07-49ae-b9fb-a87cfc70b0c8\") " pod="openshift-ingress/e2e-shared-a-w4-072ece-openshift-default-78b784fd8c-k2vfq" Nov 04 02:58:52.048065 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:58:52.047966 4405 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/652d018b-8d07-49ae-b9fb-a87cfc70b0c8-workload-certs\") pod \"e2e-shared-a-w4-072ece-openshift-default-78b784fd8c-k2vfq\" (UID: \"652d018b-8d07-49ae-b9fb-a87cfc70b0c8\") " pod="openshift-ingress/e2e-shared-a-w4-072ece-openshift-default-78b784fd8c-k2vfq" Nov 04 02:58:52.048259 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:58:52.048234 4405 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/652d018b-8d07-49ae-b9fb-a87cfc70b0c8-istio-data\") pod \"e2e-shared-a-w4-072ece-openshift-default-78b784fd8c-k2vfq\" (UID: \"652d018b-8d07-49ae-b9fb-a87cfc70b0c8\") " pod="openshift-ingress/e2e-shared-a-w4-072ece-openshift-default-78b784fd8c-k2vfq" Nov 04 02:58:52.048462 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:58:52.048434 4405 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/652d018b-8d07-49ae-b9fb-a87cfc70b0c8-istiod-ca-cert\") pod \"e2e-shared-a-w4-072ece-openshift-default-78b784fd8c-k2vfq\" (UID: \"652d018b-8d07-49ae-b9fb-a87cfc70b0c8\") " pod="openshift-ingress/e2e-shared-a-w4-072ece-openshift-default-78b784fd8c-k2vfq" Nov 04 02:58:52.048462 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:58:52.047724 4405 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/652d018b-8d07-49ae-b9fb-a87cfc70b0c8-workload-socket\") pod \"e2e-shared-a-w4-072ece-openshift-default-78b784fd8c-k2vfq\" (UID: \"652d018b-8d07-49ae-b9fb-a87cfc70b0c8\") " pod="openshift-ingress/e2e-shared-a-w4-072ece-openshift-default-78b784fd8c-k2vfq" Nov 04 02:58:52.048765 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:58:52.048742 4405 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/652d018b-8d07-49ae-b9fb-a87cfc70b0c8-credential-socket\") pod \"e2e-shared-a-w4-072ece-openshift-default-78b784fd8c-k2vfq\" (UID: \"652d018b-8d07-49ae-b9fb-a87cfc70b0c8\") " pod="openshift-ingress/e2e-shared-a-w4-072ece-openshift-default-78b784fd8c-k2vfq" Nov 04 02:58:52.050802 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:58:52.050777 4405 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/652d018b-8d07-49ae-b9fb-a87cfc70b0c8-istio-podinfo\") pod \"e2e-shared-a-w4-072ece-openshift-default-78b784fd8c-k2vfq\" (UID: \"652d018b-8d07-49ae-b9fb-a87cfc70b0c8\") " pod="openshift-ingress/e2e-shared-a-w4-072ece-openshift-default-78b784fd8c-k2vfq" Nov 04 02:58:52.052098 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:58:52.052071 4405 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/652d018b-8d07-49ae-b9fb-a87cfc70b0c8-istio-envoy\") pod \"e2e-shared-a-w4-072ece-openshift-default-78b784fd8c-k2vfq\" (UID: \"652d018b-8d07-49ae-b9fb-a87cfc70b0c8\") " pod="openshift-ingress/e2e-shared-a-w4-072ece-openshift-default-78b784fd8c-k2vfq" Nov 04 02:58:52.265774 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:58:52.220054 4405 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/652d018b-8d07-49ae-b9fb-a87cfc70b0c8-istio-token\") pod \"e2e-shared-a-w4-072ece-openshift-default-78b784fd8c-k2vfq\" (UID: \"652d018b-8d07-49ae-b9fb-a87cfc70b0c8\") " pod="openshift-ingress/e2e-shared-a-w4-072ece-openshift-default-78b784fd8c-k2vfq" Nov 04 02:58:52.265774 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:58:52.255648 4405 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2h5lc\" (UniqueName: \"kubernetes.io/projected/652d018b-8d07-49ae-b9fb-a87cfc70b0c8-kube-api-access-2h5lc\") pod \"e2e-shared-a-w4-072ece-openshift-default-78b784fd8c-k2vfq\" (UID: \"652d018b-8d07-49ae-b9fb-a87cfc70b0c8\") " pod="openshift-ingress/e2e-shared-a-w4-072ece-openshift-default-78b784fd8c-k2vfq" Nov 04 02:58:52.538961 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:58:52.538859 4405 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-072ece-openshift-default-78b784fd8c-k2vfq" Nov 04 02:58:53.001908 ip-10-0-134-125 kubenswrapper[4405]: W1104 02:58:53.001878 4405 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod652d018b_8d07_49ae_b9fb_a87cfc70b0c8.slice/crio-992aab1001ece4f8d6e5a5e38cf6a111e232936fb9f8c812606eb7135d361d71 WatchSource:0}: Error finding container 992aab1001ece4f8d6e5a5e38cf6a111e232936fb9f8c812606eb7135d361d71: Status 404 returned error can't find the container with id 992aab1001ece4f8d6e5a5e38cf6a111e232936fb9f8c812606eb7135d361d71 Nov 04 02:58:53.003825 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:58:53.003796 4405 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-072ece-openshift-default-78b784fd8c-k2vfq"] Nov 04 02:58:53.005125 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:58:53.005086 4405 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 04 02:58:53.005209 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:58:53.005173 4405 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 04 02:58:53.005248 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:58:53.005216 4405 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 04 02:58:53.356446 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:58:53.356407 4405 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-072ece-openshift-default-78b784fd8c-k2vfq" event={"ID":"652d018b-8d07-49ae-b9fb-a87cfc70b0c8","Type":"ContainerStarted","Data":"a30b7638525ea4ce26e47f3f8e99113b4d9df664c679a6e02855af22859fe884"} Nov 04 02:58:53.356659 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:58:53.356487 4405 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-072ece-openshift-default-78b784fd8c-k2vfq" event={"ID":"652d018b-8d07-49ae-b9fb-a87cfc70b0c8","Type":"ContainerStarted","Data":"992aab1001ece4f8d6e5a5e38cf6a111e232936fb9f8c812606eb7135d361d71"} Nov 04 02:58:53.479644 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:58:53.479579 4405 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w4-072ece-openshift-default-78b784fd8c-k2vfq" podStartSLOduration=2.479562613 podStartE2EDuration="2.479562613s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 02:58:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 02:58:53.475530453 +0000 UTC m=+1579.906230638" watchObservedRunningTime="2026-11-04 02:58:53.479562613 +0000 UTC m=+1579.910262733" Nov 04 02:58:53.539204 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:58:53.539164 4405 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w4-072ece-openshift-default-78b784fd8c-k2vfq" Nov 04 02:58:54.536866 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:58:54.536837 4405 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-2g6x2"] Nov 04 02:58:54.541192 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:58:54.541167 4405 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-2g6x2" Nov 04 02:58:54.545626 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:58:54.545604 4405 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w4-072ece-openshift-default-78b784fd8c-k2vfq" Nov 04 02:58:54.591972 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:58:54.591935 4405 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/23294ab6-3eaf-41fd-8b5b-c7a0214bd532-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-2g6x2\" (UID: \"23294ab6-3eaf-41fd-8b5b-c7a0214bd532\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-2g6x2" Nov 04 02:58:54.592145 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:58:54.592116 4405 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/23294ab6-3eaf-41fd-8b5b-c7a0214bd532-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-2g6x2\" (UID: \"23294ab6-3eaf-41fd-8b5b-c7a0214bd532\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-2g6x2" Nov 04 02:58:54.592221 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:58:54.592205 4405 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/23294ab6-3eaf-41fd-8b5b-c7a0214bd532-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-2g6x2\" (UID: \"23294ab6-3eaf-41fd-8b5b-c7a0214bd532\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-2g6x2" Nov 04 02:58:54.592372 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:58:54.592347 4405 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/23294ab6-3eaf-41fd-8b5b-c7a0214bd532-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-2g6x2\" (UID: \"23294ab6-3eaf-41fd-8b5b-c7a0214bd532\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-2g6x2" Nov 04 02:58:54.592510 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:58:54.592444 4405 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sksxl\" (UniqueName: \"kubernetes.io/projected/23294ab6-3eaf-41fd-8b5b-c7a0214bd532-kube-api-access-sksxl\") pod \"istiod-openshift-gateway-7ddd9cfdc8-2g6x2\" (UID: \"23294ab6-3eaf-41fd-8b5b-c7a0214bd532\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-2g6x2" Nov 04 02:58:54.592657 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:58:54.592621 4405 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/23294ab6-3eaf-41fd-8b5b-c7a0214bd532-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-2g6x2\" (UID: \"23294ab6-3eaf-41fd-8b5b-c7a0214bd532\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-2g6x2" Nov 04 02:58:54.592802 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:58:54.592689 4405 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/23294ab6-3eaf-41fd-8b5b-c7a0214bd532-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-2g6x2\" (UID: \"23294ab6-3eaf-41fd-8b5b-c7a0214bd532\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-2g6x2" Nov 04 02:58:54.593493 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:58:54.593472 4405 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\"" Nov 04 02:58:54.593637 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:58:54.593472 4405 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\"" Nov 04 02:58:54.593637 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:58:54.593477 4405 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\"" Nov 04 02:58:54.593637 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:58:54.593580 4405 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-n69rv\"" Nov 04 02:58:54.648853 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:58:54.648822 4405 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-2g6x2"] Nov 04 02:58:54.695690 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:58:54.695639 4405 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/23294ab6-3eaf-41fd-8b5b-c7a0214bd532-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-2g6x2\" (UID: \"23294ab6-3eaf-41fd-8b5b-c7a0214bd532\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-2g6x2" Nov 04 02:58:54.695977 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:58:54.695945 4405 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/23294ab6-3eaf-41fd-8b5b-c7a0214bd532-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-2g6x2\" (UID: \"23294ab6-3eaf-41fd-8b5b-c7a0214bd532\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-2g6x2" Nov 04 02:58:54.696088 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:58:54.696001 4405 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/23294ab6-3eaf-41fd-8b5b-c7a0214bd532-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-2g6x2\" (UID: \"23294ab6-3eaf-41fd-8b5b-c7a0214bd532\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-2g6x2" Nov 04 02:58:54.696133 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:58:54.696093 4405 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/23294ab6-3eaf-41fd-8b5b-c7a0214bd532-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-2g6x2\" (UID: \"23294ab6-3eaf-41fd-8b5b-c7a0214bd532\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-2g6x2" Nov 04 02:58:54.696173 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:58:54.696154 4405 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/23294ab6-3eaf-41fd-8b5b-c7a0214bd532-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-2g6x2\" (UID: \"23294ab6-3eaf-41fd-8b5b-c7a0214bd532\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-2g6x2" Nov 04 02:58:54.696256 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:58:54.696241 4405 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/23294ab6-3eaf-41fd-8b5b-c7a0214bd532-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-2g6x2\" (UID: \"23294ab6-3eaf-41fd-8b5b-c7a0214bd532\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-2g6x2" Nov 04 02:58:54.696325 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:58:54.696310 4405 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sksxl\" (UniqueName: \"kubernetes.io/projected/23294ab6-3eaf-41fd-8b5b-c7a0214bd532-kube-api-access-sksxl\") pod \"istiod-openshift-gateway-7ddd9cfdc8-2g6x2\" (UID: \"23294ab6-3eaf-41fd-8b5b-c7a0214bd532\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-2g6x2" Nov 04 02:58:54.699908 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:58:54.699884 4405 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/23294ab6-3eaf-41fd-8b5b-c7a0214bd532-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-2g6x2\" (UID: \"23294ab6-3eaf-41fd-8b5b-c7a0214bd532\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-2g6x2" Nov 04 02:58:54.700575 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:58:54.700552 4405 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/23294ab6-3eaf-41fd-8b5b-c7a0214bd532-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-2g6x2\" (UID: \"23294ab6-3eaf-41fd-8b5b-c7a0214bd532\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-2g6x2" Nov 04 02:58:54.702937 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:58:54.702918 4405 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/23294ab6-3eaf-41fd-8b5b-c7a0214bd532-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-2g6x2\" (UID: \"23294ab6-3eaf-41fd-8b5b-c7a0214bd532\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-2g6x2" Nov 04 02:58:54.705002 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:58:54.704977 4405 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/23294ab6-3eaf-41fd-8b5b-c7a0214bd532-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-2g6x2\" (UID: \"23294ab6-3eaf-41fd-8b5b-c7a0214bd532\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-2g6x2" Nov 04 02:58:54.707813 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:58:54.707795 4405 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/23294ab6-3eaf-41fd-8b5b-c7a0214bd532-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-2g6x2\" (UID: \"23294ab6-3eaf-41fd-8b5b-c7a0214bd532\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-2g6x2" Nov 04 02:58:54.757182 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:58:54.757153 4405 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sksxl\" (UniqueName: \"kubernetes.io/projected/23294ab6-3eaf-41fd-8b5b-c7a0214bd532-kube-api-access-sksxl\") pod \"istiod-openshift-gateway-7ddd9cfdc8-2g6x2\" (UID: \"23294ab6-3eaf-41fd-8b5b-c7a0214bd532\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-2g6x2" Nov 04 02:58:54.760081 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:58:54.760050 4405 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/23294ab6-3eaf-41fd-8b5b-c7a0214bd532-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-2g6x2\" (UID: \"23294ab6-3eaf-41fd-8b5b-c7a0214bd532\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-2g6x2" Nov 04 02:58:54.856089 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:58:54.856060 4405 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-2g6x2" Nov 04 02:58:55.293656 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:58:55.293624 4405 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-2g6x2"] Nov 04 02:58:55.294844 ip-10-0-134-125 kubenswrapper[4405]: W1104 02:58:55.294813 4405 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod23294ab6_3eaf_41fd_8b5b_c7a0214bd532.slice/crio-65ca1d127ca6ff52863c43c205c195534763b10b0c552c9f0beaafcd8376976a WatchSource:0}: Error finding container 65ca1d127ca6ff52863c43c205c195534763b10b0c552c9f0beaafcd8376976a: Status 404 returned error can't find the container with id 65ca1d127ca6ff52863c43c205c195534763b10b0c552c9f0beaafcd8376976a Nov 04 02:58:55.370939 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:58:55.370906 4405 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-2g6x2" event={"ID":"23294ab6-3eaf-41fd-8b5b-c7a0214bd532","Type":"ContainerStarted","Data":"65ca1d127ca6ff52863c43c205c195534763b10b0c552c9f0beaafcd8376976a"} Nov 04 02:58:55.371206 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:58:55.371189 4405 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w4-072ece-openshift-default-78b784fd8c-k2vfq" Nov 04 02:58:55.372185 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:58:55.372163 4405 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w4-072ece-openshift-default-78b784fd8c-k2vfq" Nov 04 02:58:56.812737 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:58:56.812700 4405 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-bl5mg"] Nov 04 02:59:04.803571 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:59:04.803526 4405 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 04 02:59:04.803944 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:59:04.803611 4405 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 04 02:59:05.490952 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:59:05.490903 4405 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-2g6x2" event={"ID":"23294ab6-3eaf-41fd-8b5b-c7a0214bd532","Type":"ContainerStarted","Data":"95a1b806d2ddc75ad0d3d6e784dc9ae4e493c362f6a751d7fe17176a00340424"} Nov 04 02:59:05.491161 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:59:05.491137 4405 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-2g6x2" Nov 04 02:59:05.493639 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:59:05.493598 4405 patch_prober.go:29] interesting pod/istiod-openshift-gateway-7ddd9cfdc8-2g6x2 container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 04 02:59:05.493829 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:59:05.493650 4405 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-2g6x2" podUID="23294ab6-3eaf-41fd-8b5b-c7a0214bd532" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 04 02:59:05.701115 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:59:05.701066 4405 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-2g6x2" podStartSLOduration=2.196616855 podStartE2EDuration="11.701052538s" totalImagesPullingTime="9.504435683s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 02:58:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 02:59:05.66165684 +0000 UTC m=+1592.092356972" watchObservedRunningTime="2026-11-04 02:59:05.701052538 +0000 UTC m=+1592.131752658" Nov 04 02:59:06.500751 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:59:06.500719 4405 patch_prober.go:29] interesting pod/istiod-openshift-gateway-7ddd9cfdc8-2g6x2 container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 04 02:59:06.501256 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:59:06.500771 4405 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-2g6x2" podUID="23294ab6-3eaf-41fd-8b5b-c7a0214bd532" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 04 02:59:06.885460 ip-10-0-134-125 kubenswrapper[4405]: E1104 02:59:06.885419 4405 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5f2vb" containerName="extractor" Nov 04 02:59:09.501319 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:59:09.501287 4405 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-2g6x2" Nov 04 02:59:36.856692 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:59:36.855893 4405 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-fe4e0c-openshift-default-c7f4fc84c-vh6fd"] Nov 04 02:59:36.856692 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:59:36.856149 4405 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-fe4e0c-openshift-default-c7f4fc84c-vh6fd" podUID="70d9a666-2667-41e8-8425-4e2ba085599d" containerName="istio-proxy" containerID="cri-o://e086266869b21630287ce36176cac9278dbb5759714616463595d366b59d1915" gracePeriod=30 Nov 04 02:59:36.932829 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:59:36.932792 4405 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-fe4e0c-openshift-default-c7f4fc84c-pnfhp"] Nov 04 02:59:36.933343 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:59:36.933309 4405 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-fe4e0c-openshift-default-c7f4fc84c-pnfhp" podUID="86eff01f-03b3-4d4c-a2a3-f6798ffe5a38" containerName="istio-proxy" containerID="cri-o://6f33a3f08deda349abe224c91001482797196beb9470673511e1ac1b3a5db9da" gracePeriod=30 Nov 04 02:59:37.310032 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:59:37.309919 4405 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-a0bcbc/distinct-w2-a0bcbc-kserve-596f94c7cd-szl4g" podUID="bac32934-8a57-4919-b435-13f367cbea3e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 02:59:37.456709 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:59:37.428652 4405 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-ljhhw" podUID="f66b1861-d87b-4d57-ab27-f1a24544a534" containerName="main" probeResult="failure" output="Get \"https://10.134.0.26:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 02:59:37.621441 ip-10-0-134-125 kubenswrapper[4405]: E1104 02:59:37.621398 4405 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5f2vb" containerName="extractor" Nov 04 02:59:37.629764 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:59:37.622800 4405 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-ebb7fd/distinct-2-w6-ebb7fd-kserve-6f68c5db86-zrpjs" podUID="b7b786e0-9c94-4126-9949-cd354ebbfb2a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.65:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 04 02:59:37.634304 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:59:37.634261 4405 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-a0bcbc/premium-w2-a0bcbc-kserve-78dfd6d78c-lwth2" podUID="33b620c1-389c-48b4-b277-90252018c4b6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 02:59:37.643920 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:59:37.643869 4405 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-203c35/simulator-w4-203c35-kserve-f9c4bd695-999ns" podUID="eb0d8bdd-6808-4fd0-b9b5-567dff3e7ca5" containerName="main" probeResult="failure" output="Get \"https://10.134.0.46:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 04 02:59:37.691700 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:59:37.689339 4405 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-e516ff/distinct-w1-e516ff-kserve-f4474cb9c-glpzw" podUID="8842ad09-22ff-4492-ba5d-d4cacf4eac75" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/ready\": context deadline exceeded" Nov 04 02:59:37.691700 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:59:37.690165 4405 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-cdcf67/distinct-w3-cdcf67-kserve-667cc7f7d7-5m2lq" podUID="52ed22cc-96e4-4ef0-b34d-ed58c89a9183" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 02:59:38.140697 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:59:38.139164 4405 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-fe4e0c/distinct-2-w0-fe4e0c-kserve-579bf4cc77-8pbq8" podUID="2b2ab722-d1bb-4376-bede-f5f4ee4562aa" containerName="main" probeResult="failure" output="Get \"https://10.134.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 02:59:38.150881 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:59:38.150766 4405 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-a0bcbc/distinct-w2-a0bcbc-kserve-596f94c7cd-szl4g" podUID="bac32934-8a57-4919-b435-13f367cbea3e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 02:59:38.336692 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:59:38.335804 4405 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-a0bcbc/distinct-w2-a0bcbc-kserve-596f94c7cd-szl4g" podUID="bac32934-8a57-4919-b435-13f367cbea3e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 02:59:38.336692 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:59:38.336465 4405 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-a0bcbc/distinct-w2-a0bcbc-kserve-596f94c7cd-szl4g" Nov 04 02:59:38.444076 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:59:38.443950 4405 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-ljhhw" podUID="f66b1861-d87b-4d57-ab27-f1a24544a534" containerName="main" probeResult="failure" output="Get \"https://10.134.0.26:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 02:59:38.445563 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:59:38.445543 4405 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-ljhhw" Nov 04 02:59:38.453710 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:59:38.452404 4405 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-a0bcbc/distinct-w2-a0bcbc-kserve-596f94c7cd-szl4g" Nov 04 02:59:38.662703 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:59:38.645913 4405 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-203c35/simulator-w4-203c35-kserve-f9c4bd695-999ns" podUID="eb0d8bdd-6808-4fd0-b9b5-567dff3e7ca5" containerName="main" probeResult="failure" output="Get \"https://10.134.0.46:8000/ready\": context deadline exceeded" Nov 04 02:59:38.662703 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:59:38.646543 4405 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-203c35/simulator-w4-203c35-kserve-f9c4bd695-999ns" Nov 04 02:59:38.690706 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:59:38.690212 4405 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-e516ff/distinct-w1-e516ff-kserve-f4474cb9c-glpzw" podUID="8842ad09-22ff-4492-ba5d-d4cacf4eac75" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 02:59:38.696779 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:59:38.691073 4405 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-e516ff/distinct-w1-e516ff-kserve-f4474cb9c-glpzw" Nov 04 02:59:38.696779 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:59:38.691937 4405 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-cdcf67/distinct-w3-cdcf67-kserve-667cc7f7d7-5m2lq" podUID="52ed22cc-96e4-4ef0-b34d-ed58c89a9183" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 02:59:38.715698 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:59:38.704957 4405 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-cdcf67/distinct-w3-cdcf67-kserve-667cc7f7d7-5m2lq" Nov 04 02:59:38.787042 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:59:38.787014 4405 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-e516ff/distinct-w1-e516ff-kserve-f4474cb9c-glpzw" Nov 04 02:59:38.859460 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:59:38.859415 4405 prober.go:122] "Probe failed" probeType="Liveness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-ljhhw" podUID="f66b1861-d87b-4d57-ab27-f1a24544a534" containerName="main" probeResult="failure" output="Get \"https://10.134.0.26:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 02:59:38.880704 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:59:38.880285 4405 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-cdcf67/distinct-w3-cdcf67-kserve-667cc7f7d7-5m2lq" Nov 04 02:59:38.900032 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:59:38.899529 4405 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-203c35/simulator-w4-203c35-kserve-f9c4bd695-999ns" Nov 04 02:59:38.921243 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:59:38.921187 4405 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-ljhhw" Nov 04 02:59:42.029799 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:59:42.029757 4405 patch_prober.go:29] interesting pod/e2e-worker-w0-fe4e0c-openshift-default-c7f4fc84c-pnfhp container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 04 02:59:42.030273 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:59:42.029836 4405 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-fe4e0c-openshift-default-c7f4fc84c-pnfhp" podUID="86eff01f-03b3-4d4c-a2a3-f6798ffe5a38" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 04 02:59:42.072127 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:59:42.072099 4405 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-fe4e0c-openshift-default-c7f4fc84c-vh6fd" Nov 04 02:59:42.171720 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:59:42.171630 4405 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/70d9a666-2667-41e8-8425-4e2ba085599d-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/70d9a666-2667-41e8-8425-4e2ba085599d-workload-socket\") pod \"70d9a666-2667-41e8-8425-4e2ba085599d\" (UID: \"70d9a666-2667-41e8-8425-4e2ba085599d\") " Nov 04 02:59:42.171720 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:59:42.171712 4405 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/70d9a666-2667-41e8-8425-4e2ba085599d-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/70d9a666-2667-41e8-8425-4e2ba085599d-istio-podinfo\") pod \"70d9a666-2667-41e8-8425-4e2ba085599d\" (UID: \"70d9a666-2667-41e8-8425-4e2ba085599d\") " Nov 04 02:59:42.171935 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:59:42.171749 4405 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/70d9a666-2667-41e8-8425-4e2ba085599d-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/70d9a666-2667-41e8-8425-4e2ba085599d-istio-envoy\") pod \"70d9a666-2667-41e8-8425-4e2ba085599d\" (UID: \"70d9a666-2667-41e8-8425-4e2ba085599d\") " Nov 04 02:59:42.171935 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:59:42.171897 4405 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/70d9a666-2667-41e8-8425-4e2ba085599d-workload-socket" pod "70d9a666-2667-41e8-8425-4e2ba085599d" (UID: "70d9a666-2667-41e8-8425-4e2ba085599d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 02:59:42.172026 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:59:42.171943 4405 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/70d9a666-2667-41e8-8425-4e2ba085599d-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/70d9a666-2667-41e8-8425-4e2ba085599d-istio-data\") pod \"70d9a666-2667-41e8-8425-4e2ba085599d\" (UID: \"70d9a666-2667-41e8-8425-4e2ba085599d\") " Nov 04 02:59:42.172026 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:59:42.172008 4405 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/70d9a666-2667-41e8-8425-4e2ba085599d-istio-token\" (UniqueName: \"kubernetes.io/projected/70d9a666-2667-41e8-8425-4e2ba085599d-istio-token\") pod \"70d9a666-2667-41e8-8425-4e2ba085599d\" (UID: \"70d9a666-2667-41e8-8425-4e2ba085599d\") " Nov 04 02:59:42.172134 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:59:42.172075 4405 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/70d9a666-2667-41e8-8425-4e2ba085599d-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/70d9a666-2667-41e8-8425-4e2ba085599d-credential-socket\") pod \"70d9a666-2667-41e8-8425-4e2ba085599d\" (UID: \"70d9a666-2667-41e8-8425-4e2ba085599d\") " Nov 04 02:59:42.172243 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:59:42.172218 4405 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/70d9a666-2667-41e8-8425-4e2ba085599d-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/70d9a666-2667-41e8-8425-4e2ba085599d-istiod-ca-cert\") pod \"70d9a666-2667-41e8-8425-4e2ba085599d\" (UID: \"70d9a666-2667-41e8-8425-4e2ba085599d\") " Nov 04 02:59:42.172354 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:59:42.172219 4405 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/70d9a666-2667-41e8-8425-4e2ba085599d-istio-data" pod "70d9a666-2667-41e8-8425-4e2ba085599d" (UID: "70d9a666-2667-41e8-8425-4e2ba085599d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 02:59:42.172354 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:59:42.172329 4405 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/70d9a666-2667-41e8-8425-4e2ba085599d-credential-socket" pod "70d9a666-2667-41e8-8425-4e2ba085599d" (UID: "70d9a666-2667-41e8-8425-4e2ba085599d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 02:59:42.172550 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:59:42.172515 4405 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/70d9a666-2667-41e8-8425-4e2ba085599d-istiod-ca-cert" pod "70d9a666-2667-41e8-8425-4e2ba085599d" (UID: "70d9a666-2667-41e8-8425-4e2ba085599d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 02:59:42.172550 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:59:42.172531 4405 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/70d9a666-2667-41e8-8425-4e2ba085599d-kube-api-access-k7l8h\" (UniqueName: \"kubernetes.io/projected/70d9a666-2667-41e8-8425-4e2ba085599d-kube-api-access-k7l8h\") pod \"70d9a666-2667-41e8-8425-4e2ba085599d\" (UID: \"70d9a666-2667-41e8-8425-4e2ba085599d\") " Nov 04 02:59:42.172685 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:59:42.172618 4405 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/70d9a666-2667-41e8-8425-4e2ba085599d-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/70d9a666-2667-41e8-8425-4e2ba085599d-workload-certs\") pod \"70d9a666-2667-41e8-8425-4e2ba085599d\" (UID: \"70d9a666-2667-41e8-8425-4e2ba085599d\") " Nov 04 02:59:42.172983 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:59:42.172960 4405 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/70d9a666-2667-41e8-8425-4e2ba085599d-workload-certs" pod "70d9a666-2667-41e8-8425-4e2ba085599d" (UID: "70d9a666-2667-41e8-8425-4e2ba085599d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 02:59:42.173337 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:59:42.173318 4405 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/70d9a666-2667-41e8-8425-4e2ba085599d-workload-certs\") on node \"ip-10-0-134-125.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 02:59:42.173413 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:59:42.173343 4405 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/70d9a666-2667-41e8-8425-4e2ba085599d-workload-socket\") on node \"ip-10-0-134-125.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 02:59:42.173413 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:59:42.173357 4405 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/70d9a666-2667-41e8-8425-4e2ba085599d-istio-data\") on node \"ip-10-0-134-125.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 02:59:42.173413 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:59:42.173371 4405 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/70d9a666-2667-41e8-8425-4e2ba085599d-credential-socket\") on node \"ip-10-0-134-125.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 02:59:42.173413 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:59:42.173384 4405 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/70d9a666-2667-41e8-8425-4e2ba085599d-istiod-ca-cert\") on node \"ip-10-0-134-125.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 02:59:42.174320 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:59:42.174300 4405 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/70d9a666-2667-41e8-8425-4e2ba085599d-istio-token" pod "70d9a666-2667-41e8-8425-4e2ba085599d" (UID: "70d9a666-2667-41e8-8425-4e2ba085599d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 02:59:42.174515 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:59:42.174492 4405 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/70d9a666-2667-41e8-8425-4e2ba085599d-istio-envoy" pod "70d9a666-2667-41e8-8425-4e2ba085599d" (UID: "70d9a666-2667-41e8-8425-4e2ba085599d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 02:59:42.174515 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:59:42.174502 4405 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/70d9a666-2667-41e8-8425-4e2ba085599d-istio-podinfo" pod "70d9a666-2667-41e8-8425-4e2ba085599d" (UID: "70d9a666-2667-41e8-8425-4e2ba085599d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 04 02:59:42.174722 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:59:42.174705 4405 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/70d9a666-2667-41e8-8425-4e2ba085599d-kube-api-access-k7l8h" pod "70d9a666-2667-41e8-8425-4e2ba085599d" (UID: "70d9a666-2667-41e8-8425-4e2ba085599d"). InnerVolumeSpecName "kube-api-access-k7l8h". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 02:59:42.275548 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:59:42.275505 4405 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/70d9a666-2667-41e8-8425-4e2ba085599d-istio-podinfo\") on node \"ip-10-0-134-125.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 02:59:42.275548 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:59:42.275545 4405 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/70d9a666-2667-41e8-8425-4e2ba085599d-istio-envoy\") on node \"ip-10-0-134-125.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 02:59:42.275805 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:59:42.275562 4405 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/70d9a666-2667-41e8-8425-4e2ba085599d-istio-token\") on node \"ip-10-0-134-125.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 02:59:42.275805 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:59:42.275575 4405 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k7l8h\" (UniqueName: \"kubernetes.io/projected/70d9a666-2667-41e8-8425-4e2ba085599d-kube-api-access-k7l8h\") on node \"ip-10-0-134-125.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 02:59:42.593655 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:59:42.593620 4405 generic.go:356] "Generic (PLEG): container finished" podID="86eff01f-03b3-4d4c-a2a3-f6798ffe5a38" containerID="6f33a3f08deda349abe224c91001482797196beb9470673511e1ac1b3a5db9da" exitCode=0 Nov 04 02:59:42.593865 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:59:42.593696 4405 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-fe4e0c-openshift-default-c7f4fc84c-pnfhp" event={"ID":"86eff01f-03b3-4d4c-a2a3-f6798ffe5a38","Type":"ContainerDied","Data":"6f33a3f08deda349abe224c91001482797196beb9470673511e1ac1b3a5db9da"} Nov 04 02:59:42.595354 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:59:42.595328 4405 generic.go:356] "Generic (PLEG): container finished" podID="70d9a666-2667-41e8-8425-4e2ba085599d" containerID="e086266869b21630287ce36176cac9278dbb5759714616463595d366b59d1915" exitCode=0 Nov 04 02:59:42.595468 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:59:42.595382 4405 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-fe4e0c-openshift-default-c7f4fc84c-vh6fd" event={"ID":"70d9a666-2667-41e8-8425-4e2ba085599d","Type":"ContainerDied","Data":"e086266869b21630287ce36176cac9278dbb5759714616463595d366b59d1915"} Nov 04 02:59:42.595468 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:59:42.595398 4405 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-fe4e0c-openshift-default-c7f4fc84c-vh6fd" Nov 04 02:59:42.595468 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:59:42.595446 4405 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-fe4e0c-openshift-default-c7f4fc84c-vh6fd" event={"ID":"70d9a666-2667-41e8-8425-4e2ba085599d","Type":"ContainerDied","Data":"e9d78936b5488080ab49f3fccb754690c90bf7ed5f4cc52eb1194e58ab902ee5"} Nov 04 02:59:42.595581 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:59:42.595478 4405 scope.go:122] "RemoveContainer" containerID="e086266869b21630287ce36176cac9278dbb5759714616463595d366b59d1915" Nov 04 02:59:42.608027 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:59:42.608011 4405 scope.go:122] "RemoveContainer" containerID="e086266869b21630287ce36176cac9278dbb5759714616463595d366b59d1915" Nov 04 02:59:42.608314 ip-10-0-134-125 kubenswrapper[4405]: E1104 02:59:42.608296 4405 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e086266869b21630287ce36176cac9278dbb5759714616463595d366b59d1915\": container with ID starting with e086266869b21630287ce36176cac9278dbb5759714616463595d366b59d1915 not found: ID does not exist" containerID="e086266869b21630287ce36176cac9278dbb5759714616463595d366b59d1915" Nov 04 02:59:42.608373 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:59:42.608322 4405 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e086266869b21630287ce36176cac9278dbb5759714616463595d366b59d1915"} err="failed to get container status \"e086266869b21630287ce36176cac9278dbb5759714616463595d366b59d1915\": rpc error: code = NotFound desc = could not find container \"e086266869b21630287ce36176cac9278dbb5759714616463595d366b59d1915\": container with ID starting with e086266869b21630287ce36176cac9278dbb5759714616463595d366b59d1915 not found: ID does not exist" Nov 04 02:59:42.770977 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:59:42.770939 4405 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-fe4e0c-openshift-default-c7f4fc84c-vh6fd"] Nov 04 02:59:42.831517 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:59:42.831464 4405 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-fe4e0c-openshift-default-c7f4fc84c-vh6fd"] Nov 04 02:59:42.939975 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:59:42.939946 4405 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-fe4e0c-openshift-default-c7f4fc84c-pnfhp" Nov 04 02:59:42.999611 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:59:42.999573 4405 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/86eff01f-03b3-4d4c-a2a3-f6798ffe5a38-istio-token\" (UniqueName: \"kubernetes.io/projected/86eff01f-03b3-4d4c-a2a3-f6798ffe5a38-istio-token\") pod \"86eff01f-03b3-4d4c-a2a3-f6798ffe5a38\" (UID: \"86eff01f-03b3-4d4c-a2a3-f6798ffe5a38\") " Nov 04 02:59:42.999611 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:59:42.999615 4405 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/86eff01f-03b3-4d4c-a2a3-f6798ffe5a38-kube-api-access-tjvdt\" (UniqueName: \"kubernetes.io/projected/86eff01f-03b3-4d4c-a2a3-f6798ffe5a38-kube-api-access-tjvdt\") pod \"86eff01f-03b3-4d4c-a2a3-f6798ffe5a38\" (UID: \"86eff01f-03b3-4d4c-a2a3-f6798ffe5a38\") " Nov 04 02:59:42.999891 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:59:42.999686 4405 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/86eff01f-03b3-4d4c-a2a3-f6798ffe5a38-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/86eff01f-03b3-4d4c-a2a3-f6798ffe5a38-istio-podinfo\") pod \"86eff01f-03b3-4d4c-a2a3-f6798ffe5a38\" (UID: \"86eff01f-03b3-4d4c-a2a3-f6798ffe5a38\") " Nov 04 02:59:42.999891 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:59:42.999720 4405 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/86eff01f-03b3-4d4c-a2a3-f6798ffe5a38-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/86eff01f-03b3-4d4c-a2a3-f6798ffe5a38-credential-socket\") pod \"86eff01f-03b3-4d4c-a2a3-f6798ffe5a38\" (UID: \"86eff01f-03b3-4d4c-a2a3-f6798ffe5a38\") " Nov 04 02:59:42.999891 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:59:42.999829 4405 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/86eff01f-03b3-4d4c-a2a3-f6798ffe5a38-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/86eff01f-03b3-4d4c-a2a3-f6798ffe5a38-istio-data\") pod \"86eff01f-03b3-4d4c-a2a3-f6798ffe5a38\" (UID: \"86eff01f-03b3-4d4c-a2a3-f6798ffe5a38\") " Nov 04 02:59:42.999891 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:59:42.999870 4405 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/86eff01f-03b3-4d4c-a2a3-f6798ffe5a38-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/86eff01f-03b3-4d4c-a2a3-f6798ffe5a38-istiod-ca-cert\") pod \"86eff01f-03b3-4d4c-a2a3-f6798ffe5a38\" (UID: \"86eff01f-03b3-4d4c-a2a3-f6798ffe5a38\") " Nov 04 02:59:43.000119 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:59:42.999898 4405 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/86eff01f-03b3-4d4c-a2a3-f6798ffe5a38-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/86eff01f-03b3-4d4c-a2a3-f6798ffe5a38-istio-envoy\") pod \"86eff01f-03b3-4d4c-a2a3-f6798ffe5a38\" (UID: \"86eff01f-03b3-4d4c-a2a3-f6798ffe5a38\") " Nov 04 02:59:43.000119 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:59:42.999953 4405 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/86eff01f-03b3-4d4c-a2a3-f6798ffe5a38-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/86eff01f-03b3-4d4c-a2a3-f6798ffe5a38-workload-socket\") pod \"86eff01f-03b3-4d4c-a2a3-f6798ffe5a38\" (UID: \"86eff01f-03b3-4d4c-a2a3-f6798ffe5a38\") " Nov 04 02:59:43.000119 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:59:43.000021 4405 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/86eff01f-03b3-4d4c-a2a3-f6798ffe5a38-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/86eff01f-03b3-4d4c-a2a3-f6798ffe5a38-workload-certs\") pod \"86eff01f-03b3-4d4c-a2a3-f6798ffe5a38\" (UID: \"86eff01f-03b3-4d4c-a2a3-f6798ffe5a38\") " Nov 04 02:59:43.000264 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:59:43.000188 4405 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/86eff01f-03b3-4d4c-a2a3-f6798ffe5a38-istio-data" pod "86eff01f-03b3-4d4c-a2a3-f6798ffe5a38" (UID: "86eff01f-03b3-4d4c-a2a3-f6798ffe5a38"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 02:59:43.000264 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:59:43.000200 4405 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/86eff01f-03b3-4d4c-a2a3-f6798ffe5a38-credential-socket" pod "86eff01f-03b3-4d4c-a2a3-f6798ffe5a38" (UID: "86eff01f-03b3-4d4c-a2a3-f6798ffe5a38"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 02:59:43.000453 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:59:43.000411 4405 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/86eff01f-03b3-4d4c-a2a3-f6798ffe5a38-workload-socket" pod "86eff01f-03b3-4d4c-a2a3-f6798ffe5a38" (UID: "86eff01f-03b3-4d4c-a2a3-f6798ffe5a38"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 02:59:43.000514 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:59:43.000465 4405 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/86eff01f-03b3-4d4c-a2a3-f6798ffe5a38-workload-certs" pod "86eff01f-03b3-4d4c-a2a3-f6798ffe5a38" (UID: "86eff01f-03b3-4d4c-a2a3-f6798ffe5a38"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 02:59:43.000561 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:59:43.000528 4405 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/86eff01f-03b3-4d4c-a2a3-f6798ffe5a38-istiod-ca-cert" pod "86eff01f-03b3-4d4c-a2a3-f6798ffe5a38" (UID: "86eff01f-03b3-4d4c-a2a3-f6798ffe5a38"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 02:59:43.001099 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:59:43.000909 4405 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/86eff01f-03b3-4d4c-a2a3-f6798ffe5a38-credential-socket\") on node \"ip-10-0-134-125.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 02:59:43.001099 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:59:43.000936 4405 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/86eff01f-03b3-4d4c-a2a3-f6798ffe5a38-istio-data\") on node \"ip-10-0-134-125.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 02:59:43.001099 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:59:43.000951 4405 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/86eff01f-03b3-4d4c-a2a3-f6798ffe5a38-istiod-ca-cert\") on node \"ip-10-0-134-125.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 02:59:43.001099 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:59:43.000965 4405 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/86eff01f-03b3-4d4c-a2a3-f6798ffe5a38-workload-socket\") on node \"ip-10-0-134-125.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 02:59:43.001099 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:59:43.000978 4405 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/86eff01f-03b3-4d4c-a2a3-f6798ffe5a38-workload-certs\") on node \"ip-10-0-134-125.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 02:59:43.002507 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:59:43.002485 4405 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/86eff01f-03b3-4d4c-a2a3-f6798ffe5a38-istio-token" pod "86eff01f-03b3-4d4c-a2a3-f6798ffe5a38" (UID: "86eff01f-03b3-4d4c-a2a3-f6798ffe5a38"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 02:59:43.002797 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:59:43.002776 4405 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/86eff01f-03b3-4d4c-a2a3-f6798ffe5a38-istio-podinfo" pod "86eff01f-03b3-4d4c-a2a3-f6798ffe5a38" (UID: "86eff01f-03b3-4d4c-a2a3-f6798ffe5a38"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 04 02:59:43.003116 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:59:43.003088 4405 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/86eff01f-03b3-4d4c-a2a3-f6798ffe5a38-kube-api-access-tjvdt" pod "86eff01f-03b3-4d4c-a2a3-f6798ffe5a38" (UID: "86eff01f-03b3-4d4c-a2a3-f6798ffe5a38"). InnerVolumeSpecName "kube-api-access-tjvdt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 02:59:43.003460 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:59:43.003438 4405 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/86eff01f-03b3-4d4c-a2a3-f6798ffe5a38-istio-envoy" pod "86eff01f-03b3-4d4c-a2a3-f6798ffe5a38" (UID: "86eff01f-03b3-4d4c-a2a3-f6798ffe5a38"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 02:59:43.102884 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:59:43.102840 4405 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/86eff01f-03b3-4d4c-a2a3-f6798ffe5a38-istio-token\") on node \"ip-10-0-134-125.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 02:59:43.102884 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:59:43.102878 4405 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tjvdt\" (UniqueName: \"kubernetes.io/projected/86eff01f-03b3-4d4c-a2a3-f6798ffe5a38-kube-api-access-tjvdt\") on node \"ip-10-0-134-125.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 02:59:43.103498 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:59:43.102894 4405 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/86eff01f-03b3-4d4c-a2a3-f6798ffe5a38-istio-podinfo\") on node \"ip-10-0-134-125.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 02:59:43.103498 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:59:43.103008 4405 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/86eff01f-03b3-4d4c-a2a3-f6798ffe5a38-istio-envoy\") on node \"ip-10-0-134-125.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 02:59:43.606538 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:59:43.606499 4405 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-fe4e0c-openshift-default-c7f4fc84c-pnfhp" event={"ID":"86eff01f-03b3-4d4c-a2a3-f6798ffe5a38","Type":"ContainerDied","Data":"7f9c19f1d0ddb3115bcfc74139c976dc70b286dbe4944bc99d157a8bb83032b2"} Nov 04 02:59:43.606749 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:59:43.606512 4405 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-fe4e0c-openshift-default-c7f4fc84c-pnfhp" Nov 04 02:59:43.606749 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:59:43.606571 4405 scope.go:122] "RemoveContainer" containerID="6f33a3f08deda349abe224c91001482797196beb9470673511e1ac1b3a5db9da" Nov 04 02:59:43.716807 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:59:43.716769 4405 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="70d9a666-2667-41e8-8425-4e2ba085599d" path="/var/lib/kubelet/pods/70d9a666-2667-41e8-8425-4e2ba085599d/volumes" Nov 04 02:59:43.773121 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:59:43.773082 4405 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-fe4e0c-openshift-default-c7f4fc84c-pnfhp"] Nov 04 02:59:43.825176 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:59:43.825140 4405 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-fe4e0c-openshift-default-c7f4fc84c-pnfhp"] Nov 04 02:59:45.707570 ip-10-0-134-125 kubenswrapper[4405]: I1104 02:59:45.707534 4405 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="86eff01f-03b3-4d4c-a2a3-f6798ffe5a38" path="/var/lib/kubelet/pods/86eff01f-03b3-4d4c-a2a3-f6798ffe5a38/volumes" Nov 04 03:00:00.176445 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:00:00.176401 4405 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-e516ff-29896020-g2kgw"] Nov 04 03:00:00.177575 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:00:00.177534 4405 cpu_manager.go:414] "containerMap: removing container" podUID="70d9a666-2667-41e8-8425-4e2ba085599d" containerName="istio-proxy" Nov 04 03:00:00.177575 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:00:00.177562 4405 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="70d9a666-2667-41e8-8425-4e2ba085599d" containerName="istio-proxy" Nov 04 03:00:00.177811 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:00:00.177621 4405 cpu_manager.go:414] "containerMap: removing container" podUID="86eff01f-03b3-4d4c-a2a3-f6798ffe5a38" containerName="istio-proxy" Nov 04 03:00:00.177811 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:00:00.177630 4405 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="86eff01f-03b3-4d4c-a2a3-f6798ffe5a38" containerName="istio-proxy" Nov 04 03:00:00.177889 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:00:00.177845 4405 memory_manager.go:365] "RemoveStaleState removing state" podUID="86eff01f-03b3-4d4c-a2a3-f6798ffe5a38" containerName="istio-proxy" Nov 04 03:00:00.177889 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:00:00.177867 4405 memory_manager.go:365] "RemoveStaleState removing state" podUID="70d9a666-2667-41e8-8425-4e2ba085599d" containerName="istio-proxy" Nov 04 03:00:00.181826 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:00:00.181806 4405 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-e516ff-29896020-g2kgw" Nov 04 03:00:00.206576 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:00:00.206540 4405 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-derive-5fde04ee-29896020-df5j6"] Nov 04 03:00:00.211944 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:00:00.211913 4405 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-derive-5fde04ee-29896020-df5j6" Nov 04 03:00:00.213184 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:00:00.213151 4405 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-072ece-29896020-nlhnv"] Nov 04 03:00:00.218404 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:00:00.218382 4405 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-072ece-29896020-nlhnv" Nov 04 03:00:00.228262 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:00:00.228228 4405 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-203c35-29896020-ftklh"] Nov 04 03:00:00.235045 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:00:00.234950 4405 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-203c35-29896020-ftklh" Nov 04 03:00:00.249412 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:00:00.249386 4405 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-slbq8\"" Nov 04 03:00:00.274729 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:00:00.273997 4405 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-e516ff-29896020-g2kgw"] Nov 04 03:00:00.319696 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:00:00.311795 4405 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/71ce2ec9-0edd-483f-b364-b45e15ccef58-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-e516ff-29896020-g2kgw\" (UID: \"71ce2ec9-0edd-483f-b364-b45e15ccef58\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-e516ff-29896020-g2kgw" Nov 04 03:00:00.319696 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:00:00.311951 4405 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/42766046-4bb9-4afd-8fe7-3bc0e0199d1e-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w4-072ece-29896020-nlhnv\" (UID: \"42766046-4bb9-4afd-8fe7-3bc0e0199d1e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-072ece-29896020-nlhnv" Nov 04 03:00:00.319696 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:00:00.312023 4405 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1d9a643a-06d7-4e7e-85bb-a87e2108864f-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-derive-5fde04ee-29896020-df5j6\" (UID: \"1d9a643a-06d7-4e7e-85bb-a87e2108864f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-derive-5fde04ee-29896020-df5j6" Nov 04 03:00:00.319696 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:00:00.312106 4405 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9s242\" (UniqueName: \"kubernetes.io/projected/71ce2ec9-0edd-483f-b364-b45e15ccef58-kube-api-access-9s242\") pod \"maas-api-key-cleanup-e2e-worker-w1-e516ff-29896020-g2kgw\" (UID: \"71ce2ec9-0edd-483f-b364-b45e15ccef58\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-e516ff-29896020-g2kgw" Nov 04 03:00:00.319696 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:00:00.312199 4405 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2vcgs\" (UniqueName: \"kubernetes.io/projected/42766046-4bb9-4afd-8fe7-3bc0e0199d1e-kube-api-access-2vcgs\") pod \"maas-api-key-cleanup-e2e-shared-a-w4-072ece-29896020-nlhnv\" (UID: \"42766046-4bb9-4afd-8fe7-3bc0e0199d1e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-072ece-29896020-nlhnv" Nov 04 03:00:00.319696 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:00:00.312240 4405 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lrq87\" (UniqueName: \"kubernetes.io/projected/1d9a643a-06d7-4e7e-85bb-a87e2108864f-kube-api-access-lrq87\") pod \"maas-api-key-cleanup-e2e-derive-5fde04ee-29896020-df5j6\" (UID: \"1d9a643a-06d7-4e7e-85bb-a87e2108864f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-derive-5fde04ee-29896020-df5j6" Nov 04 03:00:00.343694 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:00:00.342071 4405 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-203c35-29896020-ftklh"] Nov 04 03:00:00.363962 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:00:00.353912 4405 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-derive-5fde04ee-29896020-df5j6"] Nov 04 03:00:00.380411 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:00:00.380377 4405 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-072ece-29896020-nlhnv"] Nov 04 03:00:00.414590 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:00:00.414545 4405 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/71ce2ec9-0edd-483f-b364-b45e15ccef58-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-e516ff-29896020-g2kgw\" (UID: \"71ce2ec9-0edd-483f-b364-b45e15ccef58\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-e516ff-29896020-g2kgw" Nov 04 03:00:00.414978 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:00:00.414955 4405 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/42766046-4bb9-4afd-8fe7-3bc0e0199d1e-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w4-072ece-29896020-nlhnv\" (UID: \"42766046-4bb9-4afd-8fe7-3bc0e0199d1e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-072ece-29896020-nlhnv" Nov 04 03:00:00.415233 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:00:00.415206 4405 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1d9a643a-06d7-4e7e-85bb-a87e2108864f-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-derive-5fde04ee-29896020-df5j6\" (UID: \"1d9a643a-06d7-4e7e-85bb-a87e2108864f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-derive-5fde04ee-29896020-df5j6" Nov 04 03:00:00.415412 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:00:00.415396 4405 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9s242\" (UniqueName: \"kubernetes.io/projected/71ce2ec9-0edd-483f-b364-b45e15ccef58-kube-api-access-9s242\") pod \"maas-api-key-cleanup-e2e-worker-w1-e516ff-29896020-g2kgw\" (UID: \"71ce2ec9-0edd-483f-b364-b45e15ccef58\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-e516ff-29896020-g2kgw" Nov 04 03:00:00.415620 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:00:00.415603 4405 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2vcgs\" (UniqueName: \"kubernetes.io/projected/42766046-4bb9-4afd-8fe7-3bc0e0199d1e-kube-api-access-2vcgs\") pod \"maas-api-key-cleanup-e2e-shared-a-w4-072ece-29896020-nlhnv\" (UID: \"42766046-4bb9-4afd-8fe7-3bc0e0199d1e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-072ece-29896020-nlhnv" Nov 04 03:00:00.415764 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:00:00.415749 4405 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lrq87\" (UniqueName: \"kubernetes.io/projected/1d9a643a-06d7-4e7e-85bb-a87e2108864f-kube-api-access-lrq87\") pod \"maas-api-key-cleanup-e2e-derive-5fde04ee-29896020-df5j6\" (UID: \"1d9a643a-06d7-4e7e-85bb-a87e2108864f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-derive-5fde04ee-29896020-df5j6" Nov 04 03:00:00.415888 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:00:00.415871 4405 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nbth7\" (UniqueName: \"kubernetes.io/projected/f61715a2-4d72-491b-8236-3e4a71462e60-kube-api-access-nbth7\") pod \"maas-api-key-cleanup-e2e-worker-w4-203c35-29896020-ftklh\" (UID: \"f61715a2-4d72-491b-8236-3e4a71462e60\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-203c35-29896020-ftklh" Nov 04 03:00:00.416717 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:00:00.416690 4405 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f61715a2-4d72-491b-8236-3e4a71462e60-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-203c35-29896020-ftklh\" (UID: \"f61715a2-4d72-491b-8236-3e4a71462e60\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-203c35-29896020-ftklh" Nov 04 03:00:00.417201 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:00:00.417173 4405 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/42766046-4bb9-4afd-8fe7-3bc0e0199d1e-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w4-072ece-29896020-nlhnv\" (UID: \"42766046-4bb9-4afd-8fe7-3bc0e0199d1e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-072ece-29896020-nlhnv" Nov 04 03:00:00.417850 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:00:00.417823 4405 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/71ce2ec9-0edd-483f-b364-b45e15ccef58-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-e516ff-29896020-g2kgw\" (UID: \"71ce2ec9-0edd-483f-b364-b45e15ccef58\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-e516ff-29896020-g2kgw" Nov 04 03:00:00.417971 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:00:00.417838 4405 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1d9a643a-06d7-4e7e-85bb-a87e2108864f-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-derive-5fde04ee-29896020-df5j6\" (UID: \"1d9a643a-06d7-4e7e-85bb-a87e2108864f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-derive-5fde04ee-29896020-df5j6" Nov 04 03:00:00.479818 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:00:00.479745 4405 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9s242\" (UniqueName: \"kubernetes.io/projected/71ce2ec9-0edd-483f-b364-b45e15ccef58-kube-api-access-9s242\") pod \"maas-api-key-cleanup-e2e-worker-w1-e516ff-29896020-g2kgw\" (UID: \"71ce2ec9-0edd-483f-b364-b45e15ccef58\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-e516ff-29896020-g2kgw" Nov 04 03:00:00.479818 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:00:00.479779 4405 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lrq87\" (UniqueName: \"kubernetes.io/projected/1d9a643a-06d7-4e7e-85bb-a87e2108864f-kube-api-access-lrq87\") pod \"maas-api-key-cleanup-e2e-derive-5fde04ee-29896020-df5j6\" (UID: \"1d9a643a-06d7-4e7e-85bb-a87e2108864f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-derive-5fde04ee-29896020-df5j6" Nov 04 03:00:00.479994 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:00:00.479910 4405 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2vcgs\" (UniqueName: \"kubernetes.io/projected/42766046-4bb9-4afd-8fe7-3bc0e0199d1e-kube-api-access-2vcgs\") pod \"maas-api-key-cleanup-e2e-shared-a-w4-072ece-29896020-nlhnv\" (UID: \"42766046-4bb9-4afd-8fe7-3bc0e0199d1e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-072ece-29896020-nlhnv" Nov 04 03:00:00.497912 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:00:00.497874 4405 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-e516ff-29896020-g2kgw" Nov 04 03:00:00.519413 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:00:00.519372 4405 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nbth7\" (UniqueName: \"kubernetes.io/projected/f61715a2-4d72-491b-8236-3e4a71462e60-kube-api-access-nbth7\") pod \"maas-api-key-cleanup-e2e-worker-w4-203c35-29896020-ftklh\" (UID: \"f61715a2-4d72-491b-8236-3e4a71462e60\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-203c35-29896020-ftklh" Nov 04 03:00:00.519567 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:00:00.519557 4405 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f61715a2-4d72-491b-8236-3e4a71462e60-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-203c35-29896020-ftklh\" (UID: \"f61715a2-4d72-491b-8236-3e4a71462e60\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-203c35-29896020-ftklh" Nov 04 03:00:00.520879 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:00:00.520344 4405 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f61715a2-4d72-491b-8236-3e4a71462e60-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-203c35-29896020-ftklh\" (UID: \"f61715a2-4d72-491b-8236-3e4a71462e60\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-203c35-29896020-ftklh" Nov 04 03:00:00.541143 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:00:00.534177 4405 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-derive-5fde04ee-29896020-df5j6" Nov 04 03:00:00.542720 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:00:00.541718 4405 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-072ece-29896020-nlhnv" Nov 04 03:00:00.593566 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:00:00.592206 4405 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nbth7\" (UniqueName: \"kubernetes.io/projected/f61715a2-4d72-491b-8236-3e4a71462e60-kube-api-access-nbth7\") pod \"maas-api-key-cleanup-e2e-worker-w4-203c35-29896020-ftklh\" (UID: \"f61715a2-4d72-491b-8236-3e4a71462e60\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-203c35-29896020-ftklh" Nov 04 03:00:00.854012 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:00:00.853977 4405 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-203c35-29896020-ftklh" Nov 04 03:00:00.911887 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:00:00.911852 4405 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-e516ff-29896020-g2kgw"] Nov 04 03:00:00.912656 ip-10-0-134-125 kubenswrapper[4405]: W1104 03:00:00.912614 4405 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod71ce2ec9_0edd_483f_b364_b45e15ccef58.slice/crio-f4b06b5fdd03c26b625e95745a85d0ee8df257e4a4baaeb89f71bf87850b882b WatchSource:0}: Error finding container f4b06b5fdd03c26b625e95745a85d0ee8df257e4a4baaeb89f71bf87850b882b: Status 404 returned error can't find the container with id f4b06b5fdd03c26b625e95745a85d0ee8df257e4a4baaeb89f71bf87850b882b Nov 04 03:00:00.985070 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:00:00.985042 4405 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-derive-5fde04ee-29896020-df5j6"] Nov 04 03:00:00.991019 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:00:00.990987 4405 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-072ece-29896020-nlhnv"] Nov 04 03:00:00.995535 ip-10-0-134-125 kubenswrapper[4405]: W1104 03:00:00.993916 4405 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod42766046_4bb9_4afd_8fe7_3bc0e0199d1e.slice/crio-01e39e96dcfe751bd0da070bda6802271a43b681a243c74a2c8138914b24b470 WatchSource:0}: Error finding container 01e39e96dcfe751bd0da070bda6802271a43b681a243c74a2c8138914b24b470: Status 404 returned error can't find the container with id 01e39e96dcfe751bd0da070bda6802271a43b681a243c74a2c8138914b24b470 Nov 04 03:00:01.256172 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:00:01.256138 4405 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-203c35-29896020-ftklh"] Nov 04 03:00:01.296835 ip-10-0-134-125 kubenswrapper[4405]: W1104 03:00:01.296788 4405 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf61715a2_4d72_491b_8236_3e4a71462e60.slice/crio-5996b6fa787cc0eb6d723150fdb631e80e8ff997e0a02250fe6b9eb191703cba WatchSource:0}: Error finding container 5996b6fa787cc0eb6d723150fdb631e80e8ff997e0a02250fe6b9eb191703cba: Status 404 returned error can't find the container with id 5996b6fa787cc0eb6d723150fdb631e80e8ff997e0a02250fe6b9eb191703cba Nov 04 03:00:01.789355 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:00:01.789313 4405 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-e516ff-29896020-g2kgw" event={"ID":"71ce2ec9-0edd-483f-b364-b45e15ccef58","Type":"ContainerStarted","Data":"18126aa62f5e6ceb1fe6b85215cfe7967ce2743785f686f11a9077552740af03"} Nov 04 03:00:01.789570 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:00:01.789385 4405 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-e516ff-29896020-g2kgw" event={"ID":"71ce2ec9-0edd-483f-b364-b45e15ccef58","Type":"ContainerStarted","Data":"f4b06b5fdd03c26b625e95745a85d0ee8df257e4a4baaeb89f71bf87850b882b"} Nov 04 03:00:01.790875 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:00:01.790843 4405 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-203c35-29896020-ftklh" event={"ID":"f61715a2-4d72-491b-8236-3e4a71462e60","Type":"ContainerStarted","Data":"5996b6fa787cc0eb6d723150fdb631e80e8ff997e0a02250fe6b9eb191703cba"} Nov 04 03:00:01.793216 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:00:01.793180 4405 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-072ece-29896020-nlhnv" event={"ID":"42766046-4bb9-4afd-8fe7-3bc0e0199d1e","Type":"ContainerStarted","Data":"1bb222f535eb69225239e957e5c94f527216b4710963f056d6ba30db576c7080"} Nov 04 03:00:01.793389 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:00:01.793241 4405 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-072ece-29896020-nlhnv" event={"ID":"42766046-4bb9-4afd-8fe7-3bc0e0199d1e","Type":"ContainerStarted","Data":"01e39e96dcfe751bd0da070bda6802271a43b681a243c74a2c8138914b24b470"} Nov 04 03:00:01.796474 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:00:01.796436 4405 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-derive-5fde04ee-29896020-df5j6" event={"ID":"1d9a643a-06d7-4e7e-85bb-a87e2108864f","Type":"ContainerStarted","Data":"939097e1d7035df5267be21af4fb342579059314e20a4a8a5e4cb036f55b9f51"} Nov 04 03:00:01.796657 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:00:01.796503 4405 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-derive-5fde04ee-29896020-df5j6" event={"ID":"1d9a643a-06d7-4e7e-85bb-a87e2108864f","Type":"ContainerStarted","Data":"80b226341fc18d9e2732331f534b280fe07f7abe3d0ddc2ab5cae654b54d1e2f"} Nov 04 03:00:01.907063 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:00:01.907001 4405 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-e516ff-29896020-g2kgw" podStartSLOduration=1.906979969 podStartE2EDuration="1.906979969s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 03:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 03:00:01.903750556 +0000 UTC m=+1648.334450678" watchObservedRunningTime="2026-11-04 03:00:01.906979969 +0000 UTC m=+1648.337680092" Nov 04 03:00:02.021374 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:00:02.021318 4405 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-072ece-29896020-nlhnv" podStartSLOduration=2.021300545 podStartE2EDuration="2.021300545s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 03:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 03:00:02.018004207 +0000 UTC m=+1648.448704326" watchObservedRunningTime="2026-11-04 03:00:02.021300545 +0000 UTC m=+1648.452000664" Nov 04 03:00:02.135445 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:00:02.135376 4405 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-derive-5fde04ee-29896020-df5j6" podStartSLOduration=2.135357175 podStartE2EDuration="2.135357175s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 03:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 03:00:02.131905312 +0000 UTC m=+1648.562605436" watchObservedRunningTime="2026-11-04 03:00:02.135357175 +0000 UTC m=+1648.566057299" Nov 04 03:00:02.808274 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:00:02.808233 4405 generic.go:356] "Generic (PLEG): container finished" podID="1d9a643a-06d7-4e7e-85bb-a87e2108864f" containerID="939097e1d7035df5267be21af4fb342579059314e20a4a8a5e4cb036f55b9f51" exitCode=60 Nov 04 03:00:02.808735 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:00:02.808315 4405 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-derive-5fde04ee-29896020-df5j6" event={"ID":"1d9a643a-06d7-4e7e-85bb-a87e2108864f","Type":"ContainerDied","Data":"939097e1d7035df5267be21af4fb342579059314e20a4a8a5e4cb036f55b9f51"} Nov 04 03:00:02.808735 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:00:02.808707 4405 scope.go:122] "RemoveContainer" containerID="939097e1d7035df5267be21af4fb342579059314e20a4a8a5e4cb036f55b9f51" Nov 04 03:00:02.811063 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:00:02.811012 4405 generic.go:356] "Generic (PLEG): container finished" podID="71ce2ec9-0edd-483f-b364-b45e15ccef58" containerID="18126aa62f5e6ceb1fe6b85215cfe7967ce2743785f686f11a9077552740af03" exitCode=60 Nov 04 03:00:02.811136 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:00:02.811099 4405 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-e516ff-29896020-g2kgw" event={"ID":"71ce2ec9-0edd-483f-b364-b45e15ccef58","Type":"ContainerDied","Data":"18126aa62f5e6ceb1fe6b85215cfe7967ce2743785f686f11a9077552740af03"} Nov 04 03:00:02.811427 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:00:02.811412 4405 scope.go:122] "RemoveContainer" containerID="18126aa62f5e6ceb1fe6b85215cfe7967ce2743785f686f11a9077552740af03" Nov 04 03:00:02.813783 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:00:02.813301 4405 generic.go:356] "Generic (PLEG): container finished" podID="f61715a2-4d72-491b-8236-3e4a71462e60" containerID="cdda5353b6a2f7879876a4bf1f9d49c8e162ca5379c071130123abe12cd348be" exitCode=60 Nov 04 03:00:02.813783 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:00:02.813414 4405 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-203c35-29896020-ftklh" event={"ID":"f61715a2-4d72-491b-8236-3e4a71462e60","Type":"ContainerDied","Data":"cdda5353b6a2f7879876a4bf1f9d49c8e162ca5379c071130123abe12cd348be"} Nov 04 03:00:02.813783 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:00:02.813618 4405 scope.go:122] "RemoveContainer" containerID="cdda5353b6a2f7879876a4bf1f9d49c8e162ca5379c071130123abe12cd348be" Nov 04 03:00:02.815475 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:00:02.815457 4405 generic.go:356] "Generic (PLEG): container finished" podID="42766046-4bb9-4afd-8fe7-3bc0e0199d1e" containerID="1bb222f535eb69225239e957e5c94f527216b4710963f056d6ba30db576c7080" exitCode=60 Nov 04 03:00:02.815556 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:00:02.815482 4405 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-072ece-29896020-nlhnv" event={"ID":"42766046-4bb9-4afd-8fe7-3bc0e0199d1e","Type":"ContainerDied","Data":"1bb222f535eb69225239e957e5c94f527216b4710963f056d6ba30db576c7080"} Nov 04 03:00:02.815787 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:00:02.815770 4405 scope.go:122] "RemoveContainer" containerID="1bb222f535eb69225239e957e5c94f527216b4710963f056d6ba30db576c7080" Nov 04 03:00:03.827703 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:00:03.827557 4405 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-derive-5fde04ee-29896020-df5j6" event={"ID":"1d9a643a-06d7-4e7e-85bb-a87e2108864f","Type":"ContainerStarted","Data":"c9d7d353156d315e5a70ef6191e4690f8c72320c75bfc164e90969288904e787"} Nov 04 03:00:03.829980 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:00:03.829948 4405 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-e516ff-29896020-g2kgw" event={"ID":"71ce2ec9-0edd-483f-b364-b45e15ccef58","Type":"ContainerStarted","Data":"8e423956ee29338dbbb0a910c5dabce023426228649fd31da720e2aa30593364"} Nov 04 03:00:03.832279 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:00:03.832243 4405 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-203c35-29896020-ftklh" event={"ID":"f61715a2-4d72-491b-8236-3e4a71462e60","Type":"ContainerStarted","Data":"42a12a4580408dc96071f5ac20370f496d168fefec440ced8f164123bbb72006"} Nov 04 03:00:03.834659 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:00:03.834625 4405 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-072ece-29896020-nlhnv" event={"ID":"42766046-4bb9-4afd-8fe7-3bc0e0199d1e","Type":"ContainerStarted","Data":"2d6a63a5a2cbed4d010a0ed3b9e4ce2c04122d4bd9cdbc54ac109c98dab39fd8"} Nov 04 03:00:04.061491 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:00:04.061428 4405 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-203c35-29896020-ftklh" podStartSLOduration=4.06141417 podStartE2EDuration="4.06141417s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 03:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 03:00:04.057487749 +0000 UTC m=+1650.488187870" watchObservedRunningTime="2026-11-04 03:00:04.06141417 +0000 UTC m=+1650.492114289" Nov 04 03:00:04.845866 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:00:04.845834 4405 generic.go:356] "Generic (PLEG): container finished" podID="71ce2ec9-0edd-483f-b364-b45e15ccef58" containerID="8e423956ee29338dbbb0a910c5dabce023426228649fd31da720e2aa30593364" exitCode=60 Nov 04 03:00:04.846288 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:00:04.845929 4405 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-e516ff-29896020-g2kgw" event={"ID":"71ce2ec9-0edd-483f-b364-b45e15ccef58","Type":"ContainerDied","Data":"8e423956ee29338dbbb0a910c5dabce023426228649fd31da720e2aa30593364"} Nov 04 03:00:04.846288 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:00:04.845989 4405 scope.go:122] "RemoveContainer" containerID="18126aa62f5e6ceb1fe6b85215cfe7967ce2743785f686f11a9077552740af03" Nov 04 03:00:04.846288 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:00:04.846266 4405 scope.go:122] "RemoveContainer" containerID="8e423956ee29338dbbb0a910c5dabce023426228649fd31da720e2aa30593364" Nov 04 03:00:04.846711 ip-10-0-134-125 kubenswrapper[4405]: E1104 03:00:04.846685 4405 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-e516ff-29896020-g2kgw_odh-ai-gateway-infra(71ce2ec9-0edd-483f-b364-b45e15ccef58)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-e516ff-29896020-g2kgw" podUID="71ce2ec9-0edd-483f-b364-b45e15ccef58" Nov 04 03:00:04.848476 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:00:04.848445 4405 generic.go:356] "Generic (PLEG): container finished" podID="f61715a2-4d72-491b-8236-3e4a71462e60" containerID="42a12a4580408dc96071f5ac20370f496d168fefec440ced8f164123bbb72006" exitCode=60 Nov 04 03:00:04.848555 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:00:04.848531 4405 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-203c35-29896020-ftklh" event={"ID":"f61715a2-4d72-491b-8236-3e4a71462e60","Type":"ContainerDied","Data":"42a12a4580408dc96071f5ac20370f496d168fefec440ced8f164123bbb72006"} Nov 04 03:00:04.848841 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:00:04.848825 4405 scope.go:122] "RemoveContainer" containerID="42a12a4580408dc96071f5ac20370f496d168fefec440ced8f164123bbb72006" Nov 04 03:00:04.849208 ip-10-0-134-125 kubenswrapper[4405]: E1104 03:00:04.849183 4405 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-203c35-29896020-ftklh_odh-ai-gateway-infra(f61715a2-4d72-491b-8236-3e4a71462e60)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-203c35-29896020-ftklh" podUID="f61715a2-4d72-491b-8236-3e4a71462e60" Nov 04 03:00:04.850488 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:00:04.850466 4405 generic.go:356] "Generic (PLEG): container finished" podID="42766046-4bb9-4afd-8fe7-3bc0e0199d1e" containerID="2d6a63a5a2cbed4d010a0ed3b9e4ce2c04122d4bd9cdbc54ac109c98dab39fd8" exitCode=60 Nov 04 03:00:04.850584 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:00:04.850546 4405 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-072ece-29896020-nlhnv" event={"ID":"42766046-4bb9-4afd-8fe7-3bc0e0199d1e","Type":"ContainerDied","Data":"2d6a63a5a2cbed4d010a0ed3b9e4ce2c04122d4bd9cdbc54ac109c98dab39fd8"} Nov 04 03:00:04.850807 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:00:04.850792 4405 scope.go:122] "RemoveContainer" containerID="2d6a63a5a2cbed4d010a0ed3b9e4ce2c04122d4bd9cdbc54ac109c98dab39fd8" Nov 04 03:00:04.851153 ip-10-0-134-125 kubenswrapper[4405]: E1104 03:00:04.851134 4405 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w4-072ece-29896020-nlhnv_odh-ai-gateway-infra(42766046-4bb9-4afd-8fe7-3bc0e0199d1e)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-072ece-29896020-nlhnv" podUID="42766046-4bb9-4afd-8fe7-3bc0e0199d1e" Nov 04 03:00:04.852584 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:00:04.852565 4405 generic.go:356] "Generic (PLEG): container finished" podID="1d9a643a-06d7-4e7e-85bb-a87e2108864f" containerID="c9d7d353156d315e5a70ef6191e4690f8c72320c75bfc164e90969288904e787" exitCode=60 Nov 04 03:00:04.852707 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:00:04.852609 4405 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-derive-5fde04ee-29896020-df5j6" event={"ID":"1d9a643a-06d7-4e7e-85bb-a87e2108864f","Type":"ContainerDied","Data":"c9d7d353156d315e5a70ef6191e4690f8c72320c75bfc164e90969288904e787"} Nov 04 03:00:04.852874 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:00:04.852858 4405 scope.go:122] "RemoveContainer" containerID="c9d7d353156d315e5a70ef6191e4690f8c72320c75bfc164e90969288904e787" Nov 04 03:00:04.853148 ip-10-0-134-125 kubenswrapper[4405]: E1104 03:00:04.853129 4405 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-derive-5fde04ee-29896020-df5j6_odh-ai-gateway-infra(1d9a643a-06d7-4e7e-85bb-a87e2108864f)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-derive-5fde04ee-29896020-df5j6" podUID="1d9a643a-06d7-4e7e-85bb-a87e2108864f" Nov 04 03:00:04.859756 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:00:04.859732 4405 scope.go:122] "RemoveContainer" containerID="cdda5353b6a2f7879876a4bf1f9d49c8e162ca5379c071130123abe12cd348be" Nov 04 03:00:04.873802 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:00:04.873777 4405 scope.go:122] "RemoveContainer" containerID="1bb222f535eb69225239e957e5c94f527216b4710963f056d6ba30db576c7080" Nov 04 03:00:04.887836 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:00:04.887815 4405 scope.go:122] "RemoveContainer" containerID="939097e1d7035df5267be21af4fb342579059314e20a4a8a5e4cb036f55b9f51" Nov 04 03:00:05.861652 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:00:05.861618 4405 scope.go:122] "RemoveContainer" containerID="2d6a63a5a2cbed4d010a0ed3b9e4ce2c04122d4bd9cdbc54ac109c98dab39fd8" Nov 04 03:00:05.862128 ip-10-0-134-125 kubenswrapper[4405]: E1104 03:00:05.862094 4405 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w4-072ece-29896020-nlhnv_odh-ai-gateway-infra(42766046-4bb9-4afd-8fe7-3bc0e0199d1e)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-072ece-29896020-nlhnv" podUID="42766046-4bb9-4afd-8fe7-3bc0e0199d1e" Nov 04 03:00:05.863580 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:00:05.863560 4405 scope.go:122] "RemoveContainer" containerID="c9d7d353156d315e5a70ef6191e4690f8c72320c75bfc164e90969288904e787" Nov 04 03:00:05.863973 ip-10-0-134-125 kubenswrapper[4405]: E1104 03:00:05.863943 4405 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-derive-5fde04ee-29896020-df5j6_odh-ai-gateway-infra(1d9a643a-06d7-4e7e-85bb-a87e2108864f)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-derive-5fde04ee-29896020-df5j6" podUID="1d9a643a-06d7-4e7e-85bb-a87e2108864f" Nov 04 03:00:05.865249 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:00:05.865232 4405 scope.go:122] "RemoveContainer" containerID="8e423956ee29338dbbb0a910c5dabce023426228649fd31da720e2aa30593364" Nov 04 03:00:05.865548 ip-10-0-134-125 kubenswrapper[4405]: E1104 03:00:05.865524 4405 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-e516ff-29896020-g2kgw_odh-ai-gateway-infra(71ce2ec9-0edd-483f-b364-b45e15ccef58)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-e516ff-29896020-g2kgw" podUID="71ce2ec9-0edd-483f-b364-b45e15ccef58" Nov 04 03:00:05.866898 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:00:05.866880 4405 scope.go:122] "RemoveContainer" containerID="42a12a4580408dc96071f5ac20370f496d168fefec440ced8f164123bbb72006" Nov 04 03:00:05.867168 ip-10-0-134-125 kubenswrapper[4405]: E1104 03:00:05.867149 4405 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-203c35-29896020-ftklh_odh-ai-gateway-infra(f61715a2-4d72-491b-8236-3e4a71462e60)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-203c35-29896020-ftklh" podUID="f61715a2-4d72-491b-8236-3e4a71462e60" Nov 04 03:00:06.956662 ip-10-0-134-125 kubenswrapper[4405]: E1104 03:00:06.956611 4405 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5f2vb" containerName="extractor" Nov 04 03:00:10.197596 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:00:10.197544 4405 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-bl5mg"] Nov 04 03:00:12.835290 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:00:12.835251 4405 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-297806-75b4854d47-mplp7"] Nov 04 03:00:12.841316 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:00:12.841293 4405 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-297806-75b4854d47-mplp7" Nov 04 03:00:12.910088 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:00:12.910053 4405 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-w5-297806-serving-cert\"" Nov 04 03:00:12.963790 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:00:12.963756 4405 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-297806-75b4854d47-mplp7"] Nov 04 03:00:12.979340 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:00:12.979299 4405 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nkq6b\" (UniqueName: \"kubernetes.io/projected/ef8afda8-47fe-4cab-85a0-01397c9ab56b-kube-api-access-nkq6b\") pod \"maas-api-e2e-shared-a-w5-297806-75b4854d47-mplp7\" (UID: \"ef8afda8-47fe-4cab-85a0-01397c9ab56b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-297806-75b4854d47-mplp7" Nov 04 03:00:12.979539 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:00:12.979402 4405 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ef8afda8-47fe-4cab-85a0-01397c9ab56b-maas-api-tls\") pod \"maas-api-e2e-shared-a-w5-297806-75b4854d47-mplp7\" (UID: \"ef8afda8-47fe-4cab-85a0-01397c9ab56b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-297806-75b4854d47-mplp7" Nov 04 03:00:12.979539 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:00:12.979497 4405 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ef8afda8-47fe-4cab-85a0-01397c9ab56b-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w5-297806-75b4854d47-mplp7\" (UID: \"ef8afda8-47fe-4cab-85a0-01397c9ab56b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-297806-75b4854d47-mplp7" Nov 04 03:00:12.979703 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:00:12.979611 4405 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ef8afda8-47fe-4cab-85a0-01397c9ab56b-metrics-certs\") pod \"maas-api-e2e-shared-a-w5-297806-75b4854d47-mplp7\" (UID: \"ef8afda8-47fe-4cab-85a0-01397c9ab56b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-297806-75b4854d47-mplp7" Nov 04 03:00:13.081402 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:00:13.081354 4405 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ef8afda8-47fe-4cab-85a0-01397c9ab56b-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w5-297806-75b4854d47-mplp7\" (UID: \"ef8afda8-47fe-4cab-85a0-01397c9ab56b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-297806-75b4854d47-mplp7" Nov 04 03:00:13.081590 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:00:13.081567 4405 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ef8afda8-47fe-4cab-85a0-01397c9ab56b-metrics-certs\") pod \"maas-api-e2e-shared-a-w5-297806-75b4854d47-mplp7\" (UID: \"ef8afda8-47fe-4cab-85a0-01397c9ab56b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-297806-75b4854d47-mplp7" Nov 04 03:00:13.081713 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:00:13.081695 4405 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nkq6b\" (UniqueName: \"kubernetes.io/projected/ef8afda8-47fe-4cab-85a0-01397c9ab56b-kube-api-access-nkq6b\") pod \"maas-api-e2e-shared-a-w5-297806-75b4854d47-mplp7\" (UID: \"ef8afda8-47fe-4cab-85a0-01397c9ab56b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-297806-75b4854d47-mplp7" Nov 04 03:00:13.081845 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:00:13.081814 4405 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ef8afda8-47fe-4cab-85a0-01397c9ab56b-maas-api-tls\") pod \"maas-api-e2e-shared-a-w5-297806-75b4854d47-mplp7\" (UID: \"ef8afda8-47fe-4cab-85a0-01397c9ab56b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-297806-75b4854d47-mplp7" Nov 04 03:00:13.082053 ip-10-0-134-125 kubenswrapper[4405]: E1104 03:00:13.082034 4405 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-297806-serving-cert: secret "maas-api-e2e-shared-a-w5-297806-serving-cert" not found Nov 04 03:00:13.082274 ip-10-0-134-125 kubenswrapper[4405]: E1104 03:00:13.082262 4405 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ef8afda8-47fe-4cab-85a0-01397c9ab56b-maas-api-tls podName:ef8afda8-47fe-4cab-85a0-01397c9ab56b nodeName:}" failed. No retries permitted until 2026-11-04 03:00:13.582241246 +0000 UTC m=+1660.012941350 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/ef8afda8-47fe-4cab-85a0-01397c9ab56b-maas-api-tls") pod "maas-api-e2e-shared-a-w5-297806-75b4854d47-mplp7" (UID: "ef8afda8-47fe-4cab-85a0-01397c9ab56b") : secret "maas-api-e2e-shared-a-w5-297806-serving-cert" not found Nov 04 03:00:13.082384 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:00:13.082160 4405 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ef8afda8-47fe-4cab-85a0-01397c9ab56b-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w5-297806-75b4854d47-mplp7\" (UID: \"ef8afda8-47fe-4cab-85a0-01397c9ab56b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-297806-75b4854d47-mplp7" Nov 04 03:00:13.084258 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:00:13.084238 4405 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ef8afda8-47fe-4cab-85a0-01397c9ab56b-metrics-certs\") pod \"maas-api-e2e-shared-a-w5-297806-75b4854d47-mplp7\" (UID: \"ef8afda8-47fe-4cab-85a0-01397c9ab56b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-297806-75b4854d47-mplp7" Nov 04 03:00:13.260770 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:00:13.243385 4405 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nkq6b\" (UniqueName: \"kubernetes.io/projected/ef8afda8-47fe-4cab-85a0-01397c9ab56b-kube-api-access-nkq6b\") pod \"maas-api-e2e-shared-a-w5-297806-75b4854d47-mplp7\" (UID: \"ef8afda8-47fe-4cab-85a0-01397c9ab56b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-297806-75b4854d47-mplp7" Nov 04 03:00:13.591169 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:00:13.591134 4405 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ef8afda8-47fe-4cab-85a0-01397c9ab56b-maas-api-tls\") pod \"maas-api-e2e-shared-a-w5-297806-75b4854d47-mplp7\" (UID: \"ef8afda8-47fe-4cab-85a0-01397c9ab56b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-297806-75b4854d47-mplp7" Nov 04 03:00:13.623635 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:00:13.623595 4405 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ef8afda8-47fe-4cab-85a0-01397c9ab56b-maas-api-tls\") pod \"maas-api-e2e-shared-a-w5-297806-75b4854d47-mplp7\" (UID: \"ef8afda8-47fe-4cab-85a0-01397c9ab56b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-297806-75b4854d47-mplp7" Nov 04 03:00:13.756071 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:00:13.756034 4405 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-297806-75b4854d47-mplp7" Nov 04 03:00:14.172592 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:00:14.172559 4405 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-297806-75b4854d47-mplp7"] Nov 04 03:00:14.173066 ip-10-0-134-125 kubenswrapper[4405]: W1104 03:00:14.172684 4405 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podef8afda8_47fe_4cab_85a0_01397c9ab56b.slice/crio-263bc5d988b1174a8029467959f4aa6107635580c0df85dbd8b271f79e3a30ed WatchSource:0}: Error finding container 263bc5d988b1174a8029467959f4aa6107635580c0df85dbd8b271f79e3a30ed: Status 404 returned error can't find the container with id 263bc5d988b1174a8029467959f4aa6107635580c0df85dbd8b271f79e3a30ed Nov 04 03:00:15.039038 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:00:15.039002 4405 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-297806-75b4854d47-mplp7" event={"ID":"ef8afda8-47fe-4cab-85a0-01397c9ab56b","Type":"ContainerStarted","Data":"263bc5d988b1174a8029467959f4aa6107635580c0df85dbd8b271f79e3a30ed"} Nov 04 03:00:15.707697 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:00:15.699205 4405 scope.go:122] "RemoveContainer" containerID="8e423956ee29338dbbb0a910c5dabce023426228649fd31da720e2aa30593364" Nov 04 03:00:16.047703 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:00:16.047649 4405 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-297806-75b4854d47-mplp7" event={"ID":"ef8afda8-47fe-4cab-85a0-01397c9ab56b","Type":"ContainerStarted","Data":"b6b196b6ce9d9840cc4bdd9f34ef1e2c3ab3aef96d0467354500d11dcd08ea8a"} Nov 04 03:00:16.047881 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:00:16.047739 4405 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-297806-75b4854d47-mplp7" Nov 04 03:00:16.171898 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:00:16.171838 4405 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-297806-75b4854d47-mplp7" podStartSLOduration=3.324530007 podStartE2EDuration="4.171817996s" totalImagesPullingTime="847.287989ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 03:00:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 03:00:16.170380725 +0000 UTC m=+1662.601080859" watchObservedRunningTime="2026-11-04 03:00:16.171817996 +0000 UTC m=+1662.602518117" Nov 04 03:00:17.057700 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:00:17.057601 4405 generic.go:356] "Generic (PLEG): container finished" podID="71ce2ec9-0edd-483f-b364-b45e15ccef58" containerID="deb8f75db178d1f864e995e50d48d5acec5fa51f535ab13769ca190a7b42b743" exitCode=60 Nov 04 03:00:17.058118 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:00:17.057702 4405 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-e516ff-29896020-g2kgw" event={"ID":"71ce2ec9-0edd-483f-b364-b45e15ccef58","Type":"ContainerDied","Data":"deb8f75db178d1f864e995e50d48d5acec5fa51f535ab13769ca190a7b42b743"} Nov 04 03:00:17.058118 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:00:17.057770 4405 scope.go:122] "RemoveContainer" containerID="8e423956ee29338dbbb0a910c5dabce023426228649fd31da720e2aa30593364" Nov 04 03:00:17.058435 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:00:17.058418 4405 scope.go:122] "RemoveContainer" containerID="deb8f75db178d1f864e995e50d48d5acec5fa51f535ab13769ca190a7b42b743" Nov 04 03:00:17.058820 ip-10-0-134-125 kubenswrapper[4405]: E1104 03:00:17.058792 4405 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-e516ff-29896020-g2kgw_odh-ai-gateway-infra(71ce2ec9-0edd-483f-b364-b45e15ccef58)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-e516ff-29896020-g2kgw" podUID="71ce2ec9-0edd-483f-b364-b45e15ccef58" Nov 04 03:00:17.694569 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:00:17.694541 4405 scope.go:122] "RemoveContainer" containerID="2d6a63a5a2cbed4d010a0ed3b9e4ce2c04122d4bd9cdbc54ac109c98dab39fd8" Nov 04 03:00:17.694812 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:00:17.694741 4405 scope.go:122] "RemoveContainer" containerID="42a12a4580408dc96071f5ac20370f496d168fefec440ced8f164123bbb72006" Nov 04 03:00:18.188755 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:00:18.188719 4405 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-e516ff-29896020-g2kgw"] Nov 04 03:00:18.412046 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:00:18.412019 4405 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-e516ff-29896020-g2kgw" Nov 04 03:00:18.520526 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:00:18.520468 4405 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/71ce2ec9-0edd-483f-b364-b45e15ccef58-kube-api-access-9s242\" (UniqueName: \"kubernetes.io/projected/71ce2ec9-0edd-483f-b364-b45e15ccef58-kube-api-access-9s242\") pod \"71ce2ec9-0edd-483f-b364-b45e15ccef58\" (UID: \"71ce2ec9-0edd-483f-b364-b45e15ccef58\") " Nov 04 03:00:18.520783 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:00:18.520660 4405 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/71ce2ec9-0edd-483f-b364-b45e15ccef58-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/71ce2ec9-0edd-483f-b364-b45e15ccef58-maas-api-service-ca\") pod \"71ce2ec9-0edd-483f-b364-b45e15ccef58\" (UID: \"71ce2ec9-0edd-483f-b364-b45e15ccef58\") " Nov 04 03:00:18.521127 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:00:18.521098 4405 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/71ce2ec9-0edd-483f-b364-b45e15ccef58-maas-api-service-ca" pod "71ce2ec9-0edd-483f-b364-b45e15ccef58" (UID: "71ce2ec9-0edd-483f-b364-b45e15ccef58"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 03:00:18.521569 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:00:18.521548 4405 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/71ce2ec9-0edd-483f-b364-b45e15ccef58-maas-api-service-ca\") on node \"ip-10-0-134-125.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:00:18.523146 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:00:18.523115 4405 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/71ce2ec9-0edd-483f-b364-b45e15ccef58-kube-api-access-9s242" pod "71ce2ec9-0edd-483f-b364-b45e15ccef58" (UID: "71ce2ec9-0edd-483f-b364-b45e15ccef58"). InnerVolumeSpecName "kube-api-access-9s242". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 03:00:18.624709 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:00:18.624583 4405 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9s242\" (UniqueName: \"kubernetes.io/projected/71ce2ec9-0edd-483f-b364-b45e15ccef58-kube-api-access-9s242\") on node \"ip-10-0-134-125.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:00:18.694722 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:00:18.694562 4405 scope.go:122] "RemoveContainer" containerID="c9d7d353156d315e5a70ef6191e4690f8c72320c75bfc164e90969288904e787" Nov 04 03:00:19.084426 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:00:19.084387 4405 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-e516ff-29896020-g2kgw" event={"ID":"71ce2ec9-0edd-483f-b364-b45e15ccef58","Type":"ContainerDied","Data":"f4b06b5fdd03c26b625e95745a85d0ee8df257e4a4baaeb89f71bf87850b882b"} Nov 04 03:00:19.084426 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:00:19.084418 4405 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-e516ff-29896020-g2kgw" Nov 04 03:00:19.084715 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:00:19.084467 4405 scope.go:122] "RemoveContainer" containerID="deb8f75db178d1f864e995e50d48d5acec5fa51f535ab13769ca190a7b42b743" Nov 04 03:00:19.091295 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:00:19.091264 4405 generic.go:356] "Generic (PLEG): container finished" podID="f61715a2-4d72-491b-8236-3e4a71462e60" containerID="674d1bf21f8aad9bc661014ccda48238adfd0fb617161843937c352f16c7e01d" exitCode=60 Nov 04 03:00:19.091408 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:00:19.091340 4405 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-203c35-29896020-ftklh" event={"ID":"f61715a2-4d72-491b-8236-3e4a71462e60","Type":"ContainerDied","Data":"674d1bf21f8aad9bc661014ccda48238adfd0fb617161843937c352f16c7e01d"} Nov 04 03:00:19.091900 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:00:19.091881 4405 scope.go:122] "RemoveContainer" containerID="674d1bf21f8aad9bc661014ccda48238adfd0fb617161843937c352f16c7e01d" Nov 04 03:00:19.092277 ip-10-0-134-125 kubenswrapper[4405]: E1104 03:00:19.092254 4405 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-203c35-29896020-ftklh_odh-ai-gateway-infra(f61715a2-4d72-491b-8236-3e4a71462e60)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-203c35-29896020-ftklh" podUID="f61715a2-4d72-491b-8236-3e4a71462e60" Nov 04 03:00:19.094442 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:00:19.094284 4405 generic.go:356] "Generic (PLEG): container finished" podID="42766046-4bb9-4afd-8fe7-3bc0e0199d1e" containerID="adedbddf16dafa86c3457a58f9dc9b1c6d430464ebb8c078e6d00a86d86cd3aa" exitCode=60 Nov 04 03:00:19.094442 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:00:19.094343 4405 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-072ece-29896020-nlhnv" event={"ID":"42766046-4bb9-4afd-8fe7-3bc0e0199d1e","Type":"ContainerDied","Data":"adedbddf16dafa86c3457a58f9dc9b1c6d430464ebb8c078e6d00a86d86cd3aa"} Nov 04 03:00:19.094950 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:00:19.094931 4405 scope.go:122] "RemoveContainer" containerID="adedbddf16dafa86c3457a58f9dc9b1c6d430464ebb8c078e6d00a86d86cd3aa" Nov 04 03:00:19.095304 ip-10-0-134-125 kubenswrapper[4405]: E1104 03:00:19.095281 4405 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w4-072ece-29896020-nlhnv_odh-ai-gateway-infra(42766046-4bb9-4afd-8fe7-3bc0e0199d1e)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-072ece-29896020-nlhnv" podUID="42766046-4bb9-4afd-8fe7-3bc0e0199d1e" Nov 04 03:00:19.100169 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:00:19.100152 4405 scope.go:122] "RemoveContainer" containerID="42a12a4580408dc96071f5ac20370f496d168fefec440ced8f164123bbb72006" Nov 04 03:00:19.117005 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:00:19.116970 4405 scope.go:122] "RemoveContainer" containerID="2d6a63a5a2cbed4d010a0ed3b9e4ce2c04122d4bd9cdbc54ac109c98dab39fd8" Nov 04 03:00:19.254757 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:00:19.254722 4405 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-e516ff-29896020-g2kgw"] Nov 04 03:00:19.259637 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:00:19.259612 4405 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-e516ff-29896020-g2kgw"] Nov 04 03:00:19.810765 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:00:19.810105 4405 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="71ce2ec9-0edd-483f-b364-b45e15ccef58" path="/var/lib/kubelet/pods/71ce2ec9-0edd-483f-b364-b45e15ccef58/volumes" Nov 04 03:00:20.110294 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:00:20.109230 4405 generic.go:356] "Generic (PLEG): container finished" podID="1d9a643a-06d7-4e7e-85bb-a87e2108864f" containerID="548f59f6b8caa8c22d3a429233c484d904a077d9880090ada85930edebe759e7" exitCode=60 Nov 04 03:00:20.110294 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:00:20.109417 4405 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-derive-5fde04ee-29896020-df5j6" event={"ID":"1d9a643a-06d7-4e7e-85bb-a87e2108864f","Type":"ContainerDied","Data":"548f59f6b8caa8c22d3a429233c484d904a077d9880090ada85930edebe759e7"} Nov 04 03:00:20.110294 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:00:20.109497 4405 scope.go:122] "RemoveContainer" containerID="c9d7d353156d315e5a70ef6191e4690f8c72320c75bfc164e90969288904e787" Nov 04 03:00:20.110294 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:00:20.110151 4405 scope.go:122] "RemoveContainer" containerID="548f59f6b8caa8c22d3a429233c484d904a077d9880090ada85930edebe759e7" Nov 04 03:00:20.110728 ip-10-0-134-125 kubenswrapper[4405]: E1104 03:00:20.110629 4405 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-derive-5fde04ee-29896020-df5j6_odh-ai-gateway-infra(1d9a643a-06d7-4e7e-85bb-a87e2108864f)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-derive-5fde04ee-29896020-df5j6" podUID="1d9a643a-06d7-4e7e-85bb-a87e2108864f" Nov 04 03:00:20.379765 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:00:20.379654 4405 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-072ece-29896020-nlhnv"] Nov 04 03:00:20.500888 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:00:20.500853 4405 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-203c35-29896020-ftklh"] Nov 04 03:00:20.582711 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:00:20.581013 4405 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-072ece-29896020-nlhnv" Nov 04 03:00:20.700388 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:00:20.680807 4405 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/42766046-4bb9-4afd-8fe7-3bc0e0199d1e-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/42766046-4bb9-4afd-8fe7-3bc0e0199d1e-maas-api-service-ca\") pod \"42766046-4bb9-4afd-8fe7-3bc0e0199d1e\" (UID: \"42766046-4bb9-4afd-8fe7-3bc0e0199d1e\") " Nov 04 03:00:20.700388 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:00:20.680873 4405 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/42766046-4bb9-4afd-8fe7-3bc0e0199d1e-kube-api-access-2vcgs\" (UniqueName: \"kubernetes.io/projected/42766046-4bb9-4afd-8fe7-3bc0e0199d1e-kube-api-access-2vcgs\") pod \"42766046-4bb9-4afd-8fe7-3bc0e0199d1e\" (UID: \"42766046-4bb9-4afd-8fe7-3bc0e0199d1e\") " Nov 04 03:00:20.716113 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:00:20.716069 4405 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/42766046-4bb9-4afd-8fe7-3bc0e0199d1e-maas-api-service-ca" pod "42766046-4bb9-4afd-8fe7-3bc0e0199d1e" (UID: "42766046-4bb9-4afd-8fe7-3bc0e0199d1e"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 03:00:20.733409 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:00:20.733364 4405 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/42766046-4bb9-4afd-8fe7-3bc0e0199d1e-kube-api-access-2vcgs" pod "42766046-4bb9-4afd-8fe7-3bc0e0199d1e" (UID: "42766046-4bb9-4afd-8fe7-3bc0e0199d1e"). InnerVolumeSpecName "kube-api-access-2vcgs". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 03:00:20.773590 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:00:20.773471 4405 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-203c35-29896020-ftklh" Nov 04 03:00:20.806173 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:00:20.806136 4405 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/42766046-4bb9-4afd-8fe7-3bc0e0199d1e-maas-api-service-ca\") on node \"ip-10-0-134-125.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:00:20.806173 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:00:20.806175 4405 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2vcgs\" (UniqueName: \"kubernetes.io/projected/42766046-4bb9-4afd-8fe7-3bc0e0199d1e-kube-api-access-2vcgs\") on node \"ip-10-0-134-125.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:00:20.907368 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:00:20.907330 4405 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f61715a2-4d72-491b-8236-3e4a71462e60-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f61715a2-4d72-491b-8236-3e4a71462e60-maas-api-service-ca\") pod \"f61715a2-4d72-491b-8236-3e4a71462e60\" (UID: \"f61715a2-4d72-491b-8236-3e4a71462e60\") " Nov 04 03:00:20.907764 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:00:20.907739 4405 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f61715a2-4d72-491b-8236-3e4a71462e60-kube-api-access-nbth7\" (UniqueName: \"kubernetes.io/projected/f61715a2-4d72-491b-8236-3e4a71462e60-kube-api-access-nbth7\") pod \"f61715a2-4d72-491b-8236-3e4a71462e60\" (UID: \"f61715a2-4d72-491b-8236-3e4a71462e60\") " Nov 04 03:00:20.908462 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:00:20.908434 4405 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f61715a2-4d72-491b-8236-3e4a71462e60-maas-api-service-ca" pod "f61715a2-4d72-491b-8236-3e4a71462e60" (UID: "f61715a2-4d72-491b-8236-3e4a71462e60"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 03:00:20.910966 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:00:20.910920 4405 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f61715a2-4d72-491b-8236-3e4a71462e60-kube-api-access-nbth7" pod "f61715a2-4d72-491b-8236-3e4a71462e60" (UID: "f61715a2-4d72-491b-8236-3e4a71462e60"). InnerVolumeSpecName "kube-api-access-nbth7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 03:00:21.010139 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:00:21.010051 4405 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nbth7\" (UniqueName: \"kubernetes.io/projected/f61715a2-4d72-491b-8236-3e4a71462e60-kube-api-access-nbth7\") on node \"ip-10-0-134-125.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:00:21.010366 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:00:21.010349 4405 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f61715a2-4d72-491b-8236-3e4a71462e60-maas-api-service-ca\") on node \"ip-10-0-134-125.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:00:21.150532 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:00:21.150502 4405 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-203c35-29896020-ftklh" Nov 04 03:00:21.150733 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:00:21.150535 4405 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-203c35-29896020-ftklh" event={"ID":"f61715a2-4d72-491b-8236-3e4a71462e60","Type":"ContainerDied","Data":"5996b6fa787cc0eb6d723150fdb631e80e8ff997e0a02250fe6b9eb191703cba"} Nov 04 03:00:21.150733 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:00:21.150608 4405 scope.go:122] "RemoveContainer" containerID="674d1bf21f8aad9bc661014ccda48238adfd0fb617161843937c352f16c7e01d" Nov 04 03:00:21.152241 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:00:21.152224 4405 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-072ece-29896020-nlhnv" Nov 04 03:00:21.152342 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:00:21.152240 4405 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-072ece-29896020-nlhnv" event={"ID":"42766046-4bb9-4afd-8fe7-3bc0e0199d1e","Type":"ContainerDied","Data":"01e39e96dcfe751bd0da070bda6802271a43b681a243c74a2c8138914b24b470"} Nov 04 03:00:21.164886 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:00:21.164866 4405 scope.go:122] "RemoveContainer" containerID="adedbddf16dafa86c3457a58f9dc9b1c6d430464ebb8c078e6d00a86d86cd3aa" Nov 04 03:00:21.238543 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:00:21.238501 4405 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-derive-5fde04ee-29896020-df5j6"] Nov 04 03:00:21.320416 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:00:21.320382 4405 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-203c35-29896020-ftklh"] Nov 04 03:00:21.332830 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:00:21.332795 4405 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-203c35-29896020-ftklh"] Nov 04 03:00:21.454647 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:00:21.454617 4405 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-072ece-29896020-nlhnv"] Nov 04 03:00:21.460439 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:00:21.460411 4405 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-072ece-29896020-nlhnv"] Nov 04 03:00:21.467179 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:00:21.467154 4405 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-derive-5fde04ee-29896020-df5j6" Nov 04 03:00:21.619786 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:00:21.619755 4405 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1d9a643a-06d7-4e7e-85bb-a87e2108864f-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1d9a643a-06d7-4e7e-85bb-a87e2108864f-maas-api-service-ca\") pod \"1d9a643a-06d7-4e7e-85bb-a87e2108864f\" (UID: \"1d9a643a-06d7-4e7e-85bb-a87e2108864f\") " Nov 04 03:00:21.620002 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:00:21.619987 4405 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1d9a643a-06d7-4e7e-85bb-a87e2108864f-kube-api-access-lrq87\" (UniqueName: \"kubernetes.io/projected/1d9a643a-06d7-4e7e-85bb-a87e2108864f-kube-api-access-lrq87\") pod \"1d9a643a-06d7-4e7e-85bb-a87e2108864f\" (UID: \"1d9a643a-06d7-4e7e-85bb-a87e2108864f\") " Nov 04 03:00:21.620157 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:00:21.620130 4405 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1d9a643a-06d7-4e7e-85bb-a87e2108864f-maas-api-service-ca" pod "1d9a643a-06d7-4e7e-85bb-a87e2108864f" (UID: "1d9a643a-06d7-4e7e-85bb-a87e2108864f"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 03:00:21.620507 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:00:21.620482 4405 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1d9a643a-06d7-4e7e-85bb-a87e2108864f-maas-api-service-ca\") on node \"ip-10-0-134-125.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:00:21.622362 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:00:21.622342 4405 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1d9a643a-06d7-4e7e-85bb-a87e2108864f-kube-api-access-lrq87" pod "1d9a643a-06d7-4e7e-85bb-a87e2108864f" (UID: "1d9a643a-06d7-4e7e-85bb-a87e2108864f"). InnerVolumeSpecName "kube-api-access-lrq87". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 03:00:21.706359 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:00:21.706317 4405 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="42766046-4bb9-4afd-8fe7-3bc0e0199d1e" path="/var/lib/kubelet/pods/42766046-4bb9-4afd-8fe7-3bc0e0199d1e/volumes" Nov 04 03:00:21.706854 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:00:21.706836 4405 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f61715a2-4d72-491b-8236-3e4a71462e60" path="/var/lib/kubelet/pods/f61715a2-4d72-491b-8236-3e4a71462e60/volumes" Nov 04 03:00:21.722481 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:00:21.722456 4405 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lrq87\" (UniqueName: \"kubernetes.io/projected/1d9a643a-06d7-4e7e-85bb-a87e2108864f-kube-api-access-lrq87\") on node \"ip-10-0-134-125.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:00:21.799215 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:00:21.799184 4405 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-4d5a0200-5cdf8ddbcd-wdl8z"] Nov 04 03:00:21.903826 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:00:21.903728 4405 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-4d5a0200-5959b9699-k2dpp"] Nov 04 03:00:22.063776 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:00:22.063742 4405 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-297806-75b4854d47-mplp7" Nov 04 03:00:22.168236 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:00:22.168132 4405 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-derive-5fde04ee-29896020-df5j6" event={"ID":"1d9a643a-06d7-4e7e-85bb-a87e2108864f","Type":"ContainerDied","Data":"80b226341fc18d9e2732331f534b280fe07f7abe3d0ddc2ab5cae654b54d1e2f"} Nov 04 03:00:22.168236 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:00:22.168215 4405 scope.go:122] "RemoveContainer" containerID="548f59f6b8caa8c22d3a429233c484d904a077d9880090ada85930edebe759e7" Nov 04 03:00:22.168418 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:00:22.168312 4405 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-derive-5fde04ee-29896020-df5j6" Nov 04 03:00:22.180460 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:00:22.180431 4405 generic.go:356] "Generic (PLEG): container finished" podID="652d018b-8d07-49ae-b9fb-a87cfc70b0c8" containerID="a30b7638525ea4ce26e47f3f8e99113b4d9df664c679a6e02855af22859fe884" exitCode=0 Nov 04 03:00:22.180607 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:00:22.180501 4405 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-072ece-openshift-default-78b784fd8c-k2vfq" event={"ID":"652d018b-8d07-49ae-b9fb-a87cfc70b0c8","Type":"ContainerDied","Data":"a30b7638525ea4ce26e47f3f8e99113b4d9df664c679a6e02855af22859fe884"} Nov 04 03:00:22.181140 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:00:22.181118 4405 scope.go:122] "RemoveContainer" containerID="a30b7638525ea4ce26e47f3f8e99113b4d9df664c679a6e02855af22859fe884" Nov 04 03:00:22.183217 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:00:22.183189 4405 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 04 03:00:22.183321 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:00:22.183262 4405 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 04 03:00:22.183380 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:00:22.183320 4405 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 04 03:00:22.462804 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:00:22.455156 4405 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-derive-5fde04ee-29896020-df5j6"] Nov 04 03:00:22.462804 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:00:22.462438 4405 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-derive-5fde04ee-29896020-df5j6"] Nov 04 03:00:23.377428 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:00:23.377378 4405 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-072ece-openshift-default-78b784fd8c-k2vfq" event={"ID":"652d018b-8d07-49ae-b9fb-a87cfc70b0c8","Type":"ContainerStarted","Data":"9c2f7cc08dbbb4aa2b4f90dd96c5db012ffa85b0590ed3c70e56029b97f22278"} Nov 04 03:00:23.541689 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:00:23.541096 4405 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w4-072ece-openshift-default-78b784fd8c-k2vfq" Nov 04 03:00:23.544769 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:00:23.543793 4405 patch_prober.go:29] interesting pod/e2e-shared-a-w4-072ece-openshift-default-78b784fd8c-k2vfq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.89:15021/healthz/ready\": dial tcp 10.134.0.89:15021: connect: connection refused" start-of-body= Nov 04 03:00:23.544769 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:00:23.543865 4405 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-072ece-openshift-default-78b784fd8c-k2vfq" podUID="652d018b-8d07-49ae-b9fb-a87cfc70b0c8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.89:15021/healthz/ready\": dial tcp 10.134.0.89:15021: connect: connection refused" Nov 04 03:00:23.779749 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:00:23.779217 4405 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1d9a643a-06d7-4e7e-85bb-a87e2108864f" path="/var/lib/kubelet/pods/1d9a643a-06d7-4e7e-85bb-a87e2108864f/volumes" Nov 04 03:00:24.539923 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:00:24.539880 4405 patch_prober.go:29] interesting pod/e2e-shared-a-w4-072ece-openshift-default-78b784fd8c-k2vfq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.89:15021/healthz/ready\": dial tcp 10.134.0.89:15021: connect: connection refused" start-of-body= Nov 04 03:00:24.540118 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:00:24.539965 4405 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-072ece-openshift-default-78b784fd8c-k2vfq" podUID="652d018b-8d07-49ae-b9fb-a87cfc70b0c8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.89:15021/healthz/ready\": dial tcp 10.134.0.89:15021: connect: connection refused" Nov 04 03:00:25.371882 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:00:25.371846 4405 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w4-072ece-openshift-default-78b784fd8c-k2vfq" Nov 04 03:00:25.540244 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:00:25.540208 4405 patch_prober.go:29] interesting pod/e2e-shared-a-w4-072ece-openshift-default-78b784fd8c-k2vfq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.89:15021/healthz/ready\": dial tcp 10.134.0.89:15021: connect: connection refused" start-of-body= Nov 04 03:00:25.540420 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:00:25.540276 4405 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-072ece-openshift-default-78b784fd8c-k2vfq" podUID="652d018b-8d07-49ae-b9fb-a87cfc70b0c8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.89:15021/healthz/ready\": dial tcp 10.134.0.89:15021: connect: connection refused" Nov 04 03:00:26.539999 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:00:26.539965 4405 patch_prober.go:29] interesting pod/e2e-shared-a-w4-072ece-openshift-default-78b784fd8c-k2vfq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.89:15021/healthz/ready\": dial tcp 10.134.0.89:15021: connect: connection refused" start-of-body= Nov 04 03:00:26.540423 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:00:26.540018 4405 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-072ece-openshift-default-78b784fd8c-k2vfq" podUID="652d018b-8d07-49ae-b9fb-a87cfc70b0c8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.89:15021/healthz/ready\": dial tcp 10.134.0.89:15021: connect: connection refused" Nov 04 03:00:26.806878 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:00:26.806761 4405 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ait-4d5a0200-5cdf8ddbcd-wdl8z" podUID="e0691251-c255-4643-a6c4-4dacae0f15b3" containerName="payload-processing" containerID="cri-o://a1567458568b76116a8757e8b5c54f3a6211b21f30835edc8dc1b759b23f4db8" gracePeriod=25 Nov 04 03:00:26.914696 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:00:26.913701 4405 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ait-4d5a0200-5959b9699-k2dpp" podUID="3e760665-863f-41a0-b2fb-7b33a1768fc6" containerName="payload-pre-processing" containerID="cri-o://ca362cfaa24e49540dd86bfd538d1fb23eec5fda60f69950cf92551176119618" gracePeriod=25 Nov 04 03:00:27.226003 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:00:27.225965 4405 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-4d5a0200-5cdf8ddbcd-wdl8z" Nov 04 03:00:27.330984 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:00:27.330687 4405 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e0691251-c255-4643-a6c4-4dacae0f15b3-kube-api-access-k7bkq\" (UniqueName: \"kubernetes.io/projected/e0691251-c255-4643-a6c4-4dacae0f15b3-kube-api-access-k7bkq\") pod \"e0691251-c255-4643-a6c4-4dacae0f15b3\" (UID: \"e0691251-c255-4643-a6c4-4dacae0f15b3\") " Nov 04 03:00:27.330984 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:00:27.330769 4405 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e0691251-c255-4643-a6c4-4dacae0f15b3-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e0691251-c255-4643-a6c4-4dacae0f15b3-plugins-config-volume\") pod \"e0691251-c255-4643-a6c4-4dacae0f15b3\" (UID: \"e0691251-c255-4643-a6c4-4dacae0f15b3\") " Nov 04 03:00:27.331682 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:00:27.331343 4405 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e0691251-c255-4643-a6c4-4dacae0f15b3-plugins-config-volume" pod "e0691251-c255-4643-a6c4-4dacae0f15b3" (UID: "e0691251-c255-4643-a6c4-4dacae0f15b3"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 03:00:27.332349 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:00:27.332262 4405 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e0691251-c255-4643-a6c4-4dacae0f15b3-plugins-config-volume\") on node \"ip-10-0-134-125.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:00:27.336279 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:00:27.336241 4405 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e0691251-c255-4643-a6c4-4dacae0f15b3-kube-api-access-k7bkq" pod "e0691251-c255-4643-a6c4-4dacae0f15b3" (UID: "e0691251-c255-4643-a6c4-4dacae0f15b3"). InnerVolumeSpecName "kube-api-access-k7bkq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 03:00:27.339633 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:00:27.339609 4405 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-4d5a0200-5959b9699-k2dpp" Nov 04 03:00:27.432977 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:00:27.432873 4405 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3e760665-863f-41a0-b2fb-7b33a1768fc6-kube-api-access-mz87d\" (UniqueName: \"kubernetes.io/projected/3e760665-863f-41a0-b2fb-7b33a1768fc6-kube-api-access-mz87d\") pod \"3e760665-863f-41a0-b2fb-7b33a1768fc6\" (UID: \"3e760665-863f-41a0-b2fb-7b33a1768fc6\") " Nov 04 03:00:27.433377 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:00:27.433350 4405 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3e760665-863f-41a0-b2fb-7b33a1768fc6-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3e760665-863f-41a0-b2fb-7b33a1768fc6-plugins-config-volume\") pod \"3e760665-863f-41a0-b2fb-7b33a1768fc6\" (UID: \"3e760665-863f-41a0-b2fb-7b33a1768fc6\") " Nov 04 03:00:27.434081 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:00:27.434045 4405 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k7bkq\" (UniqueName: \"kubernetes.io/projected/e0691251-c255-4643-a6c4-4dacae0f15b3-kube-api-access-k7bkq\") on node \"ip-10-0-134-125.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:00:27.434503 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:00:27.434480 4405 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3e760665-863f-41a0-b2fb-7b33a1768fc6-plugins-config-volume" pod "3e760665-863f-41a0-b2fb-7b33a1768fc6" (UID: "3e760665-863f-41a0-b2fb-7b33a1768fc6"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 03:00:27.438022 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:00:27.437980 4405 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3e760665-863f-41a0-b2fb-7b33a1768fc6-kube-api-access-mz87d" pod "3e760665-863f-41a0-b2fb-7b33a1768fc6" (UID: "3e760665-863f-41a0-b2fb-7b33a1768fc6"). InnerVolumeSpecName "kube-api-access-mz87d". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 03:00:27.440363 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:00:27.440333 4405 generic.go:356] "Generic (PLEG): container finished" podID="e0691251-c255-4643-a6c4-4dacae0f15b3" containerID="a1567458568b76116a8757e8b5c54f3a6211b21f30835edc8dc1b759b23f4db8" exitCode=0 Nov 04 03:00:27.440468 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:00:27.440413 4405 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-4d5a0200-5cdf8ddbcd-wdl8z" event={"ID":"e0691251-c255-4643-a6c4-4dacae0f15b3","Type":"ContainerDied","Data":"a1567458568b76116a8757e8b5c54f3a6211b21f30835edc8dc1b759b23f4db8"} Nov 04 03:00:27.440533 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:00:27.440466 4405 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-4d5a0200-5cdf8ddbcd-wdl8z" event={"ID":"e0691251-c255-4643-a6c4-4dacae0f15b3","Type":"ContainerDied","Data":"0513aaa17b1d480652b36b285731d2139f531f052bfc913cd02f4116dc68d099"} Nov 04 03:00:27.440533 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:00:27.440495 4405 scope.go:122] "RemoveContainer" containerID="a1567458568b76116a8757e8b5c54f3a6211b21f30835edc8dc1b759b23f4db8" Nov 04 03:00:27.440736 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:00:27.440630 4405 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-4d5a0200-5cdf8ddbcd-wdl8z" Nov 04 03:00:27.445405 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:00:27.445368 4405 generic.go:356] "Generic (PLEG): container finished" podID="3e760665-863f-41a0-b2fb-7b33a1768fc6" containerID="ca362cfaa24e49540dd86bfd538d1fb23eec5fda60f69950cf92551176119618" exitCode=0 Nov 04 03:00:27.445517 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:00:27.445440 4405 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-4d5a0200-5959b9699-k2dpp" event={"ID":"3e760665-863f-41a0-b2fb-7b33a1768fc6","Type":"ContainerDied","Data":"ca362cfaa24e49540dd86bfd538d1fb23eec5fda60f69950cf92551176119618"} Nov 04 03:00:27.445517 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:00:27.445483 4405 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-4d5a0200-5959b9699-k2dpp" event={"ID":"3e760665-863f-41a0-b2fb-7b33a1768fc6","Type":"ContainerDied","Data":"35e93fcd052b6c43ff9684e508be59b81e5be782f56a0119447497342cfbd58f"} Nov 04 03:00:27.445636 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:00:27.445549 4405 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-4d5a0200-5959b9699-k2dpp" Nov 04 03:00:27.458769 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:00:27.458748 4405 scope.go:122] "RemoveContainer" containerID="a1567458568b76116a8757e8b5c54f3a6211b21f30835edc8dc1b759b23f4db8" Nov 04 03:00:27.459184 ip-10-0-134-125 kubenswrapper[4405]: E1104 03:00:27.459095 4405 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a1567458568b76116a8757e8b5c54f3a6211b21f30835edc8dc1b759b23f4db8\": container with ID starting with a1567458568b76116a8757e8b5c54f3a6211b21f30835edc8dc1b759b23f4db8 not found: ID does not exist" containerID="a1567458568b76116a8757e8b5c54f3a6211b21f30835edc8dc1b759b23f4db8" Nov 04 03:00:27.459184 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:00:27.459130 4405 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a1567458568b76116a8757e8b5c54f3a6211b21f30835edc8dc1b759b23f4db8"} err="failed to get container status \"a1567458568b76116a8757e8b5c54f3a6211b21f30835edc8dc1b759b23f4db8\": rpc error: code = NotFound desc = could not find container \"a1567458568b76116a8757e8b5c54f3a6211b21f30835edc8dc1b759b23f4db8\": container with ID starting with a1567458568b76116a8757e8b5c54f3a6211b21f30835edc8dc1b759b23f4db8 not found: ID does not exist" Nov 04 03:00:27.459184 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:00:27.459155 4405 scope.go:122] "RemoveContainer" containerID="ca362cfaa24e49540dd86bfd538d1fb23eec5fda60f69950cf92551176119618" Nov 04 03:00:27.473367 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:00:27.473322 4405 scope.go:122] "RemoveContainer" containerID="ca362cfaa24e49540dd86bfd538d1fb23eec5fda60f69950cf92551176119618" Nov 04 03:00:27.473799 ip-10-0-134-125 kubenswrapper[4405]: E1104 03:00:27.473767 4405 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ca362cfaa24e49540dd86bfd538d1fb23eec5fda60f69950cf92551176119618\": container with ID starting with ca362cfaa24e49540dd86bfd538d1fb23eec5fda60f69950cf92551176119618 not found: ID does not exist" containerID="ca362cfaa24e49540dd86bfd538d1fb23eec5fda60f69950cf92551176119618" Nov 04 03:00:27.473973 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:00:27.473942 4405 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ca362cfaa24e49540dd86bfd538d1fb23eec5fda60f69950cf92551176119618"} err="failed to get container status \"ca362cfaa24e49540dd86bfd538d1fb23eec5fda60f69950cf92551176119618\": rpc error: code = NotFound desc = could not find container \"ca362cfaa24e49540dd86bfd538d1fb23eec5fda60f69950cf92551176119618\": container with ID starting with ca362cfaa24e49540dd86bfd538d1fb23eec5fda60f69950cf92551176119618 not found: ID does not exist" Nov 04 03:00:27.536554 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:00:27.536517 4405 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3e760665-863f-41a0-b2fb-7b33a1768fc6-plugins-config-volume\") on node \"ip-10-0-134-125.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:00:27.536820 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:00:27.536798 4405 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mz87d\" (UniqueName: \"kubernetes.io/projected/3e760665-863f-41a0-b2fb-7b33a1768fc6-kube-api-access-mz87d\") on node \"ip-10-0-134-125.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:00:27.539566 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:00:27.539533 4405 patch_prober.go:29] interesting pod/e2e-shared-a-w4-072ece-openshift-default-78b784fd8c-k2vfq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.89:15021/healthz/ready\": dial tcp 10.134.0.89:15021: connect: connection refused" start-of-body= Nov 04 03:00:27.539728 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:00:27.539607 4405 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-072ece-openshift-default-78b784fd8c-k2vfq" podUID="652d018b-8d07-49ae-b9fb-a87cfc70b0c8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.89:15021/healthz/ready\": dial tcp 10.134.0.89:15021: connect: connection refused" Nov 04 03:00:27.628845 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:00:27.628801 4405 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-4d5a0200-5cdf8ddbcd-wdl8z"] Nov 04 03:00:27.632572 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:00:27.632542 4405 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-4d5a0200-5cdf8ddbcd-wdl8z"] Nov 04 03:00:27.729094 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:00:27.729002 4405 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e0691251-c255-4643-a6c4-4dacae0f15b3" path="/var/lib/kubelet/pods/e0691251-c255-4643-a6c4-4dacae0f15b3/volumes" Nov 04 03:00:27.731882 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:00:27.731858 4405 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-4d5a0200-5959b9699-k2dpp"] Nov 04 03:00:27.735885 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:00:27.735855 4405 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-4d5a0200-5959b9699-k2dpp"] Nov 04 03:00:28.539981 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:00:28.539893 4405 patch_prober.go:29] interesting pod/e2e-shared-a-w4-072ece-openshift-default-78b784fd8c-k2vfq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.89:15021/healthz/ready\": dial tcp 10.134.0.89:15021: connect: connection refused" start-of-body= Nov 04 03:00:28.539981 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:00:28.539966 4405 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-072ece-openshift-default-78b784fd8c-k2vfq" podUID="652d018b-8d07-49ae-b9fb-a87cfc70b0c8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.89:15021/healthz/ready\": dial tcp 10.134.0.89:15021: connect: connection refused" Nov 04 03:00:29.544420 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:00:29.544388 4405 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w4-072ece-openshift-default-78b784fd8c-k2vfq" Nov 04 03:00:29.545635 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:00:29.545613 4405 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w4-072ece-openshift-default-78b784fd8c-k2vfq" Nov 04 03:00:29.706628 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:00:29.706595 4405 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3e760665-863f-41a0-b2fb-7b33a1768fc6" path="/var/lib/kubelet/pods/3e760665-863f-41a0-b2fb-7b33a1768fc6/volumes" Nov 04 03:00:36.884523 ip-10-0-134-125 kubenswrapper[4405]: E1104 03:00:36.884485 4405 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5f2vb" containerName="extractor" Nov 04 03:00:38.230741 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:00:38.222095 4405 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-5fde04ee-5fdc7f866d-lk25n"] Nov 04 03:00:38.315774 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:00:38.315732 4405 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-5fde04ee-59df47b5d7-gnjpp"] Nov 04 03:00:43.223115 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:00:43.223062 4405 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-derive-5fde04ee-5fdc7f866d-lk25n" podUID="100fce2c-765a-4370-ac93-8dcbca038872" containerName="payload-processing" containerID="cri-o://37dfe03c937c3c23aad39a11e25a7dbca9e449ae2b01a890d6664f9bbc3e96f5" gracePeriod=25 Nov 04 03:00:43.316529 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:00:43.316478 4405 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-derive-5fde04ee-59df47b5d7-gnjpp" podUID="82ad73c2-7f63-43cb-a8af-5f17c4621c67" containerName="payload-pre-processing" containerID="cri-o://c88e436d48f06c7058f5cfef43ca5f7d5826b286ef36b16bbc264934bab87401" gracePeriod=25 Nov 04 03:00:43.618753 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:00:43.611284 4405 generic.go:356] "Generic (PLEG): container finished" podID="100fce2c-765a-4370-ac93-8dcbca038872" containerID="37dfe03c937c3c23aad39a11e25a7dbca9e449ae2b01a890d6664f9bbc3e96f5" exitCode=0 Nov 04 03:00:43.618753 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:00:43.611537 4405 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-5fde04ee-5fdc7f866d-lk25n" event={"ID":"100fce2c-765a-4370-ac93-8dcbca038872","Type":"ContainerDied","Data":"37dfe03c937c3c23aad39a11e25a7dbca9e449ae2b01a890d6664f9bbc3e96f5"} Nov 04 03:00:43.618753 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:00:43.611591 4405 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-5fde04ee-5fdc7f866d-lk25n" event={"ID":"100fce2c-765a-4370-ac93-8dcbca038872","Type":"ContainerDied","Data":"84256fdcffd99f447f7ecf8909df13801038db51d41513e1095433fff74d5002"} Nov 04 03:00:43.618753 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:00:43.611615 4405 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="84256fdcffd99f447f7ecf8909df13801038db51d41513e1095433fff74d5002" Nov 04 03:00:43.618753 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:00:43.613509 4405 generic.go:356] "Generic (PLEG): container finished" podID="82ad73c2-7f63-43cb-a8af-5f17c4621c67" containerID="c88e436d48f06c7058f5cfef43ca5f7d5826b286ef36b16bbc264934bab87401" exitCode=0 Nov 04 03:00:43.618753 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:00:43.613542 4405 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-5fde04ee-59df47b5d7-gnjpp" event={"ID":"82ad73c2-7f63-43cb-a8af-5f17c4621c67","Type":"ContainerDied","Data":"c88e436d48f06c7058f5cfef43ca5f7d5826b286ef36b16bbc264934bab87401"} Nov 04 03:00:43.639069 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:00:43.636367 4405 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-5fde04ee-5fdc7f866d-lk25n" Nov 04 03:00:43.674079 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:00:43.674052 4405 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-5fde04ee-59df47b5d7-gnjpp" Nov 04 03:00:43.758365 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:00:43.758270 4405 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/100fce2c-765a-4370-ac93-8dcbca038872-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/100fce2c-765a-4370-ac93-8dcbca038872-plugins-config-volume\") pod \"100fce2c-765a-4370-ac93-8dcbca038872\" (UID: \"100fce2c-765a-4370-ac93-8dcbca038872\") " Nov 04 03:00:43.758544 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:00:43.758441 4405 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/100fce2c-765a-4370-ac93-8dcbca038872-kube-api-access-64q7n\" (UniqueName: \"kubernetes.io/projected/100fce2c-765a-4370-ac93-8dcbca038872-kube-api-access-64q7n\") pod \"100fce2c-765a-4370-ac93-8dcbca038872\" (UID: \"100fce2c-765a-4370-ac93-8dcbca038872\") " Nov 04 03:00:43.758657 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:00:43.758631 4405 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/100fce2c-765a-4370-ac93-8dcbca038872-plugins-config-volume" pod "100fce2c-765a-4370-ac93-8dcbca038872" (UID: "100fce2c-765a-4370-ac93-8dcbca038872"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 03:00:43.759209 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:00:43.759183 4405 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/100fce2c-765a-4370-ac93-8dcbca038872-plugins-config-volume\") on node \"ip-10-0-134-125.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:00:43.760812 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:00:43.760785 4405 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/100fce2c-765a-4370-ac93-8dcbca038872-kube-api-access-64q7n" pod "100fce2c-765a-4370-ac93-8dcbca038872" (UID: "100fce2c-765a-4370-ac93-8dcbca038872"). InnerVolumeSpecName "kube-api-access-64q7n". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 03:00:43.860617 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:00:43.860587 4405 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/82ad73c2-7f63-43cb-a8af-5f17c4621c67-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/82ad73c2-7f63-43cb-a8af-5f17c4621c67-plugins-config-volume\") pod \"82ad73c2-7f63-43cb-a8af-5f17c4621c67\" (UID: \"82ad73c2-7f63-43cb-a8af-5f17c4621c67\") " Nov 04 03:00:43.860832 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:00:43.860749 4405 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/82ad73c2-7f63-43cb-a8af-5f17c4621c67-kube-api-access-z7whb\" (UniqueName: \"kubernetes.io/projected/82ad73c2-7f63-43cb-a8af-5f17c4621c67-kube-api-access-z7whb\") pod \"82ad73c2-7f63-43cb-a8af-5f17c4621c67\" (UID: \"82ad73c2-7f63-43cb-a8af-5f17c4621c67\") " Nov 04 03:00:43.861074 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:00:43.861051 4405 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/82ad73c2-7f63-43cb-a8af-5f17c4621c67-plugins-config-volume" pod "82ad73c2-7f63-43cb-a8af-5f17c4621c67" (UID: "82ad73c2-7f63-43cb-a8af-5f17c4621c67"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 03:00:43.861636 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:00:43.861613 4405 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-64q7n\" (UniqueName: \"kubernetes.io/projected/100fce2c-765a-4370-ac93-8dcbca038872-kube-api-access-64q7n\") on node \"ip-10-0-134-125.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:00:43.861763 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:00:43.861641 4405 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/82ad73c2-7f63-43cb-a8af-5f17c4621c67-plugins-config-volume\") on node \"ip-10-0-134-125.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:00:43.862946 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:00:43.862924 4405 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/82ad73c2-7f63-43cb-a8af-5f17c4621c67-kube-api-access-z7whb" pod "82ad73c2-7f63-43cb-a8af-5f17c4621c67" (UID: "82ad73c2-7f63-43cb-a8af-5f17c4621c67"). InnerVolumeSpecName "kube-api-access-z7whb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 03:00:43.963430 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:00:43.963394 4405 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z7whb\" (UniqueName: \"kubernetes.io/projected/82ad73c2-7f63-43cb-a8af-5f17c4621c67-kube-api-access-z7whb\") on node \"ip-10-0-134-125.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:00:44.623540 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:00:44.623507 4405 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-5fde04ee-59df47b5d7-gnjpp" Nov 04 03:00:44.624090 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:00:44.623541 4405 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-5fde04ee-59df47b5d7-gnjpp" event={"ID":"82ad73c2-7f63-43cb-a8af-5f17c4621c67","Type":"ContainerDied","Data":"01d758935d7e23355b862eeb5c67ba322a04e4bc07d28b1501a6b2caa388c069"} Nov 04 03:00:44.624223 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:00:44.623601 4405 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-5fde04ee-5fdc7f866d-lk25n" Nov 04 03:00:44.624525 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:00:44.624139 4405 scope.go:122] "RemoveContainer" containerID="c88e436d48f06c7058f5cfef43ca5f7d5826b286ef36b16bbc264934bab87401" Nov 04 03:00:44.841873 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:00:44.841845 4405 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-5fde04ee-59df47b5d7-gnjpp"] Nov 04 03:00:44.894322 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:00:44.894222 4405 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-5fde04ee-59df47b5d7-gnjpp"] Nov 04 03:00:44.954457 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:00:44.954349 4405 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-5fde04ee-5fdc7f866d-lk25n"] Nov 04 03:00:44.958896 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:00:44.958867 4405 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-5fde04ee-5fdc7f866d-lk25n"] Nov 04 03:00:45.703243 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:00:45.703205 4405 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="100fce2c-765a-4370-ac93-8dcbca038872" path="/var/lib/kubelet/pods/100fce2c-765a-4370-ac93-8dcbca038872/volumes" Nov 04 03:00:45.703866 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:00:45.703846 4405 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="82ad73c2-7f63-43cb-a8af-5f17c4621c67" path="/var/lib/kubelet/pods/82ad73c2-7f63-43cb-a8af-5f17c4621c67/volumes" Nov 04 03:00:51.838528 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:00:51.838490 4405 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-a0bcbc-openshift-default-65dc9d9b6c-fzwkp"] Nov 04 03:00:51.838961 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:00:51.838756 4405 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-a0bcbc-openshift-default-65dc9d9b6c-fzwkp" podUID="700d7662-7d5b-40b0-afea-e55dc01922e6" containerName="istio-proxy" containerID="cri-o://78dd8506b9d9043226ee6d6914645f7efbd422a5e486742873dbb05dbe7b5758" gracePeriod=30 Nov 04 03:00:51.889578 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:00:51.889053 4405 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-a0bcbc-openshift-default-65dc9d9b6c-m97sr"] Nov 04 03:00:51.889578 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:00:51.889366 4405 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-a0bcbc-openshift-default-65dc9d9b6c-m97sr" podUID="9a748832-c6ef-484f-8635-8c9148a38149" containerName="istio-proxy" containerID="cri-o://a1661cfbc2299c62f5a5a787738fe1d5401ae1e8caf90694fa945ac361182bce" gracePeriod=30 Nov 04 03:00:51.895562 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:00:51.895530 4405 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-a0bcbc-openshift-default-65dc9d9b6c-g52rd"] Nov 04 03:00:51.896067 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:00:51.895872 4405 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-a0bcbc-openshift-default-65dc9d9b6c-g52rd" podUID="e5b1b60a-7429-43fe-afd0-1338430750e7" containerName="istio-proxy" containerID="cri-o://f63f0a7ec7a5d0d7027b4dadfe46a71d035ff39a7a92442bf67b255cb6cceed9" gracePeriod=30 Nov 04 03:00:51.937878 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:00:51.937845 4405 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-a0bcbc-openshift-default-65dc9d9b6c-j4n6p"] Nov 04 03:00:51.938102 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:00:51.938076 4405 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-a0bcbc-openshift-default-65dc9d9b6c-j4n6p" podUID="55473cc8-d292-4f5f-bb17-3f08c2e19e8c" containerName="istio-proxy" containerID="cri-o://d0483ea48aad53b792dd83f987bec31e3af66acf77e8ca0b0d1c64621a92d5d4" gracePeriod=30 Nov 04 03:00:52.045108 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:00:52.045070 4405 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-cdcf67-openshift-default-6ff8f5cbc7-vbc52"] Nov 04 03:00:52.045395 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:00:52.045346 4405 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-cdcf67-openshift-default-6ff8f5cbc7-vbc52" podUID="8db652a6-107c-4696-a6f7-009cc7854aa3" containerName="istio-proxy" containerID="cri-o://96ed3be429ca5affc319b3368c190a474ad89ee7b9a139b572f11ca848ae44b9" gracePeriod=30 Nov 04 03:00:54.094970 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:00:54.094932 4405 patch_prober.go:29] interesting pod/e2e-worker-w2-a0bcbc-openshift-default-65dc9d9b6c-m97sr container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 04 03:00:54.095463 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:00:54.095004 4405 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-a0bcbc-openshift-default-65dc9d9b6c-m97sr" podUID="9a748832-c6ef-484f-8635-8c9148a38149" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 04 03:00:55.259051 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:00:55.259014 4405 patch_prober.go:29] interesting pod/e2e-worker-w2-a0bcbc-openshift-default-65dc9d9b6c-j4n6p container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 04 03:00:55.259443 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:00:55.259076 4405 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-a0bcbc-openshift-default-65dc9d9b6c-j4n6p" podUID="55473cc8-d292-4f5f-bb17-3f08c2e19e8c" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 04 03:00:55.724642 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:00:55.724604 4405 patch_prober.go:29] interesting pod/e2e-worker-w3-cdcf67-openshift-default-6ff8f5cbc7-vbc52 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 04 03:00:55.724825 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:00:55.724688 4405 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-cdcf67-openshift-default-6ff8f5cbc7-vbc52" podUID="8db652a6-107c-4696-a6f7-009cc7854aa3" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 04 03:00:56.314824 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:00:56.314775 4405 patch_prober.go:29] interesting pod/e2e-worker-w2-a0bcbc-openshift-default-65dc9d9b6c-g52rd container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 04 03:00:56.315235 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:00:56.314851 4405 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-a0bcbc-openshift-default-65dc9d9b6c-g52rd" podUID="e5b1b60a-7429-43fe-afd0-1338430750e7" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 04 03:00:57.205599 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:00:57.205557 4405 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-a0bcbc-openshift-default-65dc9d9b6c-fzwkp" Nov 04 03:00:57.277091 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:00:57.277062 4405 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/700d7662-7d5b-40b0-afea-e55dc01922e6-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/700d7662-7d5b-40b0-afea-e55dc01922e6-workload-certs\") pod \"700d7662-7d5b-40b0-afea-e55dc01922e6\" (UID: \"700d7662-7d5b-40b0-afea-e55dc01922e6\") " Nov 04 03:00:57.277260 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:00:57.277148 4405 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/700d7662-7d5b-40b0-afea-e55dc01922e6-istio-token\" (UniqueName: \"kubernetes.io/projected/700d7662-7d5b-40b0-afea-e55dc01922e6-istio-token\") pod \"700d7662-7d5b-40b0-afea-e55dc01922e6\" (UID: \"700d7662-7d5b-40b0-afea-e55dc01922e6\") " Nov 04 03:00:57.277327 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:00:57.277258 4405 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/700d7662-7d5b-40b0-afea-e55dc01922e6-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/700d7662-7d5b-40b0-afea-e55dc01922e6-istio-podinfo\") pod \"700d7662-7d5b-40b0-afea-e55dc01922e6\" (UID: \"700d7662-7d5b-40b0-afea-e55dc01922e6\") " Nov 04 03:00:57.277327 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:00:57.277298 4405 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/700d7662-7d5b-40b0-afea-e55dc01922e6-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/700d7662-7d5b-40b0-afea-e55dc01922e6-istio-envoy\") pod \"700d7662-7d5b-40b0-afea-e55dc01922e6\" (UID: \"700d7662-7d5b-40b0-afea-e55dc01922e6\") " Nov 04 03:00:57.277464 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:00:57.277337 4405 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/700d7662-7d5b-40b0-afea-e55dc01922e6-kube-api-access-vvn59\" (UniqueName: \"kubernetes.io/projected/700d7662-7d5b-40b0-afea-e55dc01922e6-kube-api-access-vvn59\") pod \"700d7662-7d5b-40b0-afea-e55dc01922e6\" (UID: \"700d7662-7d5b-40b0-afea-e55dc01922e6\") " Nov 04 03:00:57.277517 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:00:57.277492 4405 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/700d7662-7d5b-40b0-afea-e55dc01922e6-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/700d7662-7d5b-40b0-afea-e55dc01922e6-credential-socket\") pod \"700d7662-7d5b-40b0-afea-e55dc01922e6\" (UID: \"700d7662-7d5b-40b0-afea-e55dc01922e6\") " Nov 04 03:00:57.277628 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:00:57.277615 4405 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/700d7662-7d5b-40b0-afea-e55dc01922e6-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/700d7662-7d5b-40b0-afea-e55dc01922e6-workload-socket\") pod \"700d7662-7d5b-40b0-afea-e55dc01922e6\" (UID: \"700d7662-7d5b-40b0-afea-e55dc01922e6\") " Nov 04 03:00:57.277727 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:00:57.277689 4405 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/700d7662-7d5b-40b0-afea-e55dc01922e6-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/700d7662-7d5b-40b0-afea-e55dc01922e6-istio-data\") pod \"700d7662-7d5b-40b0-afea-e55dc01922e6\" (UID: \"700d7662-7d5b-40b0-afea-e55dc01922e6\") " Nov 04 03:00:57.277789 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:00:57.277727 4405 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/700d7662-7d5b-40b0-afea-e55dc01922e6-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/700d7662-7d5b-40b0-afea-e55dc01922e6-istiod-ca-cert\") pod \"700d7662-7d5b-40b0-afea-e55dc01922e6\" (UID: \"700d7662-7d5b-40b0-afea-e55dc01922e6\") " Nov 04 03:00:57.278577 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:00:57.278553 4405 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/700d7662-7d5b-40b0-afea-e55dc01922e6-istiod-ca-cert" pod "700d7662-7d5b-40b0-afea-e55dc01922e6" (UID: "700d7662-7d5b-40b0-afea-e55dc01922e6"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 03:00:57.278962 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:00:57.278876 4405 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/700d7662-7d5b-40b0-afea-e55dc01922e6-workload-certs" pod "700d7662-7d5b-40b0-afea-e55dc01922e6" (UID: "700d7662-7d5b-40b0-afea-e55dc01922e6"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 03:00:57.281016 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:00:57.280988 4405 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/700d7662-7d5b-40b0-afea-e55dc01922e6-credential-socket" pod "700d7662-7d5b-40b0-afea-e55dc01922e6" (UID: "700d7662-7d5b-40b0-afea-e55dc01922e6"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 03:00:57.284767 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:00:57.281155 4405 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/700d7662-7d5b-40b0-afea-e55dc01922e6-workload-socket" pod "700d7662-7d5b-40b0-afea-e55dc01922e6" (UID: "700d7662-7d5b-40b0-afea-e55dc01922e6"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 03:00:57.284767 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:00:57.281217 4405 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/700d7662-7d5b-40b0-afea-e55dc01922e6-istio-data" pod "700d7662-7d5b-40b0-afea-e55dc01922e6" (UID: "700d7662-7d5b-40b0-afea-e55dc01922e6"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 03:00:57.285026 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:00:57.284996 4405 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/700d7662-7d5b-40b0-afea-e55dc01922e6-istio-envoy" pod "700d7662-7d5b-40b0-afea-e55dc01922e6" (UID: "700d7662-7d5b-40b0-afea-e55dc01922e6"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 03:00:57.285234 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:00:57.285205 4405 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/700d7662-7d5b-40b0-afea-e55dc01922e6-kube-api-access-vvn59" pod "700d7662-7d5b-40b0-afea-e55dc01922e6" (UID: "700d7662-7d5b-40b0-afea-e55dc01922e6"). InnerVolumeSpecName "kube-api-access-vvn59". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 03:00:57.285532 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:00:57.285512 4405 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-a0bcbc-openshift-default-65dc9d9b6c-m97sr" Nov 04 03:00:57.285608 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:00:57.285584 4405 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/700d7662-7d5b-40b0-afea-e55dc01922e6-istio-token" pod "700d7662-7d5b-40b0-afea-e55dc01922e6" (UID: "700d7662-7d5b-40b0-afea-e55dc01922e6"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 03:00:57.286862 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:00:57.286840 4405 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/700d7662-7d5b-40b0-afea-e55dc01922e6-istio-podinfo" pod "700d7662-7d5b-40b0-afea-e55dc01922e6" (UID: "700d7662-7d5b-40b0-afea-e55dc01922e6"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 04 03:00:57.368260 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:00:57.367733 4405 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-a0bcbc-openshift-default-65dc9d9b6c-j4n6p" Nov 04 03:00:57.373747 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:00:57.373721 4405 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-a0bcbc-openshift-default-65dc9d9b6c-g52rd" Nov 04 03:00:57.379748 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:00:57.379717 4405 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9a748832-c6ef-484f-8635-8c9148a38149-istio-token\" (UniqueName: \"kubernetes.io/projected/9a748832-c6ef-484f-8635-8c9148a38149-istio-token\") pod \"9a748832-c6ef-484f-8635-8c9148a38149\" (UID: \"9a748832-c6ef-484f-8635-8c9148a38149\") " Nov 04 03:00:57.379908 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:00:57.379801 4405 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9a748832-c6ef-484f-8635-8c9148a38149-kube-api-access-8t7g7\" (UniqueName: \"kubernetes.io/projected/9a748832-c6ef-484f-8635-8c9148a38149-kube-api-access-8t7g7\") pod \"9a748832-c6ef-484f-8635-8c9148a38149\" (UID: \"9a748832-c6ef-484f-8635-8c9148a38149\") " Nov 04 03:00:57.380983 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:00:57.380951 4405 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9a748832-c6ef-484f-8635-8c9148a38149-workload-certs" pod "9a748832-c6ef-484f-8635-8c9148a38149" (UID: "9a748832-c6ef-484f-8635-8c9148a38149"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 03:00:57.381101 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:00:57.380613 4405 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9a748832-c6ef-484f-8635-8c9148a38149-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9a748832-c6ef-484f-8635-8c9148a38149-workload-certs\") pod \"9a748832-c6ef-484f-8635-8c9148a38149\" (UID: \"9a748832-c6ef-484f-8635-8c9148a38149\") " Nov 04 03:00:57.381101 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:00:57.381078 4405 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/9a748832-c6ef-484f-8635-8c9148a38149-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9a748832-c6ef-484f-8635-8c9148a38149-istio-podinfo\") pod \"9a748832-c6ef-484f-8635-8c9148a38149\" (UID: \"9a748832-c6ef-484f-8635-8c9148a38149\") " Nov 04 03:00:57.381467 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:00:57.381446 4405 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9a748832-c6ef-484f-8635-8c9148a38149-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9a748832-c6ef-484f-8635-8c9148a38149-istio-data\") pod \"9a748832-c6ef-484f-8635-8c9148a38149\" (UID: \"9a748832-c6ef-484f-8635-8c9148a38149\") " Nov 04 03:00:57.382779 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:00:57.382729 4405 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9a748832-c6ef-484f-8635-8c9148a38149-istio-data" pod "9a748832-c6ef-484f-8635-8c9148a38149" (UID: "9a748832-c6ef-484f-8635-8c9148a38149"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 03:00:57.383016 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:00:57.382996 4405 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9a748832-c6ef-484f-8635-8c9148a38149-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9a748832-c6ef-484f-8635-8c9148a38149-credential-socket\") pod \"9a748832-c6ef-484f-8635-8c9148a38149\" (UID: \"9a748832-c6ef-484f-8635-8c9148a38149\") " Nov 04 03:00:57.383507 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:00:57.383266 4405 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9a748832-c6ef-484f-8635-8c9148a38149-credential-socket" pod "9a748832-c6ef-484f-8635-8c9148a38149" (UID: "9a748832-c6ef-484f-8635-8c9148a38149"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 03:00:57.383803 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:00:57.383784 4405 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9a748832-c6ef-484f-8635-8c9148a38149-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9a748832-c6ef-484f-8635-8c9148a38149-istiod-ca-cert\") pod \"9a748832-c6ef-484f-8635-8c9148a38149\" (UID: \"9a748832-c6ef-484f-8635-8c9148a38149\") " Nov 04 03:00:57.384328 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:00:57.384205 4405 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9a748832-c6ef-484f-8635-8c9148a38149-istiod-ca-cert" pod "9a748832-c6ef-484f-8635-8c9148a38149" (UID: "9a748832-c6ef-484f-8635-8c9148a38149"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 03:00:57.384753 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:00:57.384714 4405 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9a748832-c6ef-484f-8635-8c9148a38149-workload-socket" pod "9a748832-c6ef-484f-8635-8c9148a38149" (UID: "9a748832-c6ef-484f-8635-8c9148a38149"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 03:00:57.384879 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:00:57.384859 4405 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9a748832-c6ef-484f-8635-8c9148a38149-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9a748832-c6ef-484f-8635-8c9148a38149-workload-socket\") pod \"9a748832-c6ef-484f-8635-8c9148a38149\" (UID: \"9a748832-c6ef-484f-8635-8c9148a38149\") " Nov 04 03:00:57.385491 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:00:57.385472 4405 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9a748832-c6ef-484f-8635-8c9148a38149-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9a748832-c6ef-484f-8635-8c9148a38149-istio-envoy\") pod \"9a748832-c6ef-484f-8635-8c9148a38149\" (UID: \"9a748832-c6ef-484f-8635-8c9148a38149\") " Nov 04 03:00:57.386302 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:00:57.386266 4405 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/700d7662-7d5b-40b0-afea-e55dc01922e6-istio-podinfo\") on node \"ip-10-0-134-125.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:00:57.386429 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:00:57.386413 4405 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/700d7662-7d5b-40b0-afea-e55dc01922e6-istio-envoy\") on node \"ip-10-0-134-125.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:00:57.386562 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:00:57.386547 4405 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vvn59\" (UniqueName: \"kubernetes.io/projected/700d7662-7d5b-40b0-afea-e55dc01922e6-kube-api-access-vvn59\") on node \"ip-10-0-134-125.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:00:57.386798 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:00:57.386782 4405 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9a748832-c6ef-484f-8635-8c9148a38149-istiod-ca-cert\") on node \"ip-10-0-134-125.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:00:57.386924 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:00:57.386911 4405 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9a748832-c6ef-484f-8635-8c9148a38149-workload-socket\") on node \"ip-10-0-134-125.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:00:57.387101 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:00:57.387086 4405 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/700d7662-7d5b-40b0-afea-e55dc01922e6-credential-socket\") on node \"ip-10-0-134-125.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:00:57.387221 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:00:57.387205 4405 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/700d7662-7d5b-40b0-afea-e55dc01922e6-workload-socket\") on node \"ip-10-0-134-125.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:00:57.387339 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:00:57.387324 4405 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/700d7662-7d5b-40b0-afea-e55dc01922e6-istio-data\") on node \"ip-10-0-134-125.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:00:57.387453 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:00:57.387440 4405 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/700d7662-7d5b-40b0-afea-e55dc01922e6-istiod-ca-cert\") on node \"ip-10-0-134-125.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:00:57.387563 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:00:57.387549 4405 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9a748832-c6ef-484f-8635-8c9148a38149-workload-certs\") on node \"ip-10-0-134-125.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:00:57.387688 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:00:57.387657 4405 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9a748832-c6ef-484f-8635-8c9148a38149-istio-data\") on node \"ip-10-0-134-125.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:00:57.390756 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:00:57.390733 4405 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/700d7662-7d5b-40b0-afea-e55dc01922e6-workload-certs\") on node \"ip-10-0-134-125.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:00:57.390876 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:00:57.390759 4405 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9a748832-c6ef-484f-8635-8c9148a38149-credential-socket\") on node \"ip-10-0-134-125.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:00:57.390876 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:00:57.386314 4405 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/9a748832-c6ef-484f-8635-8c9148a38149-istio-podinfo" pod "9a748832-c6ef-484f-8635-8c9148a38149" (UID: "9a748832-c6ef-484f-8635-8c9148a38149"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 04 03:00:57.390876 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:00:57.390801 4405 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/700d7662-7d5b-40b0-afea-e55dc01922e6-istio-token\") on node \"ip-10-0-134-125.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:00:57.390876 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:00:57.386423 4405 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9a748832-c6ef-484f-8635-8c9148a38149-kube-api-access-8t7g7" pod "9a748832-c6ef-484f-8635-8c9148a38149" (UID: "9a748832-c6ef-484f-8635-8c9148a38149"). InnerVolumeSpecName "kube-api-access-8t7g7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 03:00:57.390876 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:00:57.388283 4405 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9a748832-c6ef-484f-8635-8c9148a38149-istio-token" pod "9a748832-c6ef-484f-8635-8c9148a38149" (UID: "9a748832-c6ef-484f-8635-8c9148a38149"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 03:00:57.390876 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:00:57.388977 4405 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9a748832-c6ef-484f-8635-8c9148a38149-istio-envoy" pod "9a748832-c6ef-484f-8635-8c9148a38149" (UID: "9a748832-c6ef-484f-8635-8c9148a38149"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 03:00:57.421001 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:00:57.420972 4405 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-cdcf67-openshift-default-6ff8f5cbc7-vbc52" Nov 04 03:00:57.495441 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:00:57.492462 4405 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/e5b1b60a-7429-43fe-afd0-1338430750e7-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e5b1b60a-7429-43fe-afd0-1338430750e7-istio-podinfo\") pod \"e5b1b60a-7429-43fe-afd0-1338430750e7\" (UID: \"e5b1b60a-7429-43fe-afd0-1338430750e7\") " Nov 04 03:00:57.495441 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:00:57.492517 4405 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e5b1b60a-7429-43fe-afd0-1338430750e7-kube-api-access-sll9s\" (UniqueName: \"kubernetes.io/projected/e5b1b60a-7429-43fe-afd0-1338430750e7-kube-api-access-sll9s\") pod \"e5b1b60a-7429-43fe-afd0-1338430750e7\" (UID: \"e5b1b60a-7429-43fe-afd0-1338430750e7\") " Nov 04 03:00:57.495441 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:00:57.492560 4405 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e5b1b60a-7429-43fe-afd0-1338430750e7-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e5b1b60a-7429-43fe-afd0-1338430750e7-workload-certs\") pod \"e5b1b60a-7429-43fe-afd0-1338430750e7\" (UID: \"e5b1b60a-7429-43fe-afd0-1338430750e7\") " Nov 04 03:00:57.495441 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:00:57.492607 4405 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/55473cc8-d292-4f5f-bb17-3f08c2e19e8c-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/55473cc8-d292-4f5f-bb17-3f08c2e19e8c-workload-socket\") pod \"55473cc8-d292-4f5f-bb17-3f08c2e19e8c\" (UID: \"55473cc8-d292-4f5f-bb17-3f08c2e19e8c\") " Nov 04 03:00:57.495441 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:00:57.492781 4405 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/55473cc8-d292-4f5f-bb17-3f08c2e19e8c-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/55473cc8-d292-4f5f-bb17-3f08c2e19e8c-istio-envoy\") pod \"55473cc8-d292-4f5f-bb17-3f08c2e19e8c\" (UID: \"55473cc8-d292-4f5f-bb17-3f08c2e19e8c\") " Nov 04 03:00:57.495441 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:00:57.492860 4405 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e5b1b60a-7429-43fe-afd0-1338430750e7-istio-token\" (UniqueName: \"kubernetes.io/projected/e5b1b60a-7429-43fe-afd0-1338430750e7-istio-token\") pod \"e5b1b60a-7429-43fe-afd0-1338430750e7\" (UID: \"e5b1b60a-7429-43fe-afd0-1338430750e7\") " Nov 04 03:00:57.495441 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:00:57.492965 4405 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/55473cc8-d292-4f5f-bb17-3f08c2e19e8c-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/55473cc8-d292-4f5f-bb17-3f08c2e19e8c-credential-socket\") pod \"55473cc8-d292-4f5f-bb17-3f08c2e19e8c\" (UID: \"55473cc8-d292-4f5f-bb17-3f08c2e19e8c\") " Nov 04 03:00:57.495441 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:00:57.493001 4405 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e5b1b60a-7429-43fe-afd0-1338430750e7-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e5b1b60a-7429-43fe-afd0-1338430750e7-istiod-ca-cert\") pod \"e5b1b60a-7429-43fe-afd0-1338430750e7\" (UID: \"e5b1b60a-7429-43fe-afd0-1338430750e7\") " Nov 04 03:00:57.495441 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:00:57.493034 4405 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e5b1b60a-7429-43fe-afd0-1338430750e7-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e5b1b60a-7429-43fe-afd0-1338430750e7-workload-socket\") pod \"e5b1b60a-7429-43fe-afd0-1338430750e7\" (UID: \"e5b1b60a-7429-43fe-afd0-1338430750e7\") " Nov 04 03:00:57.495441 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:00:57.493115 4405 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/55473cc8-d292-4f5f-bb17-3f08c2e19e8c-istio-token\" (UniqueName: \"kubernetes.io/projected/55473cc8-d292-4f5f-bb17-3f08c2e19e8c-istio-token\") pod \"55473cc8-d292-4f5f-bb17-3f08c2e19e8c\" (UID: \"55473cc8-d292-4f5f-bb17-3f08c2e19e8c\") " Nov 04 03:00:57.495441 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:00:57.493150 4405 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e5b1b60a-7429-43fe-afd0-1338430750e7-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e5b1b60a-7429-43fe-afd0-1338430750e7-credential-socket\") pod \"e5b1b60a-7429-43fe-afd0-1338430750e7\" (UID: \"e5b1b60a-7429-43fe-afd0-1338430750e7\") " Nov 04 03:00:57.495441 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:00:57.493197 4405 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/55473cc8-d292-4f5f-bb17-3f08c2e19e8c-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/55473cc8-d292-4f5f-bb17-3f08c2e19e8c-workload-certs\") pod \"55473cc8-d292-4f5f-bb17-3f08c2e19e8c\" (UID: \"55473cc8-d292-4f5f-bb17-3f08c2e19e8c\") " Nov 04 03:00:57.495441 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:00:57.493230 4405 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/55473cc8-d292-4f5f-bb17-3f08c2e19e8c-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/55473cc8-d292-4f5f-bb17-3f08c2e19e8c-istiod-ca-cert\") pod \"55473cc8-d292-4f5f-bb17-3f08c2e19e8c\" (UID: \"55473cc8-d292-4f5f-bb17-3f08c2e19e8c\") " Nov 04 03:00:57.495441 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:00:57.493289 4405 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/55473cc8-d292-4f5f-bb17-3f08c2e19e8c-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/55473cc8-d292-4f5f-bb17-3f08c2e19e8c-istio-data\") pod \"55473cc8-d292-4f5f-bb17-3f08c2e19e8c\" (UID: \"55473cc8-d292-4f5f-bb17-3f08c2e19e8c\") " Nov 04 03:00:57.495441 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:00:57.493337 4405 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e5b1b60a-7429-43fe-afd0-1338430750e7-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e5b1b60a-7429-43fe-afd0-1338430750e7-istio-envoy\") pod \"e5b1b60a-7429-43fe-afd0-1338430750e7\" (UID: \"e5b1b60a-7429-43fe-afd0-1338430750e7\") " Nov 04 03:00:57.495441 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:00:57.493375 4405 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/55473cc8-d292-4f5f-bb17-3f08c2e19e8c-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/55473cc8-d292-4f5f-bb17-3f08c2e19e8c-istio-podinfo\") pod \"55473cc8-d292-4f5f-bb17-3f08c2e19e8c\" (UID: \"55473cc8-d292-4f5f-bb17-3f08c2e19e8c\") " Nov 04 03:00:57.495441 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:00:57.493517 4405 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/55473cc8-d292-4f5f-bb17-3f08c2e19e8c-kube-api-access-28xj2\" (UniqueName: \"kubernetes.io/projected/55473cc8-d292-4f5f-bb17-3f08c2e19e8c-kube-api-access-28xj2\") pod \"55473cc8-d292-4f5f-bb17-3f08c2e19e8c\" (UID: \"55473cc8-d292-4f5f-bb17-3f08c2e19e8c\") " Nov 04 03:00:57.495441 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:00:57.493574 4405 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e5b1b60a-7429-43fe-afd0-1338430750e7-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e5b1b60a-7429-43fe-afd0-1338430750e7-istio-data\") pod \"e5b1b60a-7429-43fe-afd0-1338430750e7\" (UID: \"e5b1b60a-7429-43fe-afd0-1338430750e7\") " Nov 04 03:00:57.495441 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:00:57.494523 4405 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e5b1b60a-7429-43fe-afd0-1338430750e7-workload-socket" pod "e5b1b60a-7429-43fe-afd0-1338430750e7" (UID: "e5b1b60a-7429-43fe-afd0-1338430750e7"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 03:00:57.495441 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:00:57.495139 4405 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e5b1b60a-7429-43fe-afd0-1338430750e7-istio-data" pod "e5b1b60a-7429-43fe-afd0-1338430750e7" (UID: "e5b1b60a-7429-43fe-afd0-1338430750e7"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 03:00:57.497038 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:00:57.495842 4405 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e5b1b60a-7429-43fe-afd0-1338430750e7-workload-socket\") on node \"ip-10-0-134-125.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:00:57.497038 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:00:57.495866 4405 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9a748832-c6ef-484f-8635-8c9148a38149-istio-envoy\") on node \"ip-10-0-134-125.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:00:57.497038 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:00:57.495880 4405 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e5b1b60a-7429-43fe-afd0-1338430750e7-istio-data\") on node \"ip-10-0-134-125.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:00:57.497038 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:00:57.495893 4405 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9a748832-c6ef-484f-8635-8c9148a38149-istio-token\") on node \"ip-10-0-134-125.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:00:57.497038 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:00:57.495905 4405 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8t7g7\" (UniqueName: \"kubernetes.io/projected/9a748832-c6ef-484f-8635-8c9148a38149-kube-api-access-8t7g7\") on node \"ip-10-0-134-125.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:00:57.497038 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:00:57.495919 4405 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9a748832-c6ef-484f-8635-8c9148a38149-istio-podinfo\") on node \"ip-10-0-134-125.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:00:57.497038 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:00:57.496158 4405 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/55473cc8-d292-4f5f-bb17-3f08c2e19e8c-istio-data" pod "55473cc8-d292-4f5f-bb17-3f08c2e19e8c" (UID: "55473cc8-d292-4f5f-bb17-3f08c2e19e8c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 03:00:57.497038 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:00:57.496507 4405 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e5b1b60a-7429-43fe-afd0-1338430750e7-credential-socket" pod "e5b1b60a-7429-43fe-afd0-1338430750e7" (UID: "e5b1b60a-7429-43fe-afd0-1338430750e7"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 03:00:57.497038 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:00:57.496708 4405 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/55473cc8-d292-4f5f-bb17-3f08c2e19e8c-workload-certs" pod "55473cc8-d292-4f5f-bb17-3f08c2e19e8c" (UID: "55473cc8-d292-4f5f-bb17-3f08c2e19e8c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 03:00:57.497038 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:00:57.496958 4405 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/55473cc8-d292-4f5f-bb17-3f08c2e19e8c-istiod-ca-cert" pod "55473cc8-d292-4f5f-bb17-3f08c2e19e8c" (UID: "55473cc8-d292-4f5f-bb17-3f08c2e19e8c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 03:00:57.497754 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:00:57.497728 4405 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e5b1b60a-7429-43fe-afd0-1338430750e7-workload-certs" pod "e5b1b60a-7429-43fe-afd0-1338430750e7" (UID: "e5b1b60a-7429-43fe-afd0-1338430750e7"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 03:00:57.497868 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:00:57.497767 4405 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/55473cc8-d292-4f5f-bb17-3f08c2e19e8c-credential-socket" pod "55473cc8-d292-4f5f-bb17-3f08c2e19e8c" (UID: "55473cc8-d292-4f5f-bb17-3f08c2e19e8c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 03:00:57.498423 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:00:57.498398 4405 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e5b1b60a-7429-43fe-afd0-1338430750e7-istiod-ca-cert" pod "e5b1b60a-7429-43fe-afd0-1338430750e7" (UID: "e5b1b60a-7429-43fe-afd0-1338430750e7"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 03:00:57.498723 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:00:57.498680 4405 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/55473cc8-d292-4f5f-bb17-3f08c2e19e8c-workload-socket" pod "55473cc8-d292-4f5f-bb17-3f08c2e19e8c" (UID: "55473cc8-d292-4f5f-bb17-3f08c2e19e8c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 03:00:57.503966 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:00:57.499210 4405 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/55473cc8-d292-4f5f-bb17-3f08c2e19e8c-istio-token" pod "55473cc8-d292-4f5f-bb17-3f08c2e19e8c" (UID: "55473cc8-d292-4f5f-bb17-3f08c2e19e8c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 03:00:57.503966 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:00:57.501256 4405 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e5b1b60a-7429-43fe-afd0-1338430750e7-istio-envoy" pod "e5b1b60a-7429-43fe-afd0-1338430750e7" (UID: "e5b1b60a-7429-43fe-afd0-1338430750e7"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 03:00:57.503966 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:00:57.501361 4405 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/55473cc8-d292-4f5f-bb17-3f08c2e19e8c-kube-api-access-28xj2" pod "55473cc8-d292-4f5f-bb17-3f08c2e19e8c" (UID: "55473cc8-d292-4f5f-bb17-3f08c2e19e8c"). InnerVolumeSpecName "kube-api-access-28xj2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 03:00:57.503966 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:00:57.501839 4405 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e5b1b60a-7429-43fe-afd0-1338430750e7-istio-token" pod "e5b1b60a-7429-43fe-afd0-1338430750e7" (UID: "e5b1b60a-7429-43fe-afd0-1338430750e7"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 03:00:57.503966 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:00:57.502023 4405 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e5b1b60a-7429-43fe-afd0-1338430750e7-kube-api-access-sll9s" pod "e5b1b60a-7429-43fe-afd0-1338430750e7" (UID: "e5b1b60a-7429-43fe-afd0-1338430750e7"). InnerVolumeSpecName "kube-api-access-sll9s". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 03:00:57.503966 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:00:57.502961 4405 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e5b1b60a-7429-43fe-afd0-1338430750e7-istio-podinfo" pod "e5b1b60a-7429-43fe-afd0-1338430750e7" (UID: "e5b1b60a-7429-43fe-afd0-1338430750e7"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 04 03:00:57.503966 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:00:57.502988 4405 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/55473cc8-d292-4f5f-bb17-3f08c2e19e8c-istio-podinfo" pod "55473cc8-d292-4f5f-bb17-3f08c2e19e8c" (UID: "55473cc8-d292-4f5f-bb17-3f08c2e19e8c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 04 03:00:57.504261 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:00:57.504001 4405 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/55473cc8-d292-4f5f-bb17-3f08c2e19e8c-istio-envoy" pod "55473cc8-d292-4f5f-bb17-3f08c2e19e8c" (UID: "55473cc8-d292-4f5f-bb17-3f08c2e19e8c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 03:00:57.596813 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:00:57.596782 4405 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8db652a6-107c-4696-a6f7-009cc7854aa3-kube-api-access-vjznl\" (UniqueName: \"kubernetes.io/projected/8db652a6-107c-4696-a6f7-009cc7854aa3-kube-api-access-vjznl\") pod \"8db652a6-107c-4696-a6f7-009cc7854aa3\" (UID: \"8db652a6-107c-4696-a6f7-009cc7854aa3\") " Nov 04 03:00:57.597020 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:00:57.596850 4405 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8db652a6-107c-4696-a6f7-009cc7854aa3-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8db652a6-107c-4696-a6f7-009cc7854aa3-istiod-ca-cert\") pod \"8db652a6-107c-4696-a6f7-009cc7854aa3\" (UID: \"8db652a6-107c-4696-a6f7-009cc7854aa3\") " Nov 04 03:00:57.597020 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:00:57.596964 4405 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/8db652a6-107c-4696-a6f7-009cc7854aa3-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8db652a6-107c-4696-a6f7-009cc7854aa3-istio-podinfo\") pod \"8db652a6-107c-4696-a6f7-009cc7854aa3\" (UID: \"8db652a6-107c-4696-a6f7-009cc7854aa3\") " Nov 04 03:00:57.597020 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:00:57.597011 4405 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8db652a6-107c-4696-a6f7-009cc7854aa3-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8db652a6-107c-4696-a6f7-009cc7854aa3-credential-socket\") pod \"8db652a6-107c-4696-a6f7-009cc7854aa3\" (UID: \"8db652a6-107c-4696-a6f7-009cc7854aa3\") " Nov 04 03:00:57.597218 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:00:57.597036 4405 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8db652a6-107c-4696-a6f7-009cc7854aa3-istio-token\" (UniqueName: \"kubernetes.io/projected/8db652a6-107c-4696-a6f7-009cc7854aa3-istio-token\") pod \"8db652a6-107c-4696-a6f7-009cc7854aa3\" (UID: \"8db652a6-107c-4696-a6f7-009cc7854aa3\") " Nov 04 03:00:57.597275 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:00:57.597226 4405 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8db652a6-107c-4696-a6f7-009cc7854aa3-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8db652a6-107c-4696-a6f7-009cc7854aa3-workload-certs\") pod \"8db652a6-107c-4696-a6f7-009cc7854aa3\" (UID: \"8db652a6-107c-4696-a6f7-009cc7854aa3\") " Nov 04 03:00:57.597326 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:00:57.597290 4405 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8db652a6-107c-4696-a6f7-009cc7854aa3-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8db652a6-107c-4696-a6f7-009cc7854aa3-workload-socket\") pod \"8db652a6-107c-4696-a6f7-009cc7854aa3\" (UID: \"8db652a6-107c-4696-a6f7-009cc7854aa3\") " Nov 04 03:00:57.597384 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:00:57.597316 4405 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8db652a6-107c-4696-a6f7-009cc7854aa3-istiod-ca-cert" pod "8db652a6-107c-4696-a6f7-009cc7854aa3" (UID: "8db652a6-107c-4696-a6f7-009cc7854aa3"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 03:00:57.597384 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:00:57.597314 4405 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8db652a6-107c-4696-a6f7-009cc7854aa3-credential-socket" pod "8db652a6-107c-4696-a6f7-009cc7854aa3" (UID: "8db652a6-107c-4696-a6f7-009cc7854aa3"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 03:00:57.597384 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:00:57.597321 4405 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8db652a6-107c-4696-a6f7-009cc7854aa3-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8db652a6-107c-4696-a6f7-009cc7854aa3-istio-envoy\") pod \"8db652a6-107c-4696-a6f7-009cc7854aa3\" (UID: \"8db652a6-107c-4696-a6f7-009cc7854aa3\") " Nov 04 03:00:57.597545 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:00:57.597491 4405 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8db652a6-107c-4696-a6f7-009cc7854aa3-workload-certs" pod "8db652a6-107c-4696-a6f7-009cc7854aa3" (UID: "8db652a6-107c-4696-a6f7-009cc7854aa3"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 03:00:57.597545 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:00:57.597521 4405 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8db652a6-107c-4696-a6f7-009cc7854aa3-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8db652a6-107c-4696-a6f7-009cc7854aa3-istio-data\") pod \"8db652a6-107c-4696-a6f7-009cc7854aa3\" (UID: \"8db652a6-107c-4696-a6f7-009cc7854aa3\") " Nov 04 03:00:57.597661 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:00:57.597597 4405 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8db652a6-107c-4696-a6f7-009cc7854aa3-workload-socket" pod "8db652a6-107c-4696-a6f7-009cc7854aa3" (UID: "8db652a6-107c-4696-a6f7-009cc7854aa3"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 03:00:57.597860 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:00:57.597826 4405 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8db652a6-107c-4696-a6f7-009cc7854aa3-istio-data" pod "8db652a6-107c-4696-a6f7-009cc7854aa3" (UID: "8db652a6-107c-4696-a6f7-009cc7854aa3"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 03:00:57.598600 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:00:57.598315 4405 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e5b1b60a-7429-43fe-afd0-1338430750e7-istio-token\") on node \"ip-10-0-134-125.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:00:57.598600 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:00:57.598342 4405 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8db652a6-107c-4696-a6f7-009cc7854aa3-istiod-ca-cert\") on node \"ip-10-0-134-125.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:00:57.598600 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:00:57.598359 4405 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/55473cc8-d292-4f5f-bb17-3f08c2e19e8c-credential-socket\") on node \"ip-10-0-134-125.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:00:57.598600 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:00:57.598373 4405 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e5b1b60a-7429-43fe-afd0-1338430750e7-istiod-ca-cert\") on node \"ip-10-0-134-125.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:00:57.598600 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:00:57.598387 4405 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/55473cc8-d292-4f5f-bb17-3f08c2e19e8c-istio-token\") on node \"ip-10-0-134-125.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:00:57.598600 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:00:57.598400 4405 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e5b1b60a-7429-43fe-afd0-1338430750e7-credential-socket\") on node \"ip-10-0-134-125.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:00:57.598600 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:00:57.598414 4405 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8db652a6-107c-4696-a6f7-009cc7854aa3-credential-socket\") on node \"ip-10-0-134-125.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:00:57.598600 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:00:57.598427 4405 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/55473cc8-d292-4f5f-bb17-3f08c2e19e8c-workload-certs\") on node \"ip-10-0-134-125.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:00:57.598600 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:00:57.598439 4405 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/55473cc8-d292-4f5f-bb17-3f08c2e19e8c-istiod-ca-cert\") on node \"ip-10-0-134-125.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:00:57.598600 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:00:57.598451 4405 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/55473cc8-d292-4f5f-bb17-3f08c2e19e8c-istio-data\") on node \"ip-10-0-134-125.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:00:57.598600 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:00:57.598463 4405 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e5b1b60a-7429-43fe-afd0-1338430750e7-istio-envoy\") on node \"ip-10-0-134-125.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:00:57.598600 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:00:57.598475 4405 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/55473cc8-d292-4f5f-bb17-3f08c2e19e8c-istio-podinfo\") on node \"ip-10-0-134-125.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:00:57.598600 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:00:57.598484 4405 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8db652a6-107c-4696-a6f7-009cc7854aa3-workload-certs\") on node \"ip-10-0-134-125.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:00:57.598600 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:00:57.598496 4405 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-28xj2\" (UniqueName: \"kubernetes.io/projected/55473cc8-d292-4f5f-bb17-3f08c2e19e8c-kube-api-access-28xj2\") on node \"ip-10-0-134-125.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:00:57.598600 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:00:57.598506 4405 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8db652a6-107c-4696-a6f7-009cc7854aa3-workload-socket\") on node \"ip-10-0-134-125.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:00:57.598600 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:00:57.598518 4405 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e5b1b60a-7429-43fe-afd0-1338430750e7-istio-podinfo\") on node \"ip-10-0-134-125.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:00:57.598600 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:00:57.598534 4405 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sll9s\" (UniqueName: \"kubernetes.io/projected/e5b1b60a-7429-43fe-afd0-1338430750e7-kube-api-access-sll9s\") on node \"ip-10-0-134-125.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:00:57.598600 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:00:57.598549 4405 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e5b1b60a-7429-43fe-afd0-1338430750e7-workload-certs\") on node \"ip-10-0-134-125.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:00:57.598600 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:00:57.598563 4405 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8db652a6-107c-4696-a6f7-009cc7854aa3-istio-data\") on node \"ip-10-0-134-125.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:00:57.598600 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:00:57.598574 4405 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/55473cc8-d292-4f5f-bb17-3f08c2e19e8c-workload-socket\") on node \"ip-10-0-134-125.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:00:57.598600 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:00:57.598583 4405 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/55473cc8-d292-4f5f-bb17-3f08c2e19e8c-istio-envoy\") on node \"ip-10-0-134-125.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:00:57.601024 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:00:57.600999 4405 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8db652a6-107c-4696-a6f7-009cc7854aa3-istio-envoy" pod "8db652a6-107c-4696-a6f7-009cc7854aa3" (UID: "8db652a6-107c-4696-a6f7-009cc7854aa3"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 03:00:57.601135 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:00:57.601027 4405 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8db652a6-107c-4696-a6f7-009cc7854aa3-kube-api-access-vjznl" pod "8db652a6-107c-4696-a6f7-009cc7854aa3" (UID: "8db652a6-107c-4696-a6f7-009cc7854aa3"). InnerVolumeSpecName "kube-api-access-vjznl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 03:00:57.601192 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:00:57.601160 4405 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8db652a6-107c-4696-a6f7-009cc7854aa3-istio-token" pod "8db652a6-107c-4696-a6f7-009cc7854aa3" (UID: "8db652a6-107c-4696-a6f7-009cc7854aa3"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 03:00:57.602179 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:00:57.602155 4405 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/8db652a6-107c-4696-a6f7-009cc7854aa3-istio-podinfo" pod "8db652a6-107c-4696-a6f7-009cc7854aa3" (UID: "8db652a6-107c-4696-a6f7-009cc7854aa3"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 04 03:00:57.699799 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:00:57.699710 4405 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8db652a6-107c-4696-a6f7-009cc7854aa3-istio-podinfo\") on node \"ip-10-0-134-125.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:00:57.699799 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:00:57.699738 4405 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8db652a6-107c-4696-a6f7-009cc7854aa3-istio-token\") on node \"ip-10-0-134-125.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:00:57.699799 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:00:57.699751 4405 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8db652a6-107c-4696-a6f7-009cc7854aa3-istio-envoy\") on node \"ip-10-0-134-125.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:00:57.699799 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:00:57.699763 4405 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vjznl\" (UniqueName: \"kubernetes.io/projected/8db652a6-107c-4696-a6f7-009cc7854aa3-kube-api-access-vjznl\") on node \"ip-10-0-134-125.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:00:57.724561 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:00:57.724531 4405 generic.go:356] "Generic (PLEG): container finished" podID="e5b1b60a-7429-43fe-afd0-1338430750e7" containerID="f63f0a7ec7a5d0d7027b4dadfe46a71d035ff39a7a92442bf67b255cb6cceed9" exitCode=0 Nov 04 03:00:57.724819 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:00:57.724636 4405 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-a0bcbc-openshift-default-65dc9d9b6c-g52rd" event={"ID":"e5b1b60a-7429-43fe-afd0-1338430750e7","Type":"ContainerDied","Data":"f63f0a7ec7a5d0d7027b4dadfe46a71d035ff39a7a92442bf67b255cb6cceed9"} Nov 04 03:00:57.724819 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:00:57.724743 4405 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-a0bcbc-openshift-default-65dc9d9b6c-g52rd" event={"ID":"e5b1b60a-7429-43fe-afd0-1338430750e7","Type":"ContainerDied","Data":"43d13a31145ea726c9b6daa7ad9155ad60ed8d52fd7165845d2ef934cd1c75f2"} Nov 04 03:00:57.724819 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:00:57.724793 4405 scope.go:122] "RemoveContainer" containerID="f63f0a7ec7a5d0d7027b4dadfe46a71d035ff39a7a92442bf67b255cb6cceed9" Nov 04 03:00:57.725323 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:00:57.724977 4405 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-a0bcbc-openshift-default-65dc9d9b6c-g52rd" Nov 04 03:00:57.730222 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:00:57.730188 4405 generic.go:356] "Generic (PLEG): container finished" podID="9a748832-c6ef-484f-8635-8c9148a38149" containerID="a1661cfbc2299c62f5a5a787738fe1d5401ae1e8caf90694fa945ac361182bce" exitCode=0 Nov 04 03:00:57.731089 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:00:57.730376 4405 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-a0bcbc-openshift-default-65dc9d9b6c-m97sr" Nov 04 03:00:57.731089 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:00:57.730449 4405 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-a0bcbc-openshift-default-65dc9d9b6c-m97sr" event={"ID":"9a748832-c6ef-484f-8635-8c9148a38149","Type":"ContainerDied","Data":"a1661cfbc2299c62f5a5a787738fe1d5401ae1e8caf90694fa945ac361182bce"} Nov 04 03:00:57.731089 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:00:57.730505 4405 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-a0bcbc-openshift-default-65dc9d9b6c-m97sr" event={"ID":"9a748832-c6ef-484f-8635-8c9148a38149","Type":"ContainerDied","Data":"d11d2dd3276a99552ac98a263095e432b9c548215c60471ca0588f74928e4a05"} Nov 04 03:00:57.733776 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:00:57.733745 4405 generic.go:356] "Generic (PLEG): container finished" podID="700d7662-7d5b-40b0-afea-e55dc01922e6" containerID="78dd8506b9d9043226ee6d6914645f7efbd422a5e486742873dbb05dbe7b5758" exitCode=0 Nov 04 03:00:57.733918 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:00:57.733900 4405 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-a0bcbc-openshift-default-65dc9d9b6c-fzwkp" Nov 04 03:00:57.734702 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:00:57.734639 4405 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-a0bcbc-openshift-default-65dc9d9b6c-fzwkp" event={"ID":"700d7662-7d5b-40b0-afea-e55dc01922e6","Type":"ContainerDied","Data":"78dd8506b9d9043226ee6d6914645f7efbd422a5e486742873dbb05dbe7b5758"} Nov 04 03:00:57.736054 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:00:57.734912 4405 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-a0bcbc-openshift-default-65dc9d9b6c-fzwkp" event={"ID":"700d7662-7d5b-40b0-afea-e55dc01922e6","Type":"ContainerDied","Data":"8dbab7567bfc10c8268f7166e7294c13a0c339f85203ccf7e1226e0aca4b9120"} Nov 04 03:00:57.737140 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:00:57.737112 4405 generic.go:356] "Generic (PLEG): container finished" podID="55473cc8-d292-4f5f-bb17-3f08c2e19e8c" containerID="d0483ea48aad53b792dd83f987bec31e3af66acf77e8ca0b0d1c64621a92d5d4" exitCode=0 Nov 04 03:00:57.737248 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:00:57.737141 4405 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-a0bcbc-openshift-default-65dc9d9b6c-j4n6p" event={"ID":"55473cc8-d292-4f5f-bb17-3f08c2e19e8c","Type":"ContainerDied","Data":"d0483ea48aad53b792dd83f987bec31e3af66acf77e8ca0b0d1c64621a92d5d4"} Nov 04 03:00:57.737248 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:00:57.737160 4405 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-a0bcbc-openshift-default-65dc9d9b6c-j4n6p" Nov 04 03:00:57.737248 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:00:57.737183 4405 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-a0bcbc-openshift-default-65dc9d9b6c-j4n6p" event={"ID":"55473cc8-d292-4f5f-bb17-3f08c2e19e8c","Type":"ContainerDied","Data":"07ea17ab897809cbcbab549a3a4a53a38e10dd34ee720910bffcc1c8439260b7"} Nov 04 03:00:57.739445 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:00:57.739415 4405 generic.go:356] "Generic (PLEG): container finished" podID="8db652a6-107c-4696-a6f7-009cc7854aa3" containerID="96ed3be429ca5affc319b3368c190a474ad89ee7b9a139b572f11ca848ae44b9" exitCode=0 Nov 04 03:00:57.739542 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:00:57.739527 4405 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-cdcf67-openshift-default-6ff8f5cbc7-vbc52" event={"ID":"8db652a6-107c-4696-a6f7-009cc7854aa3","Type":"ContainerDied","Data":"96ed3be429ca5affc319b3368c190a474ad89ee7b9a139b572f11ca848ae44b9"} Nov 04 03:00:57.739601 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:00:57.739571 4405 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-cdcf67-openshift-default-6ff8f5cbc7-vbc52" event={"ID":"8db652a6-107c-4696-a6f7-009cc7854aa3","Type":"ContainerDied","Data":"ca77e9275f3792de686421eef979b09048d27628aa2f03d66a0639bec9176c2d"} Nov 04 03:00:57.739661 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:00:57.739646 4405 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-cdcf67-openshift-default-6ff8f5cbc7-vbc52" Nov 04 03:00:57.756330 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:00:57.756305 4405 scope.go:122] "RemoveContainer" containerID="f63f0a7ec7a5d0d7027b4dadfe46a71d035ff39a7a92442bf67b255cb6cceed9" Nov 04 03:00:57.756850 ip-10-0-134-125 kubenswrapper[4405]: E1104 03:00:57.756801 4405 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f63f0a7ec7a5d0d7027b4dadfe46a71d035ff39a7a92442bf67b255cb6cceed9\": container with ID starting with f63f0a7ec7a5d0d7027b4dadfe46a71d035ff39a7a92442bf67b255cb6cceed9 not found: ID does not exist" containerID="f63f0a7ec7a5d0d7027b4dadfe46a71d035ff39a7a92442bf67b255cb6cceed9" Nov 04 03:00:57.757034 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:00:57.756842 4405 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f63f0a7ec7a5d0d7027b4dadfe46a71d035ff39a7a92442bf67b255cb6cceed9"} err="failed to get container status \"f63f0a7ec7a5d0d7027b4dadfe46a71d035ff39a7a92442bf67b255cb6cceed9\": rpc error: code = NotFound desc = could not find container \"f63f0a7ec7a5d0d7027b4dadfe46a71d035ff39a7a92442bf67b255cb6cceed9\": container with ID starting with f63f0a7ec7a5d0d7027b4dadfe46a71d035ff39a7a92442bf67b255cb6cceed9 not found: ID does not exist" Nov 04 03:00:57.757034 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:00:57.756873 4405 scope.go:122] "RemoveContainer" containerID="a1661cfbc2299c62f5a5a787738fe1d5401ae1e8caf90694fa945ac361182bce" Nov 04 03:00:57.774993 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:00:57.774953 4405 scope.go:122] "RemoveContainer" containerID="a1661cfbc2299c62f5a5a787738fe1d5401ae1e8caf90694fa945ac361182bce" Nov 04 03:00:57.775351 ip-10-0-134-125 kubenswrapper[4405]: E1104 03:00:57.775321 4405 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a1661cfbc2299c62f5a5a787738fe1d5401ae1e8caf90694fa945ac361182bce\": container with ID starting with a1661cfbc2299c62f5a5a787738fe1d5401ae1e8caf90694fa945ac361182bce not found: ID does not exist" containerID="a1661cfbc2299c62f5a5a787738fe1d5401ae1e8caf90694fa945ac361182bce" Nov 04 03:00:57.775462 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:00:57.775357 4405 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a1661cfbc2299c62f5a5a787738fe1d5401ae1e8caf90694fa945ac361182bce"} err="failed to get container status \"a1661cfbc2299c62f5a5a787738fe1d5401ae1e8caf90694fa945ac361182bce\": rpc error: code = NotFound desc = could not find container \"a1661cfbc2299c62f5a5a787738fe1d5401ae1e8caf90694fa945ac361182bce\": container with ID starting with a1661cfbc2299c62f5a5a787738fe1d5401ae1e8caf90694fa945ac361182bce not found: ID does not exist" Nov 04 03:00:57.775462 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:00:57.775378 4405 scope.go:122] "RemoveContainer" containerID="78dd8506b9d9043226ee6d6914645f7efbd422a5e486742873dbb05dbe7b5758" Nov 04 03:00:57.789015 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:00:57.788985 4405 scope.go:122] "RemoveContainer" containerID="78dd8506b9d9043226ee6d6914645f7efbd422a5e486742873dbb05dbe7b5758" Nov 04 03:00:57.789518 ip-10-0-134-125 kubenswrapper[4405]: E1104 03:00:57.789475 4405 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"78dd8506b9d9043226ee6d6914645f7efbd422a5e486742873dbb05dbe7b5758\": container with ID starting with 78dd8506b9d9043226ee6d6914645f7efbd422a5e486742873dbb05dbe7b5758 not found: ID does not exist" containerID="78dd8506b9d9043226ee6d6914645f7efbd422a5e486742873dbb05dbe7b5758" Nov 04 03:00:57.790168 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:00:57.790120 4405 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"78dd8506b9d9043226ee6d6914645f7efbd422a5e486742873dbb05dbe7b5758"} err="failed to get container status \"78dd8506b9d9043226ee6d6914645f7efbd422a5e486742873dbb05dbe7b5758\": rpc error: code = NotFound desc = could not find container \"78dd8506b9d9043226ee6d6914645f7efbd422a5e486742873dbb05dbe7b5758\": container with ID starting with 78dd8506b9d9043226ee6d6914645f7efbd422a5e486742873dbb05dbe7b5758 not found: ID does not exist" Nov 04 03:00:57.790391 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:00:57.790360 4405 scope.go:122] "RemoveContainer" containerID="d0483ea48aad53b792dd83f987bec31e3af66acf77e8ca0b0d1c64621a92d5d4" Nov 04 03:00:57.806180 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:00:57.806066 4405 scope.go:122] "RemoveContainer" containerID="d0483ea48aad53b792dd83f987bec31e3af66acf77e8ca0b0d1c64621a92d5d4" Nov 04 03:00:57.806500 ip-10-0-134-125 kubenswrapper[4405]: E1104 03:00:57.806466 4405 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d0483ea48aad53b792dd83f987bec31e3af66acf77e8ca0b0d1c64621a92d5d4\": container with ID starting with d0483ea48aad53b792dd83f987bec31e3af66acf77e8ca0b0d1c64621a92d5d4 not found: ID does not exist" containerID="d0483ea48aad53b792dd83f987bec31e3af66acf77e8ca0b0d1c64621a92d5d4" Nov 04 03:00:57.806634 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:00:57.806498 4405 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d0483ea48aad53b792dd83f987bec31e3af66acf77e8ca0b0d1c64621a92d5d4"} err="failed to get container status \"d0483ea48aad53b792dd83f987bec31e3af66acf77e8ca0b0d1c64621a92d5d4\": rpc error: code = NotFound desc = could not find container \"d0483ea48aad53b792dd83f987bec31e3af66acf77e8ca0b0d1c64621a92d5d4\": container with ID starting with d0483ea48aad53b792dd83f987bec31e3af66acf77e8ca0b0d1c64621a92d5d4 not found: ID does not exist" Nov 04 03:00:57.806634 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:00:57.806524 4405 scope.go:122] "RemoveContainer" containerID="96ed3be429ca5affc319b3368c190a474ad89ee7b9a139b572f11ca848ae44b9" Nov 04 03:00:57.817521 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:00:57.817500 4405 scope.go:122] "RemoveContainer" containerID="96ed3be429ca5affc319b3368c190a474ad89ee7b9a139b572f11ca848ae44b9" Nov 04 03:00:57.817915 ip-10-0-134-125 kubenswrapper[4405]: E1104 03:00:57.817892 4405 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"96ed3be429ca5affc319b3368c190a474ad89ee7b9a139b572f11ca848ae44b9\": container with ID starting with 96ed3be429ca5affc319b3368c190a474ad89ee7b9a139b572f11ca848ae44b9 not found: ID does not exist" containerID="96ed3be429ca5affc319b3368c190a474ad89ee7b9a139b572f11ca848ae44b9" Nov 04 03:00:57.818003 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:00:57.817922 4405 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"96ed3be429ca5affc319b3368c190a474ad89ee7b9a139b572f11ca848ae44b9"} err="failed to get container status \"96ed3be429ca5affc319b3368c190a474ad89ee7b9a139b572f11ca848ae44b9\": rpc error: code = NotFound desc = could not find container \"96ed3be429ca5affc319b3368c190a474ad89ee7b9a139b572f11ca848ae44b9\": container with ID starting with 96ed3be429ca5affc319b3368c190a474ad89ee7b9a139b572f11ca848ae44b9 not found: ID does not exist" Nov 04 03:00:58.254125 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:00:58.254076 4405 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-a0bcbc-openshift-default-65dc9d9b6c-g52rd"] Nov 04 03:00:58.261325 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:00:58.261295 4405 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-a0bcbc-openshift-default-65dc9d9b6c-g52rd"] Nov 04 03:00:58.371596 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:00:58.371566 4405 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-a0bcbc-openshift-default-65dc9d9b6c-m97sr"] Nov 04 03:00:58.376941 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:00:58.376916 4405 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-a0bcbc-openshift-default-65dc9d9b6c-m97sr"] Nov 04 03:00:58.487297 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:00:58.487264 4405 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-cdcf67-openshift-default-6ff8f5cbc7-vbc52"] Nov 04 03:00:58.490815 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:00:58.490793 4405 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-cdcf67-openshift-default-6ff8f5cbc7-vbc52"] Nov 04 03:00:58.601854 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:00:58.601804 4405 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-a0bcbc-openshift-default-65dc9d9b6c-j4n6p"] Nov 04 03:00:58.605880 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:00:58.605859 4405 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-a0bcbc-openshift-default-65dc9d9b6c-j4n6p"] Nov 04 03:00:58.712619 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:00:58.712582 4405 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-a0bcbc-openshift-default-65dc9d9b6c-fzwkp"] Nov 04 03:00:58.720355 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:00:58.720315 4405 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-a0bcbc-openshift-default-65dc9d9b6c-fzwkp"] Nov 04 03:00:59.702814 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:00:59.702779 4405 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="55473cc8-d292-4f5f-bb17-3f08c2e19e8c" path="/var/lib/kubelet/pods/55473cc8-d292-4f5f-bb17-3f08c2e19e8c/volumes" Nov 04 03:00:59.703355 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:00:59.703337 4405 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="700d7662-7d5b-40b0-afea-e55dc01922e6" path="/var/lib/kubelet/pods/700d7662-7d5b-40b0-afea-e55dc01922e6/volumes" Nov 04 03:00:59.703933 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:00:59.703912 4405 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8db652a6-107c-4696-a6f7-009cc7854aa3" path="/var/lib/kubelet/pods/8db652a6-107c-4696-a6f7-009cc7854aa3/volumes" Nov 04 03:00:59.704768 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:00:59.704751 4405 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9a748832-c6ef-484f-8635-8c9148a38149" path="/var/lib/kubelet/pods/9a748832-c6ef-484f-8635-8c9148a38149/volumes" Nov 04 03:00:59.705212 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:00:59.705198 4405 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e5b1b60a-7429-43fe-afd0-1338430750e7" path="/var/lib/kubelet/pods/e5b1b60a-7429-43fe-afd0-1338430750e7/volumes" Nov 04 03:01:05.319437 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:01:05.319402 4405 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w4-35df3e-openshift-default-6c585b76f-vl6wz"] Nov 04 03:01:05.320361 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:01:05.320341 4405 cpu_manager.go:414] "containerMap: removing container" podUID="f61715a2-4d72-491b-8236-3e4a71462e60" containerName="cleanup" Nov 04 03:01:05.320411 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:01:05.320365 4405 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f61715a2-4d72-491b-8236-3e4a71462e60" containerName="cleanup" Nov 04 03:01:05.320411 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:01:05.320380 4405 cpu_manager.go:414] "containerMap: removing container" podUID="f61715a2-4d72-491b-8236-3e4a71462e60" containerName="cleanup" Nov 04 03:01:05.320411 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:01:05.320389 4405 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f61715a2-4d72-491b-8236-3e4a71462e60" containerName="cleanup" Nov 04 03:01:05.320411 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:01:05.320402 4405 cpu_manager.go:414] "containerMap: removing container" podUID="3e760665-863f-41a0-b2fb-7b33a1768fc6" containerName="payload-pre-processing" Nov 04 03:01:05.320529 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:01:05.320410 4405 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3e760665-863f-41a0-b2fb-7b33a1768fc6" containerName="payload-pre-processing" Nov 04 03:01:05.320529 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:01:05.320424 4405 cpu_manager.go:414] "containerMap: removing container" podUID="e5b1b60a-7429-43fe-afd0-1338430750e7" containerName="istio-proxy" Nov 04 03:01:05.320529 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:01:05.320431 4405 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e5b1b60a-7429-43fe-afd0-1338430750e7" containerName="istio-proxy" Nov 04 03:01:05.320529 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:01:05.320445 4405 cpu_manager.go:414] "containerMap: removing container" podUID="9a748832-c6ef-484f-8635-8c9148a38149" containerName="istio-proxy" Nov 04 03:01:05.320529 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:01:05.320452 4405 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9a748832-c6ef-484f-8635-8c9148a38149" containerName="istio-proxy" Nov 04 03:01:05.320529 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:01:05.320469 4405 cpu_manager.go:414] "containerMap: removing container" podUID="700d7662-7d5b-40b0-afea-e55dc01922e6" containerName="istio-proxy" Nov 04 03:01:05.320529 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:01:05.320476 4405 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="700d7662-7d5b-40b0-afea-e55dc01922e6" containerName="istio-proxy" Nov 04 03:01:05.320529 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:01:05.320491 4405 cpu_manager.go:414] "containerMap: removing container" podUID="1d9a643a-06d7-4e7e-85bb-a87e2108864f" containerName="cleanup" Nov 04 03:01:05.320529 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:01:05.320498 4405 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1d9a643a-06d7-4e7e-85bb-a87e2108864f" containerName="cleanup" Nov 04 03:01:05.320529 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:01:05.320513 4405 cpu_manager.go:414] "containerMap: removing container" podUID="71ce2ec9-0edd-483f-b364-b45e15ccef58" containerName="cleanup" Nov 04 03:01:05.320529 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:01:05.320520 4405 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="71ce2ec9-0edd-483f-b364-b45e15ccef58" containerName="cleanup" Nov 04 03:01:05.321026 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:01:05.320539 4405 cpu_manager.go:414] "containerMap: removing container" podUID="8db652a6-107c-4696-a6f7-009cc7854aa3" containerName="istio-proxy" Nov 04 03:01:05.321026 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:01:05.320590 4405 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8db652a6-107c-4696-a6f7-009cc7854aa3" containerName="istio-proxy" Nov 04 03:01:05.321026 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:01:05.320607 4405 cpu_manager.go:414] "containerMap: removing container" podUID="e0691251-c255-4643-a6c4-4dacae0f15b3" containerName="payload-processing" Nov 04 03:01:05.321026 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:01:05.320617 4405 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e0691251-c255-4643-a6c4-4dacae0f15b3" containerName="payload-processing" Nov 04 03:01:05.321271 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:01:05.320631 4405 cpu_manager.go:414] "containerMap: removing container" podUID="1d9a643a-06d7-4e7e-85bb-a87e2108864f" containerName="cleanup" Nov 04 03:01:05.321336 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:01:05.321277 4405 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1d9a643a-06d7-4e7e-85bb-a87e2108864f" containerName="cleanup" Nov 04 03:01:05.321336 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:01:05.321306 4405 cpu_manager.go:414] "containerMap: removing container" podUID="f61715a2-4d72-491b-8236-3e4a71462e60" containerName="cleanup" Nov 04 03:01:05.321336 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:01:05.321315 4405 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f61715a2-4d72-491b-8236-3e4a71462e60" containerName="cleanup" Nov 04 03:01:05.321336 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:01:05.321330 4405 cpu_manager.go:414] "containerMap: removing container" podUID="42766046-4bb9-4afd-8fe7-3bc0e0199d1e" containerName="cleanup" Nov 04 03:01:05.321494 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:01:05.321337 4405 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="42766046-4bb9-4afd-8fe7-3bc0e0199d1e" containerName="cleanup" Nov 04 03:01:05.321494 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:01:05.321355 4405 cpu_manager.go:414] "containerMap: removing container" podUID="100fce2c-765a-4370-ac93-8dcbca038872" containerName="payload-processing" Nov 04 03:01:05.321494 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:01:05.321362 4405 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="100fce2c-765a-4370-ac93-8dcbca038872" containerName="payload-processing" Nov 04 03:01:05.321494 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:01:05.321385 4405 cpu_manager.go:414] "containerMap: removing container" podUID="55473cc8-d292-4f5f-bb17-3f08c2e19e8c" containerName="istio-proxy" Nov 04 03:01:05.321494 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:01:05.321393 4405 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="55473cc8-d292-4f5f-bb17-3f08c2e19e8c" containerName="istio-proxy" Nov 04 03:01:05.321494 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:01:05.321412 4405 cpu_manager.go:414] "containerMap: removing container" podUID="71ce2ec9-0edd-483f-b364-b45e15ccef58" containerName="cleanup" Nov 04 03:01:05.321494 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:01:05.321419 4405 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="71ce2ec9-0edd-483f-b364-b45e15ccef58" containerName="cleanup" Nov 04 03:01:05.321494 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:01:05.321433 4405 cpu_manager.go:414] "containerMap: removing container" podUID="82ad73c2-7f63-43cb-a8af-5f17c4621c67" containerName="payload-pre-processing" Nov 04 03:01:05.321494 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:01:05.321440 4405 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="82ad73c2-7f63-43cb-a8af-5f17c4621c67" containerName="payload-pre-processing" Nov 04 03:01:05.321494 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:01:05.321451 4405 cpu_manager.go:414] "containerMap: removing container" podUID="42766046-4bb9-4afd-8fe7-3bc0e0199d1e" containerName="cleanup" Nov 04 03:01:05.321494 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:01:05.321457 4405 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="42766046-4bb9-4afd-8fe7-3bc0e0199d1e" containerName="cleanup" Nov 04 03:01:05.322080 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:01:05.321637 4405 memory_manager.go:365] "RemoveStaleState removing state" podUID="e5b1b60a-7429-43fe-afd0-1338430750e7" containerName="istio-proxy" Nov 04 03:01:05.322080 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:01:05.321651 4405 memory_manager.go:365] "RemoveStaleState removing state" podUID="9a748832-c6ef-484f-8635-8c9148a38149" containerName="istio-proxy" Nov 04 03:01:05.322080 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:01:05.321685 4405 memory_manager.go:365] "RemoveStaleState removing state" podUID="700d7662-7d5b-40b0-afea-e55dc01922e6" containerName="istio-proxy" Nov 04 03:01:05.322080 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:01:05.321697 4405 memory_manager.go:365] "RemoveStaleState removing state" podUID="1d9a643a-06d7-4e7e-85bb-a87e2108864f" containerName="cleanup" Nov 04 03:01:05.322080 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:01:05.321708 4405 memory_manager.go:365] "RemoveStaleState removing state" podUID="71ce2ec9-0edd-483f-b364-b45e15ccef58" containerName="cleanup" Nov 04 03:01:05.322080 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:01:05.321717 4405 memory_manager.go:365] "RemoveStaleState removing state" podUID="1d9a643a-06d7-4e7e-85bb-a87e2108864f" containerName="cleanup" Nov 04 03:01:05.322080 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:01:05.321726 4405 memory_manager.go:365] "RemoveStaleState removing state" podUID="8db652a6-107c-4696-a6f7-009cc7854aa3" containerName="istio-proxy" Nov 04 03:01:05.322080 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:01:05.321734 4405 memory_manager.go:365] "RemoveStaleState removing state" podUID="e0691251-c255-4643-a6c4-4dacae0f15b3" containerName="payload-processing" Nov 04 03:01:05.322080 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:01:05.321746 4405 memory_manager.go:365] "RemoveStaleState removing state" podUID="1d9a643a-06d7-4e7e-85bb-a87e2108864f" containerName="cleanup" Nov 04 03:01:05.322080 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:01:05.321757 4405 memory_manager.go:365] "RemoveStaleState removing state" podUID="f61715a2-4d72-491b-8236-3e4a71462e60" containerName="cleanup" Nov 04 03:01:05.322080 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:01:05.321765 4405 memory_manager.go:365] "RemoveStaleState removing state" podUID="42766046-4bb9-4afd-8fe7-3bc0e0199d1e" containerName="cleanup" Nov 04 03:01:05.322080 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:01:05.321773 4405 memory_manager.go:365] "RemoveStaleState removing state" podUID="100fce2c-765a-4370-ac93-8dcbca038872" containerName="payload-processing" Nov 04 03:01:05.322080 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:01:05.321784 4405 memory_manager.go:365] "RemoveStaleState removing state" podUID="55473cc8-d292-4f5f-bb17-3f08c2e19e8c" containerName="istio-proxy" Nov 04 03:01:05.322080 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:01:05.321793 4405 memory_manager.go:365] "RemoveStaleState removing state" podUID="71ce2ec9-0edd-483f-b364-b45e15ccef58" containerName="cleanup" Nov 04 03:01:05.322080 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:01:05.321801 4405 memory_manager.go:365] "RemoveStaleState removing state" podUID="82ad73c2-7f63-43cb-a8af-5f17c4621c67" containerName="payload-pre-processing" Nov 04 03:01:05.322080 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:01:05.321810 4405 memory_manager.go:365] "RemoveStaleState removing state" podUID="42766046-4bb9-4afd-8fe7-3bc0e0199d1e" containerName="cleanup" Nov 04 03:01:05.322080 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:01:05.321820 4405 memory_manager.go:365] "RemoveStaleState removing state" podUID="f61715a2-4d72-491b-8236-3e4a71462e60" containerName="cleanup" Nov 04 03:01:05.322080 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:01:05.321828 4405 memory_manager.go:365] "RemoveStaleState removing state" podUID="f61715a2-4d72-491b-8236-3e4a71462e60" containerName="cleanup" Nov 04 03:01:05.322080 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:01:05.321836 4405 memory_manager.go:365] "RemoveStaleState removing state" podUID="3e760665-863f-41a0-b2fb-7b33a1768fc6" containerName="payload-pre-processing" Nov 04 03:01:05.327783 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:01:05.327760 4405 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w4-35df3e-openshift-default-6c585b76f-vl6wz" Nov 04 03:01:05.374526 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:01:05.374482 4405 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/18ffb552-4917-4c03-a3b5-aa9c6a89f0e7-istio-envoy\") pod \"e2e-shared-b-w4-35df3e-openshift-default-6c585b76f-vl6wz\" (UID: \"18ffb552-4917-4c03-a3b5-aa9c6a89f0e7\") " pod="openshift-ingress/e2e-shared-b-w4-35df3e-openshift-default-6c585b76f-vl6wz" Nov 04 03:01:05.374718 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:01:05.374620 4405 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/18ffb552-4917-4c03-a3b5-aa9c6a89f0e7-istio-podinfo\") pod \"e2e-shared-b-w4-35df3e-openshift-default-6c585b76f-vl6wz\" (UID: \"18ffb552-4917-4c03-a3b5-aa9c6a89f0e7\") " pod="openshift-ingress/e2e-shared-b-w4-35df3e-openshift-default-6c585b76f-vl6wz" Nov 04 03:01:05.374718 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:01:05.374651 4405 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lzc4n\" (UniqueName: \"kubernetes.io/projected/18ffb552-4917-4c03-a3b5-aa9c6a89f0e7-kube-api-access-lzc4n\") pod \"e2e-shared-b-w4-35df3e-openshift-default-6c585b76f-vl6wz\" (UID: \"18ffb552-4917-4c03-a3b5-aa9c6a89f0e7\") " pod="openshift-ingress/e2e-shared-b-w4-35df3e-openshift-default-6c585b76f-vl6wz" Nov 04 03:01:05.374858 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:01:05.374840 4405 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/18ffb552-4917-4c03-a3b5-aa9c6a89f0e7-istio-token\") pod \"e2e-shared-b-w4-35df3e-openshift-default-6c585b76f-vl6wz\" (UID: \"18ffb552-4917-4c03-a3b5-aa9c6a89f0e7\") " pod="openshift-ingress/e2e-shared-b-w4-35df3e-openshift-default-6c585b76f-vl6wz" Nov 04 03:01:05.374894 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:01:05.374877 4405 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/18ffb552-4917-4c03-a3b5-aa9c6a89f0e7-istio-data\") pod \"e2e-shared-b-w4-35df3e-openshift-default-6c585b76f-vl6wz\" (UID: \"18ffb552-4917-4c03-a3b5-aa9c6a89f0e7\") " pod="openshift-ingress/e2e-shared-b-w4-35df3e-openshift-default-6c585b76f-vl6wz" Nov 04 03:01:05.374934 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:01:05.374912 4405 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/18ffb552-4917-4c03-a3b5-aa9c6a89f0e7-credential-socket\") pod \"e2e-shared-b-w4-35df3e-openshift-default-6c585b76f-vl6wz\" (UID: \"18ffb552-4917-4c03-a3b5-aa9c6a89f0e7\") " pod="openshift-ingress/e2e-shared-b-w4-35df3e-openshift-default-6c585b76f-vl6wz" Nov 04 03:01:05.375122 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:01:05.375096 4405 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/18ffb552-4917-4c03-a3b5-aa9c6a89f0e7-istiod-ca-cert\") pod \"e2e-shared-b-w4-35df3e-openshift-default-6c585b76f-vl6wz\" (UID: \"18ffb552-4917-4c03-a3b5-aa9c6a89f0e7\") " pod="openshift-ingress/e2e-shared-b-w4-35df3e-openshift-default-6c585b76f-vl6wz" Nov 04 03:01:05.375259 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:01:05.375241 4405 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/18ffb552-4917-4c03-a3b5-aa9c6a89f0e7-workload-socket\") pod \"e2e-shared-b-w4-35df3e-openshift-default-6c585b76f-vl6wz\" (UID: \"18ffb552-4917-4c03-a3b5-aa9c6a89f0e7\") " pod="openshift-ingress/e2e-shared-b-w4-35df3e-openshift-default-6c585b76f-vl6wz" Nov 04 03:01:05.375297 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:01:05.375274 4405 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/18ffb552-4917-4c03-a3b5-aa9c6a89f0e7-workload-certs\") pod \"e2e-shared-b-w4-35df3e-openshift-default-6c585b76f-vl6wz\" (UID: \"18ffb552-4917-4c03-a3b5-aa9c6a89f0e7\") " pod="openshift-ingress/e2e-shared-b-w4-35df3e-openshift-default-6c585b76f-vl6wz" Nov 04 03:01:05.379803 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:01:05.379779 4405 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w4-35df3e-openshift-default-dockercfg-92ks6\"" Nov 04 03:01:05.434653 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:01:05.433250 4405 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-35df3e-openshift-default-6c585b76f-vl6wz"] Nov 04 03:01:05.477217 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:01:05.477177 4405 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/18ffb552-4917-4c03-a3b5-aa9c6a89f0e7-istio-podinfo\") pod \"e2e-shared-b-w4-35df3e-openshift-default-6c585b76f-vl6wz\" (UID: \"18ffb552-4917-4c03-a3b5-aa9c6a89f0e7\") " pod="openshift-ingress/e2e-shared-b-w4-35df3e-openshift-default-6c585b76f-vl6wz" Nov 04 03:01:05.477431 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:01:05.477240 4405 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lzc4n\" (UniqueName: \"kubernetes.io/projected/18ffb552-4917-4c03-a3b5-aa9c6a89f0e7-kube-api-access-lzc4n\") pod \"e2e-shared-b-w4-35df3e-openshift-default-6c585b76f-vl6wz\" (UID: \"18ffb552-4917-4c03-a3b5-aa9c6a89f0e7\") " pod="openshift-ingress/e2e-shared-b-w4-35df3e-openshift-default-6c585b76f-vl6wz" Nov 04 03:01:05.477431 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:01:05.477380 4405 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/18ffb552-4917-4c03-a3b5-aa9c6a89f0e7-istio-token\") pod \"e2e-shared-b-w4-35df3e-openshift-default-6c585b76f-vl6wz\" (UID: \"18ffb552-4917-4c03-a3b5-aa9c6a89f0e7\") " pod="openshift-ingress/e2e-shared-b-w4-35df3e-openshift-default-6c585b76f-vl6wz" Nov 04 03:01:05.477431 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:01:05.477420 4405 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/18ffb552-4917-4c03-a3b5-aa9c6a89f0e7-istio-data\") pod \"e2e-shared-b-w4-35df3e-openshift-default-6c585b76f-vl6wz\" (UID: \"18ffb552-4917-4c03-a3b5-aa9c6a89f0e7\") " pod="openshift-ingress/e2e-shared-b-w4-35df3e-openshift-default-6c585b76f-vl6wz" Nov 04 03:01:05.477599 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:01:05.477474 4405 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/18ffb552-4917-4c03-a3b5-aa9c6a89f0e7-credential-socket\") pod \"e2e-shared-b-w4-35df3e-openshift-default-6c585b76f-vl6wz\" (UID: \"18ffb552-4917-4c03-a3b5-aa9c6a89f0e7\") " pod="openshift-ingress/e2e-shared-b-w4-35df3e-openshift-default-6c585b76f-vl6wz" Nov 04 03:01:05.477599 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:01:05.477578 4405 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/18ffb552-4917-4c03-a3b5-aa9c6a89f0e7-istiod-ca-cert\") pod \"e2e-shared-b-w4-35df3e-openshift-default-6c585b76f-vl6wz\" (UID: \"18ffb552-4917-4c03-a3b5-aa9c6a89f0e7\") " pod="openshift-ingress/e2e-shared-b-w4-35df3e-openshift-default-6c585b76f-vl6wz" Nov 04 03:01:05.479094 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:01:05.477746 4405 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/18ffb552-4917-4c03-a3b5-aa9c6a89f0e7-workload-socket\") pod \"e2e-shared-b-w4-35df3e-openshift-default-6c585b76f-vl6wz\" (UID: \"18ffb552-4917-4c03-a3b5-aa9c6a89f0e7\") " pod="openshift-ingress/e2e-shared-b-w4-35df3e-openshift-default-6c585b76f-vl6wz" Nov 04 03:01:05.479094 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:01:05.477774 4405 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/18ffb552-4917-4c03-a3b5-aa9c6a89f0e7-workload-certs\") pod \"e2e-shared-b-w4-35df3e-openshift-default-6c585b76f-vl6wz\" (UID: \"18ffb552-4917-4c03-a3b5-aa9c6a89f0e7\") " pod="openshift-ingress/e2e-shared-b-w4-35df3e-openshift-default-6c585b76f-vl6wz" Nov 04 03:01:05.479094 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:01:05.477820 4405 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/18ffb552-4917-4c03-a3b5-aa9c6a89f0e7-istio-envoy\") pod \"e2e-shared-b-w4-35df3e-openshift-default-6c585b76f-vl6wz\" (UID: \"18ffb552-4917-4c03-a3b5-aa9c6a89f0e7\") " pod="openshift-ingress/e2e-shared-b-w4-35df3e-openshift-default-6c585b76f-vl6wz" Nov 04 03:01:05.479094 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:01:05.479067 4405 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/18ffb552-4917-4c03-a3b5-aa9c6a89f0e7-istio-data\") pod \"e2e-shared-b-w4-35df3e-openshift-default-6c585b76f-vl6wz\" (UID: \"18ffb552-4917-4c03-a3b5-aa9c6a89f0e7\") " pod="openshift-ingress/e2e-shared-b-w4-35df3e-openshift-default-6c585b76f-vl6wz" Nov 04 03:01:05.482362 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:01:05.482335 4405 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/18ffb552-4917-4c03-a3b5-aa9c6a89f0e7-istio-podinfo\") pod \"e2e-shared-b-w4-35df3e-openshift-default-6c585b76f-vl6wz\" (UID: \"18ffb552-4917-4c03-a3b5-aa9c6a89f0e7\") " pod="openshift-ingress/e2e-shared-b-w4-35df3e-openshift-default-6c585b76f-vl6wz" Nov 04 03:01:05.483633 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:01:05.483612 4405 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/18ffb552-4917-4c03-a3b5-aa9c6a89f0e7-istiod-ca-cert\") pod \"e2e-shared-b-w4-35df3e-openshift-default-6c585b76f-vl6wz\" (UID: \"18ffb552-4917-4c03-a3b5-aa9c6a89f0e7\") " pod="openshift-ingress/e2e-shared-b-w4-35df3e-openshift-default-6c585b76f-vl6wz" Nov 04 03:01:05.484008 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:01:05.483990 4405 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/18ffb552-4917-4c03-a3b5-aa9c6a89f0e7-credential-socket\") pod \"e2e-shared-b-w4-35df3e-openshift-default-6c585b76f-vl6wz\" (UID: \"18ffb552-4917-4c03-a3b5-aa9c6a89f0e7\") " pod="openshift-ingress/e2e-shared-b-w4-35df3e-openshift-default-6c585b76f-vl6wz" Nov 04 03:01:05.485483 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:01:05.485465 4405 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/18ffb552-4917-4c03-a3b5-aa9c6a89f0e7-workload-socket\") pod \"e2e-shared-b-w4-35df3e-openshift-default-6c585b76f-vl6wz\" (UID: \"18ffb552-4917-4c03-a3b5-aa9c6a89f0e7\") " pod="openshift-ingress/e2e-shared-b-w4-35df3e-openshift-default-6c585b76f-vl6wz" Nov 04 03:01:05.485887 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:01:05.485866 4405 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/18ffb552-4917-4c03-a3b5-aa9c6a89f0e7-workload-certs\") pod \"e2e-shared-b-w4-35df3e-openshift-default-6c585b76f-vl6wz\" (UID: \"18ffb552-4917-4c03-a3b5-aa9c6a89f0e7\") " pod="openshift-ingress/e2e-shared-b-w4-35df3e-openshift-default-6c585b76f-vl6wz" Nov 04 03:01:05.491258 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:01:05.491236 4405 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/18ffb552-4917-4c03-a3b5-aa9c6a89f0e7-istio-envoy\") pod \"e2e-shared-b-w4-35df3e-openshift-default-6c585b76f-vl6wz\" (UID: \"18ffb552-4917-4c03-a3b5-aa9c6a89f0e7\") " pod="openshift-ingress/e2e-shared-b-w4-35df3e-openshift-default-6c585b76f-vl6wz" Nov 04 03:01:05.557961 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:01:05.557923 4405 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lzc4n\" (UniqueName: \"kubernetes.io/projected/18ffb552-4917-4c03-a3b5-aa9c6a89f0e7-kube-api-access-lzc4n\") pod \"e2e-shared-b-w4-35df3e-openshift-default-6c585b76f-vl6wz\" (UID: \"18ffb552-4917-4c03-a3b5-aa9c6a89f0e7\") " pod="openshift-ingress/e2e-shared-b-w4-35df3e-openshift-default-6c585b76f-vl6wz" Nov 04 03:01:05.558325 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:01:05.558305 4405 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/18ffb552-4917-4c03-a3b5-aa9c6a89f0e7-istio-token\") pod \"e2e-shared-b-w4-35df3e-openshift-default-6c585b76f-vl6wz\" (UID: \"18ffb552-4917-4c03-a3b5-aa9c6a89f0e7\") " pod="openshift-ingress/e2e-shared-b-w4-35df3e-openshift-default-6c585b76f-vl6wz" Nov 04 03:01:05.646941 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:01:05.646904 4405 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w4-35df3e-openshift-default-6c585b76f-vl6wz" Nov 04 03:01:06.082331 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:01:06.082303 4405 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-35df3e-openshift-default-6c585b76f-vl6wz"] Nov 04 03:01:06.083129 ip-10-0-134-125 kubenswrapper[4405]: W1104 03:01:06.083106 4405 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod18ffb552_4917_4c03_a3b5_aa9c6a89f0e7.slice/crio-97fb8ff966e8335a2d910a8a5b467f8f5867f0880723f76a2a55ed2254797200 WatchSource:0}: Error finding container 97fb8ff966e8335a2d910a8a5b467f8f5867f0880723f76a2a55ed2254797200: Status 404 returned error can't find the container with id 97fb8ff966e8335a2d910a8a5b467f8f5867f0880723f76a2a55ed2254797200 Nov 04 03:01:06.085938 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:01:06.085899 4405 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 04 03:01:06.086028 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:01:06.085993 4405 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 04 03:01:06.086065 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:01:06.086033 4405 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 04 03:01:06.838555 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:01:06.838509 4405 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-35df3e-openshift-default-6c585b76f-vl6wz" event={"ID":"18ffb552-4917-4c03-a3b5-aa9c6a89f0e7","Type":"ContainerStarted","Data":"9b9bed2e07dc5af1f69f981953d14d27e67d46f16bd7a3d70d9975fe219dfc75"} Nov 04 03:01:06.839011 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:01:06.838614 4405 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-35df3e-openshift-default-6c585b76f-vl6wz" event={"ID":"18ffb552-4917-4c03-a3b5-aa9c6a89f0e7","Type":"ContainerStarted","Data":"97fb8ff966e8335a2d910a8a5b467f8f5867f0880723f76a2a55ed2254797200"} Nov 04 03:01:06.885686 ip-10-0-134-125 kubenswrapper[4405]: E1104 03:01:06.885632 4405 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5f2vb" containerName="extractor" Nov 04 03:01:06.960163 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:01:06.960106 4405 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w4-35df3e-openshift-default-6c585b76f-vl6wz" podStartSLOduration=1.9600909149999999 podStartE2EDuration="1.960090915s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 03:01:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 03:01:06.958017667 +0000 UTC m=+1713.388717824" watchObservedRunningTime="2026-11-04 03:01:06.960090915 +0000 UTC m=+1713.390791035" Nov 04 03:01:07.647849 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:01:07.647811 4405 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w4-35df3e-openshift-default-6c585b76f-vl6wz" Nov 04 03:01:07.653510 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:01:07.653481 4405 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w4-35df3e-openshift-default-6c585b76f-vl6wz" Nov 04 03:01:07.844899 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:01:07.844868 4405 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w4-35df3e-openshift-default-6c585b76f-vl6wz" Nov 04 03:01:07.846030 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:01:07.846010 4405 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w4-35df3e-openshift-default-6c585b76f-vl6wz" Nov 04 03:01:08.056577 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:01:08.056499 4405 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-203c35-openshift-default-56497cdcc-ljwqc"] Nov 04 03:01:08.056786 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:01:08.056764 4405 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-203c35-openshift-default-56497cdcc-ljwqc" podUID="fec507be-e742-4226-a6c4-6fb41e6f3479" containerName="istio-proxy" containerID="cri-o://dc932a567f19c37532b049ce74104a0a9cefc967ab7c48ef6616658d921d5c76" gracePeriod=30 Nov 04 03:01:08.060152 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:01:08.060126 4405 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-203c35-openshift-default-56497cdcc-w6c48"] Nov 04 03:01:08.060406 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:01:08.060326 4405 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-203c35-openshift-default-56497cdcc-w6c48" podUID="89ddc69b-94de-4443-a62a-fa5d360aa368" containerName="istio-proxy" containerID="cri-o://9c709331c40b10e0a93eacdbe663fc93145c774d2e72f230525dbf41228825c3" gracePeriod=30 Nov 04 03:01:08.063727 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:01:08.063701 4405 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-203c35-openshift-default-56497cdcc-2b66r"] Nov 04 03:01:08.063953 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:01:08.063904 4405 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-203c35-openshift-default-56497cdcc-2b66r" podUID="147e25a7-dda1-4170-9453-44896e44e2bc" containerName="istio-proxy" containerID="cri-o://9493dae8e6eaed626d077d1b19415b29a1cf533152cc9a5774624d4c6db883b5" gracePeriod=30 Nov 04 03:01:08.066996 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:01:08.066969 4405 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-203c35-openshift-default-56497cdcc-z59vn"] Nov 04 03:01:08.067199 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:01:08.067177 4405 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-203c35-openshift-default-56497cdcc-z59vn" podUID="5ebc4fc9-8882-43c6-8713-1ee995a8ee00" containerName="istio-proxy" containerID="cri-o://ffba5bc76b8a20d2d74d2d5516d95428cd0e26f7e0bfddb238c77058347ceb52" gracePeriod=30 Nov 04 03:01:11.012250 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:01:11.012210 4405 patch_prober.go:29] interesting pod/e2e-worker-w4-203c35-openshift-default-56497cdcc-2b66r container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 04 03:01:11.012716 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:01:11.012278 4405 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-203c35-openshift-default-56497cdcc-2b66r" podUID="147e25a7-dda1-4170-9453-44896e44e2bc" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 04 03:01:12.155405 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:01:12.155362 4405 patch_prober.go:29] interesting pod/e2e-worker-w4-203c35-openshift-default-56497cdcc-z59vn container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 04 03:01:12.155915 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:01:12.155433 4405 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-203c35-openshift-default-56497cdcc-z59vn" podUID="5ebc4fc9-8882-43c6-8713-1ee995a8ee00" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 04 03:01:13.438928 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:01:13.438888 4405 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-203c35-openshift-default-56497cdcc-w6c48" Nov 04 03:01:13.451410 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:01:13.451317 4405 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/89ddc69b-94de-4443-a62a-fa5d360aa368-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/89ddc69b-94de-4443-a62a-fa5d360aa368-istio-podinfo\") pod \"89ddc69b-94de-4443-a62a-fa5d360aa368\" (UID: \"89ddc69b-94de-4443-a62a-fa5d360aa368\") " Nov 04 03:01:13.452711 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:01:13.451625 4405 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/89ddc69b-94de-4443-a62a-fa5d360aa368-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/89ddc69b-94de-4443-a62a-fa5d360aa368-credential-socket\") pod \"89ddc69b-94de-4443-a62a-fa5d360aa368\" (UID: \"89ddc69b-94de-4443-a62a-fa5d360aa368\") " Nov 04 03:01:13.452711 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:01:13.451788 4405 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/89ddc69b-94de-4443-a62a-fa5d360aa368-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/89ddc69b-94de-4443-a62a-fa5d360aa368-istiod-ca-cert\") pod \"89ddc69b-94de-4443-a62a-fa5d360aa368\" (UID: \"89ddc69b-94de-4443-a62a-fa5d360aa368\") " Nov 04 03:01:13.452711 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:01:13.451838 4405 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/89ddc69b-94de-4443-a62a-fa5d360aa368-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/89ddc69b-94de-4443-a62a-fa5d360aa368-istio-envoy\") pod \"89ddc69b-94de-4443-a62a-fa5d360aa368\" (UID: \"89ddc69b-94de-4443-a62a-fa5d360aa368\") " Nov 04 03:01:13.452711 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:01:13.451942 4405 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/89ddc69b-94de-4443-a62a-fa5d360aa368-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/89ddc69b-94de-4443-a62a-fa5d360aa368-workload-socket\") pod \"89ddc69b-94de-4443-a62a-fa5d360aa368\" (UID: \"89ddc69b-94de-4443-a62a-fa5d360aa368\") " Nov 04 03:01:13.452711 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:01:13.451997 4405 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/89ddc69b-94de-4443-a62a-fa5d360aa368-istio-token\" (UniqueName: \"kubernetes.io/projected/89ddc69b-94de-4443-a62a-fa5d360aa368-istio-token\") pod \"89ddc69b-94de-4443-a62a-fa5d360aa368\" (UID: \"89ddc69b-94de-4443-a62a-fa5d360aa368\") " Nov 04 03:01:13.452711 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:01:13.452061 4405 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/89ddc69b-94de-4443-a62a-fa5d360aa368-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/89ddc69b-94de-4443-a62a-fa5d360aa368-workload-certs\") pod \"89ddc69b-94de-4443-a62a-fa5d360aa368\" (UID: \"89ddc69b-94de-4443-a62a-fa5d360aa368\") " Nov 04 03:01:13.452711 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:01:13.452086 4405 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/89ddc69b-94de-4443-a62a-fa5d360aa368-credential-socket" pod "89ddc69b-94de-4443-a62a-fa5d360aa368" (UID: "89ddc69b-94de-4443-a62a-fa5d360aa368"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 03:01:13.452711 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:01:13.452249 4405 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/89ddc69b-94de-4443-a62a-fa5d360aa368-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/89ddc69b-94de-4443-a62a-fa5d360aa368-istio-data\") pod \"89ddc69b-94de-4443-a62a-fa5d360aa368\" (UID: \"89ddc69b-94de-4443-a62a-fa5d360aa368\") " Nov 04 03:01:13.452711 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:01:13.452293 4405 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/89ddc69b-94de-4443-a62a-fa5d360aa368-kube-api-access-qlgg7\" (UniqueName: \"kubernetes.io/projected/89ddc69b-94de-4443-a62a-fa5d360aa368-kube-api-access-qlgg7\") pod \"89ddc69b-94de-4443-a62a-fa5d360aa368\" (UID: \"89ddc69b-94de-4443-a62a-fa5d360aa368\") " Nov 04 03:01:13.452711 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:01:13.452397 4405 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/89ddc69b-94de-4443-a62a-fa5d360aa368-workload-socket" pod "89ddc69b-94de-4443-a62a-fa5d360aa368" (UID: "89ddc69b-94de-4443-a62a-fa5d360aa368"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 03:01:13.452711 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:01:13.452616 4405 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/89ddc69b-94de-4443-a62a-fa5d360aa368-istio-data" pod "89ddc69b-94de-4443-a62a-fa5d360aa368" (UID: "89ddc69b-94de-4443-a62a-fa5d360aa368"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 03:01:13.453372 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:01:13.452942 4405 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/89ddc69b-94de-4443-a62a-fa5d360aa368-workload-certs" pod "89ddc69b-94de-4443-a62a-fa5d360aa368" (UID: "89ddc69b-94de-4443-a62a-fa5d360aa368"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 03:01:13.453693 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:01:13.453595 4405 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/89ddc69b-94de-4443-a62a-fa5d360aa368-istiod-ca-cert" pod "89ddc69b-94de-4443-a62a-fa5d360aa368" (UID: "89ddc69b-94de-4443-a62a-fa5d360aa368"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 03:01:13.454916 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:01:13.454882 4405 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/89ddc69b-94de-4443-a62a-fa5d360aa368-istio-token" pod "89ddc69b-94de-4443-a62a-fa5d360aa368" (UID: "89ddc69b-94de-4443-a62a-fa5d360aa368"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 03:01:13.455101 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:01:13.455078 4405 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/89ddc69b-94de-4443-a62a-fa5d360aa368-istio-envoy" pod "89ddc69b-94de-4443-a62a-fa5d360aa368" (UID: "89ddc69b-94de-4443-a62a-fa5d360aa368"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 03:01:13.455432 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:01:13.455416 4405 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/89ddc69b-94de-4443-a62a-fa5d360aa368-istiod-ca-cert\") on node \"ip-10-0-134-125.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:01:13.455495 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:01:13.455437 4405 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/89ddc69b-94de-4443-a62a-fa5d360aa368-istio-envoy\") on node \"ip-10-0-134-125.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:01:13.455495 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:01:13.455446 4405 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/89ddc69b-94de-4443-a62a-fa5d360aa368-workload-socket\") on node \"ip-10-0-134-125.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:01:13.455495 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:01:13.455455 4405 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/89ddc69b-94de-4443-a62a-fa5d360aa368-istio-token\") on node \"ip-10-0-134-125.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:01:13.455495 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:01:13.455463 4405 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/89ddc69b-94de-4443-a62a-fa5d360aa368-workload-certs\") on node \"ip-10-0-134-125.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:01:13.455495 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:01:13.455472 4405 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/89ddc69b-94de-4443-a62a-fa5d360aa368-istio-data\") on node \"ip-10-0-134-125.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:01:13.455495 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:01:13.455479 4405 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/89ddc69b-94de-4443-a62a-fa5d360aa368-credential-socket\") on node \"ip-10-0-134-125.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:01:13.455846 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:01:13.455805 4405 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/89ddc69b-94de-4443-a62a-fa5d360aa368-kube-api-access-qlgg7" pod "89ddc69b-94de-4443-a62a-fa5d360aa368" (UID: "89ddc69b-94de-4443-a62a-fa5d360aa368"). InnerVolumeSpecName "kube-api-access-qlgg7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 03:01:13.457588 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:01:13.457564 4405 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/89ddc69b-94de-4443-a62a-fa5d360aa368-istio-podinfo" pod "89ddc69b-94de-4443-a62a-fa5d360aa368" (UID: "89ddc69b-94de-4443-a62a-fa5d360aa368"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 04 03:01:13.479355 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:01:13.479330 4405 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-203c35-openshift-default-56497cdcc-2b66r" Nov 04 03:01:13.489368 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:01:13.489339 4405 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-203c35-openshift-default-56497cdcc-z59vn" Nov 04 03:01:13.556277 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:01:13.556246 4405 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/147e25a7-dda1-4170-9453-44896e44e2bc-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/147e25a7-dda1-4170-9453-44896e44e2bc-istiod-ca-cert\") pod \"147e25a7-dda1-4170-9453-44896e44e2bc\" (UID: \"147e25a7-dda1-4170-9453-44896e44e2bc\") " Nov 04 03:01:13.556488 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:01:13.556297 4405 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/147e25a7-dda1-4170-9453-44896e44e2bc-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/147e25a7-dda1-4170-9453-44896e44e2bc-istio-envoy\") pod \"147e25a7-dda1-4170-9453-44896e44e2bc\" (UID: \"147e25a7-dda1-4170-9453-44896e44e2bc\") " Nov 04 03:01:13.556488 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:01:13.556335 4405 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5ebc4fc9-8882-43c6-8713-1ee995a8ee00-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5ebc4fc9-8882-43c6-8713-1ee995a8ee00-istio-envoy\") pod \"5ebc4fc9-8882-43c6-8713-1ee995a8ee00\" (UID: \"5ebc4fc9-8882-43c6-8713-1ee995a8ee00\") " Nov 04 03:01:13.556488 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:01:13.556351 4405 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/147e25a7-dda1-4170-9453-44896e44e2bc-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/147e25a7-dda1-4170-9453-44896e44e2bc-workload-certs\") pod \"147e25a7-dda1-4170-9453-44896e44e2bc\" (UID: \"147e25a7-dda1-4170-9453-44896e44e2bc\") " Nov 04 03:01:13.556488 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:01:13.556385 4405 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5ebc4fc9-8882-43c6-8713-1ee995a8ee00-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5ebc4fc9-8882-43c6-8713-1ee995a8ee00-credential-socket\") pod \"5ebc4fc9-8882-43c6-8713-1ee995a8ee00\" (UID: \"5ebc4fc9-8882-43c6-8713-1ee995a8ee00\") " Nov 04 03:01:13.556488 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:01:13.556431 4405 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/147e25a7-dda1-4170-9453-44896e44e2bc-kube-api-access-jf2xt\" (UniqueName: \"kubernetes.io/projected/147e25a7-dda1-4170-9453-44896e44e2bc-kube-api-access-jf2xt\") pod \"147e25a7-dda1-4170-9453-44896e44e2bc\" (UID: \"147e25a7-dda1-4170-9453-44896e44e2bc\") " Nov 04 03:01:13.556805 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:01:13.556557 4405 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/147e25a7-dda1-4170-9453-44896e44e2bc-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/147e25a7-dda1-4170-9453-44896e44e2bc-workload-socket\") pod \"147e25a7-dda1-4170-9453-44896e44e2bc\" (UID: \"147e25a7-dda1-4170-9453-44896e44e2bc\") " Nov 04 03:01:13.556805 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:01:13.556584 4405 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5ebc4fc9-8882-43c6-8713-1ee995a8ee00-istio-token\" (UniqueName: \"kubernetes.io/projected/5ebc4fc9-8882-43c6-8713-1ee995a8ee00-istio-token\") pod \"5ebc4fc9-8882-43c6-8713-1ee995a8ee00\" (UID: \"5ebc4fc9-8882-43c6-8713-1ee995a8ee00\") " Nov 04 03:01:13.556805 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:01:13.556654 4405 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5ebc4fc9-8882-43c6-8713-1ee995a8ee00-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5ebc4fc9-8882-43c6-8713-1ee995a8ee00-workload-certs\") pod \"5ebc4fc9-8882-43c6-8713-1ee995a8ee00\" (UID: \"5ebc4fc9-8882-43c6-8713-1ee995a8ee00\") " Nov 04 03:01:13.556805 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:01:13.556714 4405 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5ebc4fc9-8882-43c6-8713-1ee995a8ee00-kube-api-access-br6lx\" (UniqueName: \"kubernetes.io/projected/5ebc4fc9-8882-43c6-8713-1ee995a8ee00-kube-api-access-br6lx\") pod \"5ebc4fc9-8882-43c6-8713-1ee995a8ee00\" (UID: \"5ebc4fc9-8882-43c6-8713-1ee995a8ee00\") " Nov 04 03:01:13.556805 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:01:13.556757 4405 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/147e25a7-dda1-4170-9453-44896e44e2bc-istiod-ca-cert" pod "147e25a7-dda1-4170-9453-44896e44e2bc" (UID: "147e25a7-dda1-4170-9453-44896e44e2bc"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 03:01:13.556805 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:01:13.556793 4405 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5ebc4fc9-8882-43c6-8713-1ee995a8ee00-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5ebc4fc9-8882-43c6-8713-1ee995a8ee00-istiod-ca-cert\") pod \"5ebc4fc9-8882-43c6-8713-1ee995a8ee00\" (UID: \"5ebc4fc9-8882-43c6-8713-1ee995a8ee00\") " Nov 04 03:01:13.557127 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:01:13.556855 4405 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/147e25a7-dda1-4170-9453-44896e44e2bc-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/147e25a7-dda1-4170-9453-44896e44e2bc-istio-data\") pod \"147e25a7-dda1-4170-9453-44896e44e2bc\" (UID: \"147e25a7-dda1-4170-9453-44896e44e2bc\") " Nov 04 03:01:13.557127 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:01:13.556919 4405 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5ebc4fc9-8882-43c6-8713-1ee995a8ee00-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5ebc4fc9-8882-43c6-8713-1ee995a8ee00-workload-socket\") pod \"5ebc4fc9-8882-43c6-8713-1ee995a8ee00\" (UID: \"5ebc4fc9-8882-43c6-8713-1ee995a8ee00\") " Nov 04 03:01:13.557127 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:01:13.556990 4405 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/147e25a7-dda1-4170-9453-44896e44e2bc-workload-certs" pod "147e25a7-dda1-4170-9453-44896e44e2bc" (UID: "147e25a7-dda1-4170-9453-44896e44e2bc"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 03:01:13.557127 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:01:13.557060 4405 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/147e25a7-dda1-4170-9453-44896e44e2bc-istio-token\" (UniqueName: \"kubernetes.io/projected/147e25a7-dda1-4170-9453-44896e44e2bc-istio-token\") pod \"147e25a7-dda1-4170-9453-44896e44e2bc\" (UID: \"147e25a7-dda1-4170-9453-44896e44e2bc\") " Nov 04 03:01:13.557127 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:01:13.557074 4405 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5ebc4fc9-8882-43c6-8713-1ee995a8ee00-credential-socket" pod "5ebc4fc9-8882-43c6-8713-1ee995a8ee00" (UID: "5ebc4fc9-8882-43c6-8713-1ee995a8ee00"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 03:01:13.557127 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:01:13.557095 4405 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/5ebc4fc9-8882-43c6-8713-1ee995a8ee00-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5ebc4fc9-8882-43c6-8713-1ee995a8ee00-istio-podinfo\") pod \"5ebc4fc9-8882-43c6-8713-1ee995a8ee00\" (UID: \"5ebc4fc9-8882-43c6-8713-1ee995a8ee00\") " Nov 04 03:01:13.557463 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:01:13.557129 4405 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5ebc4fc9-8882-43c6-8713-1ee995a8ee00-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5ebc4fc9-8882-43c6-8713-1ee995a8ee00-istio-data\") pod \"5ebc4fc9-8882-43c6-8713-1ee995a8ee00\" (UID: \"5ebc4fc9-8882-43c6-8713-1ee995a8ee00\") " Nov 04 03:01:13.557463 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:01:13.557194 4405 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/147e25a7-dda1-4170-9453-44896e44e2bc-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/147e25a7-dda1-4170-9453-44896e44e2bc-istio-podinfo\") pod \"147e25a7-dda1-4170-9453-44896e44e2bc\" (UID: \"147e25a7-dda1-4170-9453-44896e44e2bc\") " Nov 04 03:01:13.557463 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:01:13.557232 4405 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/147e25a7-dda1-4170-9453-44896e44e2bc-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/147e25a7-dda1-4170-9453-44896e44e2bc-credential-socket\") pod \"147e25a7-dda1-4170-9453-44896e44e2bc\" (UID: \"147e25a7-dda1-4170-9453-44896e44e2bc\") " Nov 04 03:01:13.557463 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:01:13.557310 4405 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5ebc4fc9-8882-43c6-8713-1ee995a8ee00-workload-socket" pod "5ebc4fc9-8882-43c6-8713-1ee995a8ee00" (UID: "5ebc4fc9-8882-43c6-8713-1ee995a8ee00"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 03:01:13.557463 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:01:13.557331 4405 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5ebc4fc9-8882-43c6-8713-1ee995a8ee00-workload-certs" pod "5ebc4fc9-8882-43c6-8713-1ee995a8ee00" (UID: "5ebc4fc9-8882-43c6-8713-1ee995a8ee00"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 03:01:13.557463 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:01:13.557377 4405 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5ebc4fc9-8882-43c6-8713-1ee995a8ee00-istiod-ca-cert" pod "5ebc4fc9-8882-43c6-8713-1ee995a8ee00" (UID: "5ebc4fc9-8882-43c6-8713-1ee995a8ee00"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 03:01:13.557794 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:01:13.557495 4405 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/147e25a7-dda1-4170-9453-44896e44e2bc-istio-data" pod "147e25a7-dda1-4170-9453-44896e44e2bc" (UID: "147e25a7-dda1-4170-9453-44896e44e2bc"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 03:01:13.557794 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:01:13.557642 4405 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/147e25a7-dda1-4170-9453-44896e44e2bc-workload-socket" pod "147e25a7-dda1-4170-9453-44896e44e2bc" (UID: "147e25a7-dda1-4170-9453-44896e44e2bc"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 03:01:13.557794 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:01:13.557727 4405 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/147e25a7-dda1-4170-9453-44896e44e2bc-credential-socket" pod "147e25a7-dda1-4170-9453-44896e44e2bc" (UID: "147e25a7-dda1-4170-9453-44896e44e2bc"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 03:01:13.557971 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:01:13.557951 4405 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5ebc4fc9-8882-43c6-8713-1ee995a8ee00-istio-data" pod "5ebc4fc9-8882-43c6-8713-1ee995a8ee00" (UID: "5ebc4fc9-8882-43c6-8713-1ee995a8ee00"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 03:01:13.558111 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:01:13.558086 4405 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/147e25a7-dda1-4170-9453-44896e44e2bc-workload-certs\") on node \"ip-10-0-134-125.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:01:13.558191 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:01:13.558121 4405 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5ebc4fc9-8882-43c6-8713-1ee995a8ee00-credential-socket\") on node \"ip-10-0-134-125.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:01:13.558191 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:01:13.558134 4405 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/147e25a7-dda1-4170-9453-44896e44e2bc-workload-socket\") on node \"ip-10-0-134-125.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:01:13.558191 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:01:13.558150 4405 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5ebc4fc9-8882-43c6-8713-1ee995a8ee00-workload-certs\") on node \"ip-10-0-134-125.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:01:13.558191 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:01:13.558163 4405 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5ebc4fc9-8882-43c6-8713-1ee995a8ee00-istiod-ca-cert\") on node \"ip-10-0-134-125.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:01:13.558191 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:01:13.558175 4405 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/147e25a7-dda1-4170-9453-44896e44e2bc-istio-data\") on node \"ip-10-0-134-125.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:01:13.558191 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:01:13.558188 4405 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5ebc4fc9-8882-43c6-8713-1ee995a8ee00-workload-socket\") on node \"ip-10-0-134-125.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:01:13.558726 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:01:13.558204 4405 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qlgg7\" (UniqueName: \"kubernetes.io/projected/89ddc69b-94de-4443-a62a-fa5d360aa368-kube-api-access-qlgg7\") on node \"ip-10-0-134-125.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:01:13.558726 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:01:13.558220 4405 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/89ddc69b-94de-4443-a62a-fa5d360aa368-istio-podinfo\") on node \"ip-10-0-134-125.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:01:13.558726 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:01:13.558233 4405 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5ebc4fc9-8882-43c6-8713-1ee995a8ee00-istio-data\") on node \"ip-10-0-134-125.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:01:13.558726 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:01:13.558246 4405 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/147e25a7-dda1-4170-9453-44896e44e2bc-credential-socket\") on node \"ip-10-0-134-125.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:01:13.558726 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:01:13.558260 4405 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/147e25a7-dda1-4170-9453-44896e44e2bc-istiod-ca-cert\") on node \"ip-10-0-134-125.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:01:13.559842 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:01:13.559806 4405 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/147e25a7-dda1-4170-9453-44896e44e2bc-kube-api-access-jf2xt" pod "147e25a7-dda1-4170-9453-44896e44e2bc" (UID: "147e25a7-dda1-4170-9453-44896e44e2bc"). InnerVolumeSpecName "kube-api-access-jf2xt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 03:01:13.560353 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:01:13.560301 4405 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/147e25a7-dda1-4170-9453-44896e44e2bc-istio-envoy" pod "147e25a7-dda1-4170-9453-44896e44e2bc" (UID: "147e25a7-dda1-4170-9453-44896e44e2bc"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 03:01:13.560504 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:01:13.560478 4405 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5ebc4fc9-8882-43c6-8713-1ee995a8ee00-istio-token" pod "5ebc4fc9-8882-43c6-8713-1ee995a8ee00" (UID: "5ebc4fc9-8882-43c6-8713-1ee995a8ee00"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 03:01:13.561120 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:01:13.561098 4405 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/147e25a7-dda1-4170-9453-44896e44e2bc-istio-podinfo" pod "147e25a7-dda1-4170-9453-44896e44e2bc" (UID: "147e25a7-dda1-4170-9453-44896e44e2bc"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 04 03:01:13.561218 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:01:13.561141 4405 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/5ebc4fc9-8882-43c6-8713-1ee995a8ee00-istio-podinfo" pod "5ebc4fc9-8882-43c6-8713-1ee995a8ee00" (UID: "5ebc4fc9-8882-43c6-8713-1ee995a8ee00"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 04 03:01:13.561218 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:01:13.561150 4405 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/147e25a7-dda1-4170-9453-44896e44e2bc-istio-token" pod "147e25a7-dda1-4170-9453-44896e44e2bc" (UID: "147e25a7-dda1-4170-9453-44896e44e2bc"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 03:01:13.561488 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:01:13.561458 4405 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5ebc4fc9-8882-43c6-8713-1ee995a8ee00-istio-envoy" pod "5ebc4fc9-8882-43c6-8713-1ee995a8ee00" (UID: "5ebc4fc9-8882-43c6-8713-1ee995a8ee00"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 03:01:13.561608 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:01:13.561583 4405 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5ebc4fc9-8882-43c6-8713-1ee995a8ee00-kube-api-access-br6lx" pod "5ebc4fc9-8882-43c6-8713-1ee995a8ee00" (UID: "5ebc4fc9-8882-43c6-8713-1ee995a8ee00"). InnerVolumeSpecName "kube-api-access-br6lx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 03:01:13.666814 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:01:13.661134 4405 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/147e25a7-dda1-4170-9453-44896e44e2bc-istio-token\") on node \"ip-10-0-134-125.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:01:13.666814 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:01:13.661170 4405 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5ebc4fc9-8882-43c6-8713-1ee995a8ee00-istio-podinfo\") on node \"ip-10-0-134-125.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:01:13.666814 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:01:13.661185 4405 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/147e25a7-dda1-4170-9453-44896e44e2bc-istio-podinfo\") on node \"ip-10-0-134-125.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:01:13.666814 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:01:13.661199 4405 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/147e25a7-dda1-4170-9453-44896e44e2bc-istio-envoy\") on node \"ip-10-0-134-125.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:01:13.666814 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:01:13.661211 4405 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5ebc4fc9-8882-43c6-8713-1ee995a8ee00-istio-envoy\") on node \"ip-10-0-134-125.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:01:13.666814 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:01:13.661226 4405 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jf2xt\" (UniqueName: \"kubernetes.io/projected/147e25a7-dda1-4170-9453-44896e44e2bc-kube-api-access-jf2xt\") on node \"ip-10-0-134-125.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:01:13.666814 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:01:13.661241 4405 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5ebc4fc9-8882-43c6-8713-1ee995a8ee00-istio-token\") on node \"ip-10-0-134-125.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:01:13.666814 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:01:13.661253 4405 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-br6lx\" (UniqueName: \"kubernetes.io/projected/5ebc4fc9-8882-43c6-8713-1ee995a8ee00-kube-api-access-br6lx\") on node \"ip-10-0-134-125.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:01:13.932955 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:01:13.931811 4405 generic.go:356] "Generic (PLEG): container finished" podID="89ddc69b-94de-4443-a62a-fa5d360aa368" containerID="9c709331c40b10e0a93eacdbe663fc93145c774d2e72f230525dbf41228825c3" exitCode=0 Nov 04 03:01:13.932955 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:01:13.931967 4405 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-203c35-openshift-default-56497cdcc-w6c48" event={"ID":"89ddc69b-94de-4443-a62a-fa5d360aa368","Type":"ContainerDied","Data":"9c709331c40b10e0a93eacdbe663fc93145c774d2e72f230525dbf41228825c3"} Nov 04 03:01:13.932955 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:01:13.932029 4405 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-203c35-openshift-default-56497cdcc-w6c48" event={"ID":"89ddc69b-94de-4443-a62a-fa5d360aa368","Type":"ContainerDied","Data":"ecde9492b94a720eaedb5dfa26aa3cef4a4e26b55be9b0fd80ac46517ed34105"} Nov 04 03:01:13.932955 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:01:13.932062 4405 scope.go:122] "RemoveContainer" containerID="9c709331c40b10e0a93eacdbe663fc93145c774d2e72f230525dbf41228825c3" Nov 04 03:01:13.932955 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:01:13.932276 4405 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-203c35-openshift-default-56497cdcc-w6c48" Nov 04 03:01:13.941525 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:01:13.941403 4405 generic.go:356] "Generic (PLEG): container finished" podID="5ebc4fc9-8882-43c6-8713-1ee995a8ee00" containerID="ffba5bc76b8a20d2d74d2d5516d95428cd0e26f7e0bfddb238c77058347ceb52" exitCode=0 Nov 04 03:01:13.942172 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:01:13.942099 4405 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-203c35-openshift-default-56497cdcc-z59vn" Nov 04 03:01:13.943589 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:01:13.943532 4405 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-203c35-openshift-default-56497cdcc-z59vn" event={"ID":"5ebc4fc9-8882-43c6-8713-1ee995a8ee00","Type":"ContainerDied","Data":"ffba5bc76b8a20d2d74d2d5516d95428cd0e26f7e0bfddb238c77058347ceb52"} Nov 04 03:01:13.943789 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:01:13.943642 4405 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-203c35-openshift-default-56497cdcc-z59vn" event={"ID":"5ebc4fc9-8882-43c6-8713-1ee995a8ee00","Type":"ContainerDied","Data":"2b0362fac9ac89e44165fc0ad4de04a0f5171d85b3479a0e693e3c2a202cb580"} Nov 04 03:01:13.951501 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:01:13.949158 4405 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w4-35df3e-7d67b7f575-zbnm5"] Nov 04 03:01:13.951501 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:01:13.951139 4405 cpu_manager.go:414] "containerMap: removing container" podUID="1d9a643a-06d7-4e7e-85bb-a87e2108864f" containerName="cleanup" Nov 04 03:01:13.951501 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:01:13.951166 4405 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1d9a643a-06d7-4e7e-85bb-a87e2108864f" containerName="cleanup" Nov 04 03:01:13.951501 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:01:13.951227 4405 cpu_manager.go:414] "containerMap: removing container" podUID="89ddc69b-94de-4443-a62a-fa5d360aa368" containerName="istio-proxy" Nov 04 03:01:13.951501 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:01:13.951237 4405 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="89ddc69b-94de-4443-a62a-fa5d360aa368" containerName="istio-proxy" Nov 04 03:01:13.951501 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:01:13.951404 4405 cpu_manager.go:414] "containerMap: removing container" podUID="71ce2ec9-0edd-483f-b364-b45e15ccef58" containerName="cleanup" Nov 04 03:01:13.951501 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:01:13.951415 4405 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="71ce2ec9-0edd-483f-b364-b45e15ccef58" containerName="cleanup" Nov 04 03:01:13.951501 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:01:13.951441 4405 cpu_manager.go:414] "containerMap: removing container" podUID="42766046-4bb9-4afd-8fe7-3bc0e0199d1e" containerName="cleanup" Nov 04 03:01:13.951501 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:01:13.951449 4405 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="42766046-4bb9-4afd-8fe7-3bc0e0199d1e" containerName="cleanup" Nov 04 03:01:13.952441 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:01:13.951687 4405 cpu_manager.go:414] "containerMap: removing container" podUID="147e25a7-dda1-4170-9453-44896e44e2bc" containerName="istio-proxy" Nov 04 03:01:13.952441 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:01:13.951701 4405 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="147e25a7-dda1-4170-9453-44896e44e2bc" containerName="istio-proxy" Nov 04 03:01:13.952441 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:01:13.951731 4405 cpu_manager.go:414] "containerMap: removing container" podUID="5ebc4fc9-8882-43c6-8713-1ee995a8ee00" containerName="istio-proxy" Nov 04 03:01:13.952441 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:01:13.951738 4405 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5ebc4fc9-8882-43c6-8713-1ee995a8ee00" containerName="istio-proxy" Nov 04 03:01:13.953049 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:01:13.952843 4405 generic.go:356] "Generic (PLEG): container finished" podID="147e25a7-dda1-4170-9453-44896e44e2bc" containerID="9493dae8e6eaed626d077d1b19415b29a1cf533152cc9a5774624d4c6db883b5" exitCode=0 Nov 04 03:01:13.953049 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:01:13.952957 4405 memory_manager.go:365] "RemoveStaleState removing state" podUID="42766046-4bb9-4afd-8fe7-3bc0e0199d1e" containerName="cleanup" Nov 04 03:01:13.953049 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:01:13.952987 4405 memory_manager.go:365] "RemoveStaleState removing state" podUID="147e25a7-dda1-4170-9453-44896e44e2bc" containerName="istio-proxy" Nov 04 03:01:13.953049 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:01:13.953019 4405 memory_manager.go:365] "RemoveStaleState removing state" podUID="5ebc4fc9-8882-43c6-8713-1ee995a8ee00" containerName="istio-proxy" Nov 04 03:01:13.953049 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:01:13.953032 4405 memory_manager.go:365] "RemoveStaleState removing state" podUID="71ce2ec9-0edd-483f-b364-b45e15ccef58" containerName="cleanup" Nov 04 03:01:13.953394 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:01:13.953058 4405 memory_manager.go:365] "RemoveStaleState removing state" podUID="89ddc69b-94de-4443-a62a-fa5d360aa368" containerName="istio-proxy" Nov 04 03:01:13.953442 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:01:13.953423 4405 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-203c35-openshift-default-56497cdcc-2b66r" Nov 04 03:01:13.960048 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:01:13.959708 4405 generic.go:356] "Generic (PLEG): container finished" podID="fec507be-e742-4226-a6c4-6fb41e6f3479" containerID="dc932a567f19c37532b049ce74104a0a9cefc967ab7c48ef6616658d921d5c76" exitCode=0 Nov 04 03:01:13.968652 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:01:13.968613 4405 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-203c35-openshift-default-56497cdcc-ljwqc" Nov 04 03:01:13.972857 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:01:13.972820 4405 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-203c35-openshift-default-56497cdcc-2b66r" event={"ID":"147e25a7-dda1-4170-9453-44896e44e2bc","Type":"ContainerDied","Data":"9493dae8e6eaed626d077d1b19415b29a1cf533152cc9a5774624d4c6db883b5"} Nov 04 03:01:13.973027 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:01:13.973012 4405 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-203c35-openshift-default-56497cdcc-2b66r" event={"ID":"147e25a7-dda1-4170-9453-44896e44e2bc","Type":"ContainerDied","Data":"c7c598b47e4fc9fb9d4530d8381ac46d074538393d9c17b8d34a86609b51689d"} Nov 04 03:01:13.973120 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:01:13.973108 4405 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-203c35-openshift-default-56497cdcc-ljwqc" event={"ID":"fec507be-e742-4226-a6c4-6fb41e6f3479","Type":"ContainerDied","Data":"dc932a567f19c37532b049ce74104a0a9cefc967ab7c48ef6616658d921d5c76"} Nov 04 03:01:13.973285 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:01:13.973271 4405 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-35df3e-7d67b7f575-zbnm5" Nov 04 03:01:13.979729 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:01:13.979702 4405 scope.go:122] "RemoveContainer" containerID="9c709331c40b10e0a93eacdbe663fc93145c774d2e72f230525dbf41228825c3" Nov 04 03:01:13.980133 ip-10-0-134-125 kubenswrapper[4405]: E1104 03:01:13.980111 4405 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9c709331c40b10e0a93eacdbe663fc93145c774d2e72f230525dbf41228825c3\": container with ID starting with 9c709331c40b10e0a93eacdbe663fc93145c774d2e72f230525dbf41228825c3 not found: ID does not exist" containerID="9c709331c40b10e0a93eacdbe663fc93145c774d2e72f230525dbf41228825c3" Nov 04 03:01:13.980244 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:01:13.980142 4405 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9c709331c40b10e0a93eacdbe663fc93145c774d2e72f230525dbf41228825c3"} err="failed to get container status \"9c709331c40b10e0a93eacdbe663fc93145c774d2e72f230525dbf41228825c3\": rpc error: code = NotFound desc = could not find container \"9c709331c40b10e0a93eacdbe663fc93145c774d2e72f230525dbf41228825c3\": container with ID starting with 9c709331c40b10e0a93eacdbe663fc93145c774d2e72f230525dbf41228825c3 not found: ID does not exist" Nov 04 03:01:13.980244 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:01:13.980161 4405 scope.go:122] "RemoveContainer" containerID="ffba5bc76b8a20d2d74d2d5516d95428cd0e26f7e0bfddb238c77058347ceb52" Nov 04 03:01:13.997179 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:01:13.997141 4405 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w4-35df3e-7d67b7f575-zbnm5"] Nov 04 03:01:13.997344 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:01:13.997165 4405 scope.go:122] "RemoveContainer" containerID="ffba5bc76b8a20d2d74d2d5516d95428cd0e26f7e0bfddb238c77058347ceb52" Nov 04 03:01:13.997881 ip-10-0-134-125 kubenswrapper[4405]: E1104 03:01:13.997841 4405 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ffba5bc76b8a20d2d74d2d5516d95428cd0e26f7e0bfddb238c77058347ceb52\": container with ID starting with ffba5bc76b8a20d2d74d2d5516d95428cd0e26f7e0bfddb238c77058347ceb52 not found: ID does not exist" containerID="ffba5bc76b8a20d2d74d2d5516d95428cd0e26f7e0bfddb238c77058347ceb52" Nov 04 03:01:13.998039 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:01:13.997891 4405 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ffba5bc76b8a20d2d74d2d5516d95428cd0e26f7e0bfddb238c77058347ceb52"} err="failed to get container status \"ffba5bc76b8a20d2d74d2d5516d95428cd0e26f7e0bfddb238c77058347ceb52\": rpc error: code = NotFound desc = could not find container \"ffba5bc76b8a20d2d74d2d5516d95428cd0e26f7e0bfddb238c77058347ceb52\": container with ID starting with ffba5bc76b8a20d2d74d2d5516d95428cd0e26f7e0bfddb238c77058347ceb52 not found: ID does not exist" Nov 04 03:01:13.998039 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:01:13.997914 4405 scope.go:122] "RemoveContainer" containerID="9493dae8e6eaed626d077d1b19415b29a1cf533152cc9a5774624d4c6db883b5" Nov 04 03:01:14.014458 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:01:14.014432 4405 scope.go:122] "RemoveContainer" containerID="9493dae8e6eaed626d077d1b19415b29a1cf533152cc9a5774624d4c6db883b5" Nov 04 03:01:14.015433 ip-10-0-134-125 kubenswrapper[4405]: E1104 03:01:14.015405 4405 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9493dae8e6eaed626d077d1b19415b29a1cf533152cc9a5774624d4c6db883b5\": container with ID starting with 9493dae8e6eaed626d077d1b19415b29a1cf533152cc9a5774624d4c6db883b5 not found: ID does not exist" containerID="9493dae8e6eaed626d077d1b19415b29a1cf533152cc9a5774624d4c6db883b5" Nov 04 03:01:14.015551 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:01:14.015447 4405 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9493dae8e6eaed626d077d1b19415b29a1cf533152cc9a5774624d4c6db883b5"} err="failed to get container status \"9493dae8e6eaed626d077d1b19415b29a1cf533152cc9a5774624d4c6db883b5\": rpc error: code = NotFound desc = could not find container \"9493dae8e6eaed626d077d1b19415b29a1cf533152cc9a5774624d4c6db883b5\": container with ID starting with 9493dae8e6eaed626d077d1b19415b29a1cf533152cc9a5774624d4c6db883b5 not found: ID does not exist" Nov 04 03:01:14.015551 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:01:14.015473 4405 scope.go:122] "RemoveContainer" containerID="dc932a567f19c37532b049ce74104a0a9cefc967ab7c48ef6616658d921d5c76" Nov 04 03:01:14.028299 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:01:14.028242 4405 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w4-35df3e-dockercfg-hr7xx\"" Nov 04 03:01:14.029729 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:01:14.029709 4405 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w4-35df3e\"" Nov 04 03:01:14.073414 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:01:14.073385 4405 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w4-35df3e-7969f6bd6b-ptzqg"] Nov 04 03:01:14.074103 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:01:14.074087 4405 cpu_manager.go:414] "containerMap: removing container" podUID="fec507be-e742-4226-a6c4-6fb41e6f3479" containerName="istio-proxy" Nov 04 03:01:14.074157 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:01:14.074105 4405 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fec507be-e742-4226-a6c4-6fb41e6f3479" containerName="istio-proxy" Nov 04 03:01:14.074209 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:01:14.074200 4405 memory_manager.go:365] "RemoveStaleState removing state" podUID="fec507be-e742-4226-a6c4-6fb41e6f3479" containerName="istio-proxy" Nov 04 03:01:14.079295 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:01:14.079271 4405 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-35df3e-7969f6bd6b-ptzqg" Nov 04 03:01:14.088554 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:01:14.088516 4405 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fec507be-e742-4226-a6c4-6fb41e6f3479-istio-token\" (UniqueName: \"kubernetes.io/projected/fec507be-e742-4226-a6c4-6fb41e6f3479-istio-token\") pod \"fec507be-e742-4226-a6c4-6fb41e6f3479\" (UID: \"fec507be-e742-4226-a6c4-6fb41e6f3479\") " Nov 04 03:01:14.088733 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:01:14.088568 4405 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fec507be-e742-4226-a6c4-6fb41e6f3479-kube-api-access-ntjgq\" (UniqueName: \"kubernetes.io/projected/fec507be-e742-4226-a6c4-6fb41e6f3479-kube-api-access-ntjgq\") pod \"fec507be-e742-4226-a6c4-6fb41e6f3479\" (UID: \"fec507be-e742-4226-a6c4-6fb41e6f3479\") " Nov 04 03:01:14.088733 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:01:14.088602 4405 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fec507be-e742-4226-a6c4-6fb41e6f3479-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fec507be-e742-4226-a6c4-6fb41e6f3479-workload-certs\") pod \"fec507be-e742-4226-a6c4-6fb41e6f3479\" (UID: \"fec507be-e742-4226-a6c4-6fb41e6f3479\") " Nov 04 03:01:14.088864 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:01:14.088735 4405 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fec507be-e742-4226-a6c4-6fb41e6f3479-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fec507be-e742-4226-a6c4-6fb41e6f3479-workload-socket\") pod \"fec507be-e742-4226-a6c4-6fb41e6f3479\" (UID: \"fec507be-e742-4226-a6c4-6fb41e6f3479\") " Nov 04 03:01:14.088864 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:01:14.088815 4405 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fec507be-e742-4226-a6c4-6fb41e6f3479-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fec507be-e742-4226-a6c4-6fb41e6f3479-credential-socket\") pod \"fec507be-e742-4226-a6c4-6fb41e6f3479\" (UID: \"fec507be-e742-4226-a6c4-6fb41e6f3479\") " Nov 04 03:01:14.088970 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:01:14.088876 4405 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fec507be-e742-4226-a6c4-6fb41e6f3479-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fec507be-e742-4226-a6c4-6fb41e6f3479-istio-data\") pod \"fec507be-e742-4226-a6c4-6fb41e6f3479\" (UID: \"fec507be-e742-4226-a6c4-6fb41e6f3479\") " Nov 04 03:01:14.088970 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:01:14.088909 4405 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/fec507be-e742-4226-a6c4-6fb41e6f3479-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fec507be-e742-4226-a6c4-6fb41e6f3479-istio-podinfo\") pod \"fec507be-e742-4226-a6c4-6fb41e6f3479\" (UID: \"fec507be-e742-4226-a6c4-6fb41e6f3479\") " Nov 04 03:01:14.088970 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:01:14.088955 4405 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fec507be-e742-4226-a6c4-6fb41e6f3479-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fec507be-e742-4226-a6c4-6fb41e6f3479-istiod-ca-cert\") pod \"fec507be-e742-4226-a6c4-6fb41e6f3479\" (UID: \"fec507be-e742-4226-a6c4-6fb41e6f3479\") " Nov 04 03:01:14.089139 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:01:14.088987 4405 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fec507be-e742-4226-a6c4-6fb41e6f3479-workload-certs" pod "fec507be-e742-4226-a6c4-6fb41e6f3479" (UID: "fec507be-e742-4226-a6c4-6fb41e6f3479"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 03:01:14.089196 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:01:14.089147 4405 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fec507be-e742-4226-a6c4-6fb41e6f3479-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fec507be-e742-4226-a6c4-6fb41e6f3479-istio-envoy\") pod \"fec507be-e742-4226-a6c4-6fb41e6f3479\" (UID: \"fec507be-e742-4226-a6c4-6fb41e6f3479\") " Nov 04 03:01:14.089252 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:01:14.089201 4405 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fec507be-e742-4226-a6c4-6fb41e6f3479-workload-socket" pod "fec507be-e742-4226-a6c4-6fb41e6f3479" (UID: "fec507be-e742-4226-a6c4-6fb41e6f3479"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 03:01:14.089358 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:01:14.089242 4405 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fec507be-e742-4226-a6c4-6fb41e6f3479-istio-data" pod "fec507be-e742-4226-a6c4-6fb41e6f3479" (UID: "fec507be-e742-4226-a6c4-6fb41e6f3479"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 03:01:14.089486 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:01:14.089458 4405 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jp8pj\" (UniqueName: \"kubernetes.io/projected/317d0b2a-1ab0-4278-add3-cfaee93aa69c-kube-api-access-jp8pj\") pod \"payload-pre-processing-e2e-shared-b-w4-35df3e-7d67b7f575-zbnm5\" (UID: \"317d0b2a-1ab0-4278-add3-cfaee93aa69c\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-35df3e-7d67b7f575-zbnm5" Nov 04 03:01:14.089650 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:01:14.089469 4405 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fec507be-e742-4226-a6c4-6fb41e6f3479-credential-socket" pod "fec507be-e742-4226-a6c4-6fb41e6f3479" (UID: "fec507be-e742-4226-a6c4-6fb41e6f3479"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 03:01:14.089650 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:01:14.089590 4405 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fec507be-e742-4226-a6c4-6fb41e6f3479-istiod-ca-cert" pod "fec507be-e742-4226-a6c4-6fb41e6f3479" (UID: "fec507be-e742-4226-a6c4-6fb41e6f3479"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 03:01:14.090085 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:01:14.090059 4405 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/317d0b2a-1ab0-4278-add3-cfaee93aa69c-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w4-35df3e-7d67b7f575-zbnm5\" (UID: \"317d0b2a-1ab0-4278-add3-cfaee93aa69c\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-35df3e-7d67b7f575-zbnm5" Nov 04 03:01:14.090337 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:01:14.090318 4405 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fec507be-e742-4226-a6c4-6fb41e6f3479-workload-certs\") on node \"ip-10-0-134-125.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:01:14.090422 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:01:14.090343 4405 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fec507be-e742-4226-a6c4-6fb41e6f3479-workload-socket\") on node \"ip-10-0-134-125.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:01:14.090422 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:01:14.090357 4405 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fec507be-e742-4226-a6c4-6fb41e6f3479-credential-socket\") on node \"ip-10-0-134-125.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:01:14.090422 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:01:14.090371 4405 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fec507be-e742-4226-a6c4-6fb41e6f3479-istio-data\") on node \"ip-10-0-134-125.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:01:14.090422 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:01:14.090385 4405 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fec507be-e742-4226-a6c4-6fb41e6f3479-istiod-ca-cert\") on node \"ip-10-0-134-125.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:01:14.091379 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:01:14.091358 4405 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fec507be-e742-4226-a6c4-6fb41e6f3479-kube-api-access-ntjgq" pod "fec507be-e742-4226-a6c4-6fb41e6f3479" (UID: "fec507be-e742-4226-a6c4-6fb41e6f3479"). InnerVolumeSpecName "kube-api-access-ntjgq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 03:01:14.091755 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:01:14.091717 4405 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/fec507be-e742-4226-a6c4-6fb41e6f3479-istio-podinfo" pod "fec507be-e742-4226-a6c4-6fb41e6f3479" (UID: "fec507be-e742-4226-a6c4-6fb41e6f3479"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 04 03:01:14.091861 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:01:14.091802 4405 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fec507be-e742-4226-a6c4-6fb41e6f3479-istio-token" pod "fec507be-e742-4226-a6c4-6fb41e6f3479" (UID: "fec507be-e742-4226-a6c4-6fb41e6f3479"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 03:01:14.093501 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:01:14.093476 4405 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fec507be-e742-4226-a6c4-6fb41e6f3479-istio-envoy" pod "fec507be-e742-4226-a6c4-6fb41e6f3479" (UID: "fec507be-e742-4226-a6c4-6fb41e6f3479"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 03:01:14.146059 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:01:14.145995 4405 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-203c35-openshift-default-56497cdcc-z59vn"] Nov 04 03:01:14.156768 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:01:14.156734 4405 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w4-35df3e-7969f6bd6b-ptzqg"] Nov 04 03:01:14.214349 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:01:14.214247 4405 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qr8hr\" (UniqueName: \"kubernetes.io/projected/8d7f28e3-d62b-4813-81e3-364d4700983a-kube-api-access-qr8hr\") pod \"payload-processing-e2e-shared-b-w4-35df3e-7969f6bd6b-ptzqg\" (UID: \"8d7f28e3-d62b-4813-81e3-364d4700983a\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w4-35df3e-7969f6bd6b-ptzqg" Nov 04 03:01:14.217938 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:01:14.217898 4405 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/317d0b2a-1ab0-4278-add3-cfaee93aa69c-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w4-35df3e-7d67b7f575-zbnm5\" (UID: \"317d0b2a-1ab0-4278-add3-cfaee93aa69c\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-35df3e-7d67b7f575-zbnm5" Nov 04 03:01:14.218267 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:01:14.218245 4405 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jp8pj\" (UniqueName: \"kubernetes.io/projected/317d0b2a-1ab0-4278-add3-cfaee93aa69c-kube-api-access-jp8pj\") pod \"payload-pre-processing-e2e-shared-b-w4-35df3e-7d67b7f575-zbnm5\" (UID: \"317d0b2a-1ab0-4278-add3-cfaee93aa69c\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-35df3e-7d67b7f575-zbnm5" Nov 04 03:01:14.218365 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:01:14.217055 4405 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-203c35-openshift-default-56497cdcc-z59vn"] Nov 04 03:01:14.218483 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:01:14.218451 4405 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8d7f28e3-d62b-4813-81e3-364d4700983a-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w4-35df3e-7969f6bd6b-ptzqg\" (UID: \"8d7f28e3-d62b-4813-81e3-364d4700983a\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w4-35df3e-7969f6bd6b-ptzqg" Nov 04 03:01:14.218640 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:01:14.218621 4405 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fec507be-e742-4226-a6c4-6fb41e6f3479-istio-envoy\") on node \"ip-10-0-134-125.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:01:14.218721 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:01:14.218646 4405 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fec507be-e742-4226-a6c4-6fb41e6f3479-istio-token\") on node \"ip-10-0-134-125.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:01:14.218721 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:01:14.218662 4405 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ntjgq\" (UniqueName: \"kubernetes.io/projected/fec507be-e742-4226-a6c4-6fb41e6f3479-kube-api-access-ntjgq\") on node \"ip-10-0-134-125.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:01:14.218721 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:01:14.218698 4405 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fec507be-e742-4226-a6c4-6fb41e6f3479-istio-podinfo\") on node \"ip-10-0-134-125.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:01:14.219650 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:01:14.219627 4405 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/317d0b2a-1ab0-4278-add3-cfaee93aa69c-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w4-35df3e-7d67b7f575-zbnm5\" (UID: \"317d0b2a-1ab0-4278-add3-cfaee93aa69c\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-35df3e-7d67b7f575-zbnm5" Nov 04 03:01:14.334751 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:01:14.322636 4405 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qr8hr\" (UniqueName: \"kubernetes.io/projected/8d7f28e3-d62b-4813-81e3-364d4700983a-kube-api-access-qr8hr\") pod \"payload-processing-e2e-shared-b-w4-35df3e-7969f6bd6b-ptzqg\" (UID: \"8d7f28e3-d62b-4813-81e3-364d4700983a\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w4-35df3e-7969f6bd6b-ptzqg" Nov 04 03:01:14.334751 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:01:14.322998 4405 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8d7f28e3-d62b-4813-81e3-364d4700983a-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w4-35df3e-7969f6bd6b-ptzqg\" (UID: \"8d7f28e3-d62b-4813-81e3-364d4700983a\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w4-35df3e-7969f6bd6b-ptzqg" Nov 04 03:01:14.334751 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:01:14.323987 4405 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8d7f28e3-d62b-4813-81e3-364d4700983a-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w4-35df3e-7969f6bd6b-ptzqg\" (UID: \"8d7f28e3-d62b-4813-81e3-364d4700983a\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w4-35df3e-7969f6bd6b-ptzqg" Nov 04 03:01:14.334751 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:01:14.325310 4405 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jp8pj\" (UniqueName: \"kubernetes.io/projected/317d0b2a-1ab0-4278-add3-cfaee93aa69c-kube-api-access-jp8pj\") pod \"payload-pre-processing-e2e-shared-b-w4-35df3e-7d67b7f575-zbnm5\" (UID: \"317d0b2a-1ab0-4278-add3-cfaee93aa69c\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-35df3e-7d67b7f575-zbnm5" Nov 04 03:01:14.385865 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:01:14.385826 4405 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qr8hr\" (UniqueName: \"kubernetes.io/projected/8d7f28e3-d62b-4813-81e3-364d4700983a-kube-api-access-qr8hr\") pod \"payload-processing-e2e-shared-b-w4-35df3e-7969f6bd6b-ptzqg\" (UID: \"8d7f28e3-d62b-4813-81e3-364d4700983a\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w4-35df3e-7969f6bd6b-ptzqg" Nov 04 03:01:14.444378 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:01:14.443373 4405 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-35df3e-7969f6bd6b-ptzqg" Nov 04 03:01:14.444378 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:01:14.444130 4405 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w2-57f116-openshift-default-75f4fc499-znglh"] Nov 04 03:01:14.448619 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:01:14.448600 4405 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-57f116-openshift-default-75f4fc499-znglh" Nov 04 03:01:14.492216 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:01:14.492174 4405 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-203c35-openshift-default-56497cdcc-2b66r"] Nov 04 03:01:14.498061 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:01:14.498029 4405 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-203c35-openshift-default-56497cdcc-2b66r"] Nov 04 03:01:14.502980 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:01:14.502952 4405 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w2-57f116-openshift-default-dockercfg-w8lwk\"" Nov 04 03:01:14.508288 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:01:14.508261 4405 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-57f116-openshift-default-75f4fc499-znglh"] Nov 04 03:01:14.532368 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:01:14.532311 4405 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fd2b4daf-d65f-44ca-8939-67778fe8ef5a-workload-socket\") pod \"e2e-shared-a-w2-57f116-openshift-default-75f4fc499-znglh\" (UID: \"fd2b4daf-d65f-44ca-8939-67778fe8ef5a\") " pod="openshift-ingress/e2e-shared-a-w2-57f116-openshift-default-75f4fc499-znglh" Nov 04 03:01:14.532636 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:01:14.532609 4405 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fd2b4daf-d65f-44ca-8939-67778fe8ef5a-istio-data\") pod \"e2e-shared-a-w2-57f116-openshift-default-75f4fc499-znglh\" (UID: \"fd2b4daf-d65f-44ca-8939-67778fe8ef5a\") " pod="openshift-ingress/e2e-shared-a-w2-57f116-openshift-default-75f4fc499-znglh" Nov 04 03:01:14.533130 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:01:14.533106 4405 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fd2b4daf-d65f-44ca-8939-67778fe8ef5a-istio-token\") pod \"e2e-shared-a-w2-57f116-openshift-default-75f4fc499-znglh\" (UID: \"fd2b4daf-d65f-44ca-8939-67778fe8ef5a\") " pod="openshift-ingress/e2e-shared-a-w2-57f116-openshift-default-75f4fc499-znglh" Nov 04 03:01:14.533338 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:01:14.533319 4405 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fd2b4daf-d65f-44ca-8939-67778fe8ef5a-istio-envoy\") pod \"e2e-shared-a-w2-57f116-openshift-default-75f4fc499-znglh\" (UID: \"fd2b4daf-d65f-44ca-8939-67778fe8ef5a\") " pod="openshift-ingress/e2e-shared-a-w2-57f116-openshift-default-75f4fc499-znglh" Nov 04 03:01:14.533478 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:01:14.533461 4405 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fd2b4daf-d65f-44ca-8939-67778fe8ef5a-istiod-ca-cert\") pod \"e2e-shared-a-w2-57f116-openshift-default-75f4fc499-znglh\" (UID: \"fd2b4daf-d65f-44ca-8939-67778fe8ef5a\") " pod="openshift-ingress/e2e-shared-a-w2-57f116-openshift-default-75f4fc499-znglh" Nov 04 03:01:14.533656 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:01:14.533640 4405 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fd2b4daf-d65f-44ca-8939-67778fe8ef5a-workload-certs\") pod \"e2e-shared-a-w2-57f116-openshift-default-75f4fc499-znglh\" (UID: \"fd2b4daf-d65f-44ca-8939-67778fe8ef5a\") " pod="openshift-ingress/e2e-shared-a-w2-57f116-openshift-default-75f4fc499-znglh" Nov 04 03:01:14.533834 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:01:14.533813 4405 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fd2b4daf-d65f-44ca-8939-67778fe8ef5a-credential-socket\") pod \"e2e-shared-a-w2-57f116-openshift-default-75f4fc499-znglh\" (UID: \"fd2b4daf-d65f-44ca-8939-67778fe8ef5a\") " pod="openshift-ingress/e2e-shared-a-w2-57f116-openshift-default-75f4fc499-znglh" Nov 04 03:01:14.533979 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:01:14.533962 4405 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fd2b4daf-d65f-44ca-8939-67778fe8ef5a-istio-podinfo\") pod \"e2e-shared-a-w2-57f116-openshift-default-75f4fc499-znglh\" (UID: \"fd2b4daf-d65f-44ca-8939-67778fe8ef5a\") " pod="openshift-ingress/e2e-shared-a-w2-57f116-openshift-default-75f4fc499-znglh" Nov 04 03:01:14.534189 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:01:14.534169 4405 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zrt6m\" (UniqueName: \"kubernetes.io/projected/fd2b4daf-d65f-44ca-8939-67778fe8ef5a-kube-api-access-zrt6m\") pod \"e2e-shared-a-w2-57f116-openshift-default-75f4fc499-znglh\" (UID: \"fd2b4daf-d65f-44ca-8939-67778fe8ef5a\") " pod="openshift-ingress/e2e-shared-a-w2-57f116-openshift-default-75f4fc499-znglh" Nov 04 03:01:14.603525 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:01:14.603449 4405 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-35df3e-7d67b7f575-zbnm5" Nov 04 03:01:14.647597 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:01:14.647548 4405 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fd2b4daf-d65f-44ca-8939-67778fe8ef5a-istio-envoy\") pod \"e2e-shared-a-w2-57f116-openshift-default-75f4fc499-znglh\" (UID: \"fd2b4daf-d65f-44ca-8939-67778fe8ef5a\") " pod="openshift-ingress/e2e-shared-a-w2-57f116-openshift-default-75f4fc499-znglh" Nov 04 03:01:14.648235 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:01:14.647616 4405 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fd2b4daf-d65f-44ca-8939-67778fe8ef5a-istiod-ca-cert\") pod \"e2e-shared-a-w2-57f116-openshift-default-75f4fc499-znglh\" (UID: \"fd2b4daf-d65f-44ca-8939-67778fe8ef5a\") " pod="openshift-ingress/e2e-shared-a-w2-57f116-openshift-default-75f4fc499-znglh" Nov 04 03:01:14.648235 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:01:14.647806 4405 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fd2b4daf-d65f-44ca-8939-67778fe8ef5a-workload-certs\") pod \"e2e-shared-a-w2-57f116-openshift-default-75f4fc499-znglh\" (UID: \"fd2b4daf-d65f-44ca-8939-67778fe8ef5a\") " pod="openshift-ingress/e2e-shared-a-w2-57f116-openshift-default-75f4fc499-znglh" Nov 04 03:01:14.648235 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:01:14.647851 4405 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fd2b4daf-d65f-44ca-8939-67778fe8ef5a-credential-socket\") pod \"e2e-shared-a-w2-57f116-openshift-default-75f4fc499-znglh\" (UID: \"fd2b4daf-d65f-44ca-8939-67778fe8ef5a\") " pod="openshift-ingress/e2e-shared-a-w2-57f116-openshift-default-75f4fc499-znglh" Nov 04 03:01:14.648235 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:01:14.647896 4405 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fd2b4daf-d65f-44ca-8939-67778fe8ef5a-istio-podinfo\") pod \"e2e-shared-a-w2-57f116-openshift-default-75f4fc499-znglh\" (UID: \"fd2b4daf-d65f-44ca-8939-67778fe8ef5a\") " pod="openshift-ingress/e2e-shared-a-w2-57f116-openshift-default-75f4fc499-znglh" Nov 04 03:01:14.648235 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:01:14.647996 4405 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zrt6m\" (UniqueName: \"kubernetes.io/projected/fd2b4daf-d65f-44ca-8939-67778fe8ef5a-kube-api-access-zrt6m\") pod \"e2e-shared-a-w2-57f116-openshift-default-75f4fc499-znglh\" (UID: \"fd2b4daf-d65f-44ca-8939-67778fe8ef5a\") " pod="openshift-ingress/e2e-shared-a-w2-57f116-openshift-default-75f4fc499-znglh" Nov 04 03:01:14.648235 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:01:14.648098 4405 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fd2b4daf-d65f-44ca-8939-67778fe8ef5a-workload-socket\") pod \"e2e-shared-a-w2-57f116-openshift-default-75f4fc499-znglh\" (UID: \"fd2b4daf-d65f-44ca-8939-67778fe8ef5a\") " pod="openshift-ingress/e2e-shared-a-w2-57f116-openshift-default-75f4fc499-znglh" Nov 04 03:01:14.648235 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:01:14.648146 4405 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fd2b4daf-d65f-44ca-8939-67778fe8ef5a-istio-data\") pod \"e2e-shared-a-w2-57f116-openshift-default-75f4fc499-znglh\" (UID: \"fd2b4daf-d65f-44ca-8939-67778fe8ef5a\") " pod="openshift-ingress/e2e-shared-a-w2-57f116-openshift-default-75f4fc499-znglh" Nov 04 03:01:14.648235 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:01:14.648210 4405 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fd2b4daf-d65f-44ca-8939-67778fe8ef5a-istio-token\") pod \"e2e-shared-a-w2-57f116-openshift-default-75f4fc499-znglh\" (UID: \"fd2b4daf-d65f-44ca-8939-67778fe8ef5a\") " pod="openshift-ingress/e2e-shared-a-w2-57f116-openshift-default-75f4fc499-znglh" Nov 04 03:01:14.649485 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:01:14.649035 4405 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fd2b4daf-d65f-44ca-8939-67778fe8ef5a-workload-socket\") pod \"e2e-shared-a-w2-57f116-openshift-default-75f4fc499-znglh\" (UID: \"fd2b4daf-d65f-44ca-8939-67778fe8ef5a\") " pod="openshift-ingress/e2e-shared-a-w2-57f116-openshift-default-75f4fc499-znglh" Nov 04 03:01:14.649485 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:01:14.649221 4405 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fd2b4daf-d65f-44ca-8939-67778fe8ef5a-workload-certs\") pod \"e2e-shared-a-w2-57f116-openshift-default-75f4fc499-znglh\" (UID: \"fd2b4daf-d65f-44ca-8939-67778fe8ef5a\") " pod="openshift-ingress/e2e-shared-a-w2-57f116-openshift-default-75f4fc499-znglh" Nov 04 03:01:14.649485 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:01:14.649235 4405 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fd2b4daf-d65f-44ca-8939-67778fe8ef5a-istiod-ca-cert\") pod \"e2e-shared-a-w2-57f116-openshift-default-75f4fc499-znglh\" (UID: \"fd2b4daf-d65f-44ca-8939-67778fe8ef5a\") " pod="openshift-ingress/e2e-shared-a-w2-57f116-openshift-default-75f4fc499-znglh" Nov 04 03:01:14.649485 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:01:14.649278 4405 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fd2b4daf-d65f-44ca-8939-67778fe8ef5a-istio-data\") pod \"e2e-shared-a-w2-57f116-openshift-default-75f4fc499-znglh\" (UID: \"fd2b4daf-d65f-44ca-8939-67778fe8ef5a\") " pod="openshift-ingress/e2e-shared-a-w2-57f116-openshift-default-75f4fc499-znglh" Nov 04 03:01:14.649485 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:01:14.649438 4405 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fd2b4daf-d65f-44ca-8939-67778fe8ef5a-credential-socket\") pod \"e2e-shared-a-w2-57f116-openshift-default-75f4fc499-znglh\" (UID: \"fd2b4daf-d65f-44ca-8939-67778fe8ef5a\") " pod="openshift-ingress/e2e-shared-a-w2-57f116-openshift-default-75f4fc499-znglh" Nov 04 03:01:14.659115 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:01:14.659076 4405 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-203c35-openshift-default-56497cdcc-w6c48"] Nov 04 03:01:14.660794 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:01:14.660766 4405 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fd2b4daf-d65f-44ca-8939-67778fe8ef5a-istio-podinfo\") pod \"e2e-shared-a-w2-57f116-openshift-default-75f4fc499-znglh\" (UID: \"fd2b4daf-d65f-44ca-8939-67778fe8ef5a\") " pod="openshift-ingress/e2e-shared-a-w2-57f116-openshift-default-75f4fc499-znglh" Nov 04 03:01:14.664428 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:01:14.664397 4405 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-203c35-openshift-default-56497cdcc-w6c48"] Nov 04 03:01:14.672532 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:01:14.672470 4405 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fd2b4daf-d65f-44ca-8939-67778fe8ef5a-istio-envoy\") pod \"e2e-shared-a-w2-57f116-openshift-default-75f4fc499-znglh\" (UID: \"fd2b4daf-d65f-44ca-8939-67778fe8ef5a\") " pod="openshift-ingress/e2e-shared-a-w2-57f116-openshift-default-75f4fc499-znglh" Nov 04 03:01:14.743193 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:01:14.743062 4405 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fd2b4daf-d65f-44ca-8939-67778fe8ef5a-istio-token\") pod \"e2e-shared-a-w2-57f116-openshift-default-75f4fc499-znglh\" (UID: \"fd2b4daf-d65f-44ca-8939-67778fe8ef5a\") " pod="openshift-ingress/e2e-shared-a-w2-57f116-openshift-default-75f4fc499-znglh" Nov 04 03:01:14.786907 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:01:14.784097 4405 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zrt6m\" (UniqueName: \"kubernetes.io/projected/fd2b4daf-d65f-44ca-8939-67778fe8ef5a-kube-api-access-zrt6m\") pod \"e2e-shared-a-w2-57f116-openshift-default-75f4fc499-znglh\" (UID: \"fd2b4daf-d65f-44ca-8939-67778fe8ef5a\") " pod="openshift-ingress/e2e-shared-a-w2-57f116-openshift-default-75f4fc499-znglh" Nov 04 03:01:14.881566 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:01:14.880976 4405 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w4-35df3e-7969f6bd6b-ptzqg"] Nov 04 03:01:14.884601 ip-10-0-134-125 kubenswrapper[4405]: W1104 03:01:14.884568 4405 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8d7f28e3_d62b_4813_81e3_364d4700983a.slice/crio-4241ec894de25db5e18ebb33d31b9b8a9aa5e7106dd987c7ad88d19dd91c78d5 WatchSource:0}: Error finding container 4241ec894de25db5e18ebb33d31b9b8a9aa5e7106dd987c7ad88d19dd91c78d5: Status 404 returned error can't find the container with id 4241ec894de25db5e18ebb33d31b9b8a9aa5e7106dd987c7ad88d19dd91c78d5 Nov 04 03:01:14.991727 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:01:14.979922 4405 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-35df3e-7969f6bd6b-ptzqg" event={"ID":"8d7f28e3-d62b-4813-81e3-364d4700983a","Type":"ContainerStarted","Data":"4241ec894de25db5e18ebb33d31b9b8a9aa5e7106dd987c7ad88d19dd91c78d5"} Nov 04 03:01:15.012708 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:01:15.012640 4405 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-203c35-openshift-default-56497cdcc-ljwqc" event={"ID":"fec507be-e742-4226-a6c4-6fb41e6f3479","Type":"ContainerDied","Data":"59cf3d394fb94451fe8c0a974b0dc71ad93e35cabc9c20b3428b27ee9c1a8612"} Nov 04 03:01:15.013053 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:01:15.013035 4405 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-203c35-openshift-default-56497cdcc-ljwqc" Nov 04 03:01:15.083469 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:01:15.083439 4405 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-57f116-openshift-default-75f4fc499-znglh" Nov 04 03:01:15.135165 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:01:15.135133 4405 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w4-35df3e-7d67b7f575-zbnm5"] Nov 04 03:01:15.159200 ip-10-0-134-125 kubenswrapper[4405]: W1104 03:01:15.158553 4405 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod317d0b2a_1ab0_4278_add3_cfaee93aa69c.slice/crio-25d7468a99e3df632cdc7709d38822dedbf196fd35b075e3c65ec29cc7233b2c WatchSource:0}: Error finding container 25d7468a99e3df632cdc7709d38822dedbf196fd35b075e3c65ec29cc7233b2c: Status 404 returned error can't find the container with id 25d7468a99e3df632cdc7709d38822dedbf196fd35b075e3c65ec29cc7233b2c Nov 04 03:01:15.191174 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:01:15.190281 4405 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-203c35-openshift-default-56497cdcc-ljwqc"] Nov 04 03:01:15.203199 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:01:15.202735 4405 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-203c35-openshift-default-56497cdcc-ljwqc"] Nov 04 03:01:15.552901 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:01:15.552868 4405 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-57f116-openshift-default-75f4fc499-znglh"] Nov 04 03:01:15.559590 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:01:15.559544 4405 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 04 03:01:15.559730 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:01:15.559658 4405 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 04 03:01:15.559730 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:01:15.559725 4405 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 04 03:01:15.706591 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:01:15.706546 4405 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="147e25a7-dda1-4170-9453-44896e44e2bc" path="/var/lib/kubelet/pods/147e25a7-dda1-4170-9453-44896e44e2bc/volumes" Nov 04 03:01:15.707114 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:01:15.707091 4405 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5ebc4fc9-8882-43c6-8713-1ee995a8ee00" path="/var/lib/kubelet/pods/5ebc4fc9-8882-43c6-8713-1ee995a8ee00/volumes" Nov 04 03:01:15.707526 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:01:15.707512 4405 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="89ddc69b-94de-4443-a62a-fa5d360aa368" path="/var/lib/kubelet/pods/89ddc69b-94de-4443-a62a-fa5d360aa368/volumes" Nov 04 03:01:15.708355 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:01:15.708335 4405 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fec507be-e742-4226-a6c4-6fb41e6f3479" path="/var/lib/kubelet/pods/fec507be-e742-4226-a6c4-6fb41e6f3479/volumes" Nov 04 03:01:16.029279 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:01:16.029176 4405 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-57f116-openshift-default-75f4fc499-znglh" event={"ID":"fd2b4daf-d65f-44ca-8939-67778fe8ef5a","Type":"ContainerStarted","Data":"7192f537f98158d936b5e94e6c021bd5b8a8ba642b9a06dea35cddcc897bf35f"} Nov 04 03:01:16.029279 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:01:16.029250 4405 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-57f116-openshift-default-75f4fc499-znglh" event={"ID":"fd2b4daf-d65f-44ca-8939-67778fe8ef5a","Type":"ContainerStarted","Data":"db1b50b8c67c645f7848e2e9bc5f8ed67e7a4841e0f82ea9ef1698e9a03c89ab"} Nov 04 03:01:16.031340 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:01:16.031310 4405 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-35df3e-7d67b7f575-zbnm5" event={"ID":"317d0b2a-1ab0-4278-add3-cfaee93aa69c","Type":"ContainerStarted","Data":"a35ede810afa95faec96e58c54ae75899966b69c6c91d76a4a40173fd5a4c274"} Nov 04 03:01:16.031491 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:01:16.031361 4405 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-35df3e-7d67b7f575-zbnm5" event={"ID":"317d0b2a-1ab0-4278-add3-cfaee93aa69c","Type":"ContainerStarted","Data":"25d7468a99e3df632cdc7709d38822dedbf196fd35b075e3c65ec29cc7233b2c"} Nov 04 03:01:16.031491 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:01:16.031390 4405 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-35df3e-7d67b7f575-zbnm5" Nov 04 03:01:16.033279 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:01:16.033249 4405 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-35df3e-7969f6bd6b-ptzqg" event={"ID":"8d7f28e3-d62b-4813-81e3-364d4700983a","Type":"ContainerStarted","Data":"a574bdaf53a239cbffbcb9c5f7d32def80a9d146817eeca5bccdc1ea96d3fd74"} Nov 04 03:01:16.033418 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:01:16.033387 4405 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-35df3e-7969f6bd6b-ptzqg" Nov 04 03:01:16.083629 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:01:16.083599 4405 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w2-57f116-openshift-default-75f4fc499-znglh" Nov 04 03:01:16.090225 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:01:16.090197 4405 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w2-57f116-openshift-default-75f4fc499-znglh" Nov 04 03:01:16.151080 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:01:16.151027 4405 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w2-57f116-openshift-default-75f4fc499-znglh" podStartSLOduration=2.151010872 podStartE2EDuration="2.151010872s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 03:01:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 03:01:16.148611386 +0000 UTC m=+1722.579311507" watchObservedRunningTime="2026-11-04 03:01:16.151010872 +0000 UTC m=+1722.581710991" Nov 04 03:01:16.389884 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:01:16.389819 4405 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-35df3e-7969f6bd6b-ptzqg" podStartSLOduration=2.389800804 podStartE2EDuration="2.389800804s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 03:01:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 03:01:16.270020878 +0000 UTC m=+1722.700720999" watchObservedRunningTime="2026-11-04 03:01:16.389800804 +0000 UTC m=+1722.820500925" Nov 04 03:01:16.390090 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:01:16.390063 4405 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-35df3e-7d67b7f575-zbnm5" podStartSLOduration=3.39005455 podStartE2EDuration="3.39005455s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 03:01:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 03:01:16.387000614 +0000 UTC m=+1722.817700735" watchObservedRunningTime="2026-11-04 03:01:16.39005455 +0000 UTC m=+1722.820754671" Nov 04 03:01:16.790744 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:01:16.790644 4405 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-bl5mg"] Nov 04 03:01:17.041398 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:01:17.041311 4405 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w2-57f116-openshift-default-75f4fc499-znglh" Nov 04 03:01:17.042498 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:01:17.042478 4405 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w2-57f116-openshift-default-75f4fc499-znglh" Nov 04 03:01:26.445251 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:01:26.445217 4405 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w5-4b2bf4-openshift-default-59785c9d99-b52gb"] Nov 04 03:01:26.449420 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:01:26.449397 4405 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w5-4b2bf4-openshift-default-59785c9d99-b52gb" Nov 04 03:01:26.502051 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:01:26.502025 4405 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w5-4b2bf4-openshift-default-dockercfg-8gcmj\"" Nov 04 03:01:26.555220 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:01:26.555179 4405 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w5-4b2bf4-openshift-default-59785c9d99-b52gb"] Nov 04 03:01:26.599757 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:01:26.591602 4405 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e21df403-3af7-4a07-8793-452782d57a3b-istio-envoy\") pod \"e2e-shared-b-w5-4b2bf4-openshift-default-59785c9d99-b52gb\" (UID: \"e21df403-3af7-4a07-8793-452782d57a3b\") " pod="openshift-ingress/e2e-shared-b-w5-4b2bf4-openshift-default-59785c9d99-b52gb" Nov 04 03:01:26.599757 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:01:26.591658 4405 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e21df403-3af7-4a07-8793-452782d57a3b-workload-certs\") pod \"e2e-shared-b-w5-4b2bf4-openshift-default-59785c9d99-b52gb\" (UID: \"e21df403-3af7-4a07-8793-452782d57a3b\") " pod="openshift-ingress/e2e-shared-b-w5-4b2bf4-openshift-default-59785c9d99-b52gb" Nov 04 03:01:26.599757 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:01:26.591717 4405 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e21df403-3af7-4a07-8793-452782d57a3b-istiod-ca-cert\") pod \"e2e-shared-b-w5-4b2bf4-openshift-default-59785c9d99-b52gb\" (UID: \"e21df403-3af7-4a07-8793-452782d57a3b\") " pod="openshift-ingress/e2e-shared-b-w5-4b2bf4-openshift-default-59785c9d99-b52gb" Nov 04 03:01:26.599757 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:01:26.591787 4405 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e21df403-3af7-4a07-8793-452782d57a3b-credential-socket\") pod \"e2e-shared-b-w5-4b2bf4-openshift-default-59785c9d99-b52gb\" (UID: \"e21df403-3af7-4a07-8793-452782d57a3b\") " pod="openshift-ingress/e2e-shared-b-w5-4b2bf4-openshift-default-59785c9d99-b52gb" Nov 04 03:01:26.599757 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:01:26.591813 4405 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e21df403-3af7-4a07-8793-452782d57a3b-istio-podinfo\") pod \"e2e-shared-b-w5-4b2bf4-openshift-default-59785c9d99-b52gb\" (UID: \"e21df403-3af7-4a07-8793-452782d57a3b\") " pod="openshift-ingress/e2e-shared-b-w5-4b2bf4-openshift-default-59785c9d99-b52gb" Nov 04 03:01:26.599757 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:01:26.591837 4405 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e21df403-3af7-4a07-8793-452782d57a3b-workload-socket\") pod \"e2e-shared-b-w5-4b2bf4-openshift-default-59785c9d99-b52gb\" (UID: \"e21df403-3af7-4a07-8793-452782d57a3b\") " pod="openshift-ingress/e2e-shared-b-w5-4b2bf4-openshift-default-59785c9d99-b52gb" Nov 04 03:01:26.599757 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:01:26.591884 4405 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gbcdk\" (UniqueName: \"kubernetes.io/projected/e21df403-3af7-4a07-8793-452782d57a3b-kube-api-access-gbcdk\") pod \"e2e-shared-b-w5-4b2bf4-openshift-default-59785c9d99-b52gb\" (UID: \"e21df403-3af7-4a07-8793-452782d57a3b\") " pod="openshift-ingress/e2e-shared-b-w5-4b2bf4-openshift-default-59785c9d99-b52gb" Nov 04 03:01:26.599757 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:01:26.591931 4405 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e21df403-3af7-4a07-8793-452782d57a3b-istio-token\") pod \"e2e-shared-b-w5-4b2bf4-openshift-default-59785c9d99-b52gb\" (UID: \"e21df403-3af7-4a07-8793-452782d57a3b\") " pod="openshift-ingress/e2e-shared-b-w5-4b2bf4-openshift-default-59785c9d99-b52gb" Nov 04 03:01:26.599757 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:01:26.592001 4405 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e21df403-3af7-4a07-8793-452782d57a3b-istio-data\") pod \"e2e-shared-b-w5-4b2bf4-openshift-default-59785c9d99-b52gb\" (UID: \"e21df403-3af7-4a07-8793-452782d57a3b\") " pod="openshift-ingress/e2e-shared-b-w5-4b2bf4-openshift-default-59785c9d99-b52gb" Nov 04 03:01:26.695005 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:01:26.693979 4405 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e21df403-3af7-4a07-8793-452782d57a3b-istio-envoy\") pod \"e2e-shared-b-w5-4b2bf4-openshift-default-59785c9d99-b52gb\" (UID: \"e21df403-3af7-4a07-8793-452782d57a3b\") " pod="openshift-ingress/e2e-shared-b-w5-4b2bf4-openshift-default-59785c9d99-b52gb" Nov 04 03:01:26.695005 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:01:26.694040 4405 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e21df403-3af7-4a07-8793-452782d57a3b-workload-certs\") pod \"e2e-shared-b-w5-4b2bf4-openshift-default-59785c9d99-b52gb\" (UID: \"e21df403-3af7-4a07-8793-452782d57a3b\") " pod="openshift-ingress/e2e-shared-b-w5-4b2bf4-openshift-default-59785c9d99-b52gb" Nov 04 03:01:26.695005 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:01:26.694078 4405 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e21df403-3af7-4a07-8793-452782d57a3b-istiod-ca-cert\") pod \"e2e-shared-b-w5-4b2bf4-openshift-default-59785c9d99-b52gb\" (UID: \"e21df403-3af7-4a07-8793-452782d57a3b\") " pod="openshift-ingress/e2e-shared-b-w5-4b2bf4-openshift-default-59785c9d99-b52gb" Nov 04 03:01:26.695005 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:01:26.694170 4405 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e21df403-3af7-4a07-8793-452782d57a3b-credential-socket\") pod \"e2e-shared-b-w5-4b2bf4-openshift-default-59785c9d99-b52gb\" (UID: \"e21df403-3af7-4a07-8793-452782d57a3b\") " pod="openshift-ingress/e2e-shared-b-w5-4b2bf4-openshift-default-59785c9d99-b52gb" Nov 04 03:01:26.695005 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:01:26.694198 4405 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e21df403-3af7-4a07-8793-452782d57a3b-istio-podinfo\") pod \"e2e-shared-b-w5-4b2bf4-openshift-default-59785c9d99-b52gb\" (UID: \"e21df403-3af7-4a07-8793-452782d57a3b\") " pod="openshift-ingress/e2e-shared-b-w5-4b2bf4-openshift-default-59785c9d99-b52gb" Nov 04 03:01:26.695005 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:01:26.694225 4405 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e21df403-3af7-4a07-8793-452782d57a3b-workload-socket\") pod \"e2e-shared-b-w5-4b2bf4-openshift-default-59785c9d99-b52gb\" (UID: \"e21df403-3af7-4a07-8793-452782d57a3b\") " pod="openshift-ingress/e2e-shared-b-w5-4b2bf4-openshift-default-59785c9d99-b52gb" Nov 04 03:01:26.695005 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:01:26.694266 4405 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gbcdk\" (UniqueName: \"kubernetes.io/projected/e21df403-3af7-4a07-8793-452782d57a3b-kube-api-access-gbcdk\") pod \"e2e-shared-b-w5-4b2bf4-openshift-default-59785c9d99-b52gb\" (UID: \"e21df403-3af7-4a07-8793-452782d57a3b\") " pod="openshift-ingress/e2e-shared-b-w5-4b2bf4-openshift-default-59785c9d99-b52gb" Nov 04 03:01:26.695005 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:01:26.694314 4405 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e21df403-3af7-4a07-8793-452782d57a3b-istio-token\") pod \"e2e-shared-b-w5-4b2bf4-openshift-default-59785c9d99-b52gb\" (UID: \"e21df403-3af7-4a07-8793-452782d57a3b\") " pod="openshift-ingress/e2e-shared-b-w5-4b2bf4-openshift-default-59785c9d99-b52gb" Nov 04 03:01:26.695005 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:01:26.694410 4405 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e21df403-3af7-4a07-8793-452782d57a3b-istio-data\") pod \"e2e-shared-b-w5-4b2bf4-openshift-default-59785c9d99-b52gb\" (UID: \"e21df403-3af7-4a07-8793-452782d57a3b\") " pod="openshift-ingress/e2e-shared-b-w5-4b2bf4-openshift-default-59785c9d99-b52gb" Nov 04 03:01:26.695563 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:01:26.695039 4405 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e21df403-3af7-4a07-8793-452782d57a3b-istio-data\") pod \"e2e-shared-b-w5-4b2bf4-openshift-default-59785c9d99-b52gb\" (UID: \"e21df403-3af7-4a07-8793-452782d57a3b\") " pod="openshift-ingress/e2e-shared-b-w5-4b2bf4-openshift-default-59785c9d99-b52gb" Nov 04 03:01:26.696591 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:01:26.696562 4405 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e21df403-3af7-4a07-8793-452782d57a3b-workload-certs\") pod \"e2e-shared-b-w5-4b2bf4-openshift-default-59785c9d99-b52gb\" (UID: \"e21df403-3af7-4a07-8793-452782d57a3b\") " pod="openshift-ingress/e2e-shared-b-w5-4b2bf4-openshift-default-59785c9d99-b52gb" Nov 04 03:01:26.697226 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:01:26.697202 4405 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e21df403-3af7-4a07-8793-452782d57a3b-credential-socket\") pod \"e2e-shared-b-w5-4b2bf4-openshift-default-59785c9d99-b52gb\" (UID: \"e21df403-3af7-4a07-8793-452782d57a3b\") " pod="openshift-ingress/e2e-shared-b-w5-4b2bf4-openshift-default-59785c9d99-b52gb" Nov 04 03:01:26.700217 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:01:26.698394 4405 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e21df403-3af7-4a07-8793-452782d57a3b-istiod-ca-cert\") pod \"e2e-shared-b-w5-4b2bf4-openshift-default-59785c9d99-b52gb\" (UID: \"e21df403-3af7-4a07-8793-452782d57a3b\") " pod="openshift-ingress/e2e-shared-b-w5-4b2bf4-openshift-default-59785c9d99-b52gb" Nov 04 03:01:26.700217 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:01:26.699121 4405 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e21df403-3af7-4a07-8793-452782d57a3b-workload-socket\") pod \"e2e-shared-b-w5-4b2bf4-openshift-default-59785c9d99-b52gb\" (UID: \"e21df403-3af7-4a07-8793-452782d57a3b\") " pod="openshift-ingress/e2e-shared-b-w5-4b2bf4-openshift-default-59785c9d99-b52gb" Nov 04 03:01:26.701069 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:01:26.701045 4405 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e21df403-3af7-4a07-8793-452782d57a3b-istio-envoy\") pod \"e2e-shared-b-w5-4b2bf4-openshift-default-59785c9d99-b52gb\" (UID: \"e21df403-3af7-4a07-8793-452782d57a3b\") " pod="openshift-ingress/e2e-shared-b-w5-4b2bf4-openshift-default-59785c9d99-b52gb" Nov 04 03:01:26.702433 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:01:26.702410 4405 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-57f116-599cbf5bbc-2fp4d"] Nov 04 03:01:26.703995 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:01:26.703948 4405 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e21df403-3af7-4a07-8793-452782d57a3b-istio-podinfo\") pod \"e2e-shared-b-w5-4b2bf4-openshift-default-59785c9d99-b52gb\" (UID: \"e21df403-3af7-4a07-8793-452782d57a3b\") " pod="openshift-ingress/e2e-shared-b-w5-4b2bf4-openshift-default-59785c9d99-b52gb" Nov 04 03:01:26.708627 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:01:26.708608 4405 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-57f116-599cbf5bbc-2fp4d" Nov 04 03:01:26.760031 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:01:26.759999 4405 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gbcdk\" (UniqueName: \"kubernetes.io/projected/e21df403-3af7-4a07-8793-452782d57a3b-kube-api-access-gbcdk\") pod \"e2e-shared-b-w5-4b2bf4-openshift-default-59785c9d99-b52gb\" (UID: \"e21df403-3af7-4a07-8793-452782d57a3b\") " pod="openshift-ingress/e2e-shared-b-w5-4b2bf4-openshift-default-59785c9d99-b52gb" Nov 04 03:01:26.760234 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:01:26.760205 4405 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e21df403-3af7-4a07-8793-452782d57a3b-istio-token\") pod \"e2e-shared-b-w5-4b2bf4-openshift-default-59785c9d99-b52gb\" (UID: \"e21df403-3af7-4a07-8793-452782d57a3b\") " pod="openshift-ingress/e2e-shared-b-w5-4b2bf4-openshift-default-59785c9d99-b52gb" Nov 04 03:01:26.761120 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:01:26.761090 4405 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w2-57f116-dockercfg-6n2r6\"" Nov 04 03:01:26.764919 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:01:26.764890 4405 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w2-57f116\"" Nov 04 03:01:26.765044 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:01:26.764983 4405 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w5-4b2bf4-openshift-default-59785c9d99-b52gb" Nov 04 03:01:26.766460 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:01:26.766436 4405 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-57f116-599cbf5bbc-2fp4d"] Nov 04 03:01:26.810042 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:01:26.810004 4405 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/449e5858-ec18-4e0e-ab17-520312115a05-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w2-57f116-599cbf5bbc-2fp4d\" (UID: \"449e5858-ec18-4e0e-ab17-520312115a05\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-57f116-599cbf5bbc-2fp4d" Nov 04 03:01:26.840705 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:01:26.827737 4405 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2n4l6\" (UniqueName: \"kubernetes.io/projected/449e5858-ec18-4e0e-ab17-520312115a05-kube-api-access-2n4l6\") pod \"payload-pre-processing-e2e-shared-a-w2-57f116-599cbf5bbc-2fp4d\" (UID: \"449e5858-ec18-4e0e-ab17-520312115a05\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-57f116-599cbf5bbc-2fp4d" Nov 04 03:01:26.882536 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:01:26.882506 4405 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-57f116-5b5dd4bd44-fzgp7"] Nov 04 03:01:26.899338 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:01:26.899307 4405 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-57f116-5b5dd4bd44-fzgp7" Nov 04 03:01:26.935391 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:01:26.934994 4405 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/449e5858-ec18-4e0e-ab17-520312115a05-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w2-57f116-599cbf5bbc-2fp4d\" (UID: \"449e5858-ec18-4e0e-ab17-520312115a05\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-57f116-599cbf5bbc-2fp4d" Nov 04 03:01:26.935391 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:01:26.935293 4405 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2n4l6\" (UniqueName: \"kubernetes.io/projected/449e5858-ec18-4e0e-ab17-520312115a05-kube-api-access-2n4l6\") pod \"payload-pre-processing-e2e-shared-a-w2-57f116-599cbf5bbc-2fp4d\" (UID: \"449e5858-ec18-4e0e-ab17-520312115a05\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-57f116-599cbf5bbc-2fp4d" Nov 04 03:01:26.936631 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:01:26.936036 4405 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/449e5858-ec18-4e0e-ab17-520312115a05-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w2-57f116-599cbf5bbc-2fp4d\" (UID: \"449e5858-ec18-4e0e-ab17-520312115a05\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-57f116-599cbf5bbc-2fp4d" Nov 04 03:01:26.943318 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:01:26.943221 4405 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-57f116-5b5dd4bd44-fzgp7"] Nov 04 03:01:27.047854 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:01:27.047826 4405 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-35df3e-7d67b7f575-zbnm5" Nov 04 03:01:27.049495 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:01:27.048472 4405 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6ff412ca-a023-4cde-8c75-6f9acbfb5121-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w2-57f116-5b5dd4bd44-fzgp7\" (UID: \"6ff412ca-a023-4cde-8c75-6f9acbfb5121\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-57f116-5b5dd4bd44-fzgp7" Nov 04 03:01:27.049495 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:01:27.048815 4405 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ft8vl\" (UniqueName: \"kubernetes.io/projected/6ff412ca-a023-4cde-8c75-6f9acbfb5121-kube-api-access-ft8vl\") pod \"payload-processing-e2e-shared-a-w2-57f116-5b5dd4bd44-fzgp7\" (UID: \"6ff412ca-a023-4cde-8c75-6f9acbfb5121\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-57f116-5b5dd4bd44-fzgp7" Nov 04 03:01:27.050347 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:01:27.050322 4405 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2n4l6\" (UniqueName: \"kubernetes.io/projected/449e5858-ec18-4e0e-ab17-520312115a05-kube-api-access-2n4l6\") pod \"payload-pre-processing-e2e-shared-a-w2-57f116-599cbf5bbc-2fp4d\" (UID: \"449e5858-ec18-4e0e-ab17-520312115a05\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-57f116-599cbf5bbc-2fp4d" Nov 04 03:01:27.057501 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:01:27.057461 4405 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-35df3e-7969f6bd6b-ptzqg" Nov 04 03:01:27.150165 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:01:27.150129 4405 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6ff412ca-a023-4cde-8c75-6f9acbfb5121-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w2-57f116-5b5dd4bd44-fzgp7\" (UID: \"6ff412ca-a023-4cde-8c75-6f9acbfb5121\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-57f116-5b5dd4bd44-fzgp7" Nov 04 03:01:27.152089 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:01:27.151377 4405 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6ff412ca-a023-4cde-8c75-6f9acbfb5121-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w2-57f116-5b5dd4bd44-fzgp7\" (UID: \"6ff412ca-a023-4cde-8c75-6f9acbfb5121\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-57f116-5b5dd4bd44-fzgp7" Nov 04 03:01:27.153441 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:01:27.153125 4405 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ft8vl\" (UniqueName: \"kubernetes.io/projected/6ff412ca-a023-4cde-8c75-6f9acbfb5121-kube-api-access-ft8vl\") pod \"payload-processing-e2e-shared-a-w2-57f116-5b5dd4bd44-fzgp7\" (UID: \"6ff412ca-a023-4cde-8c75-6f9acbfb5121\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-57f116-5b5dd4bd44-fzgp7" Nov 04 03:01:27.219427 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:01:27.219345 4405 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ft8vl\" (UniqueName: \"kubernetes.io/projected/6ff412ca-a023-4cde-8c75-6f9acbfb5121-kube-api-access-ft8vl\") pod \"payload-processing-e2e-shared-a-w2-57f116-5b5dd4bd44-fzgp7\" (UID: \"6ff412ca-a023-4cde-8c75-6f9acbfb5121\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-57f116-5b5dd4bd44-fzgp7" Nov 04 03:01:27.317593 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:01:27.317549 4405 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 04 03:01:27.317776 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:01:27.317645 4405 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 04 03:01:27.317776 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:01:27.317709 4405 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 04 03:01:27.322647 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:01:27.322610 4405 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-57f116-5b5dd4bd44-fzgp7" Nov 04 03:01:27.330943 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:01:27.328061 4405 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w5-4b2bf4-openshift-default-59785c9d99-b52gb"] Nov 04 03:01:27.349252 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:01:27.349214 4405 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-57f116-599cbf5bbc-2fp4d" Nov 04 03:01:27.759463 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:01:27.759433 4405 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-57f116-5b5dd4bd44-fzgp7"] Nov 04 03:01:27.760122 ip-10-0-134-125 kubenswrapper[4405]: W1104 03:01:27.760096 4405 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6ff412ca_a023_4cde_8c75_6f9acbfb5121.slice/crio-fc88e64e72fa679d6d555a244472de5e267213bc0c556004e9d5a4dd1711b3db WatchSource:0}: Error finding container fc88e64e72fa679d6d555a244472de5e267213bc0c556004e9d5a4dd1711b3db: Status 404 returned error can't find the container with id fc88e64e72fa679d6d555a244472de5e267213bc0c556004e9d5a4dd1711b3db Nov 04 03:01:27.809359 ip-10-0-134-125 kubenswrapper[4405]: W1104 03:01:27.809316 4405 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod449e5858_ec18_4e0e_ab17_520312115a05.slice/crio-4ddf520445cffebb967548194bb4b68d6abd53c0d70c61c6178cf4b21c2b0eec WatchSource:0}: Error finding container 4ddf520445cffebb967548194bb4b68d6abd53c0d70c61c6178cf4b21c2b0eec: Status 404 returned error can't find the container with id 4ddf520445cffebb967548194bb4b68d6abd53c0d70c61c6178cf4b21c2b0eec Nov 04 03:01:27.811017 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:01:27.810988 4405 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-57f116-599cbf5bbc-2fp4d"] Nov 04 03:01:28.168260 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:01:28.168221 4405 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-57f116-5b5dd4bd44-fzgp7" event={"ID":"6ff412ca-a023-4cde-8c75-6f9acbfb5121","Type":"ContainerStarted","Data":"592f36e31a6613b7397f2e567a3e9c6b74f02023978134ed29d82f766ff4120f"} Nov 04 03:01:28.168415 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:01:28.168289 4405 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-57f116-5b5dd4bd44-fzgp7" event={"ID":"6ff412ca-a023-4cde-8c75-6f9acbfb5121","Type":"ContainerStarted","Data":"fc88e64e72fa679d6d555a244472de5e267213bc0c556004e9d5a4dd1711b3db"} Nov 04 03:01:28.168415 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:01:28.168352 4405 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-57f116-5b5dd4bd44-fzgp7" Nov 04 03:01:28.169926 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:01:28.169896 4405 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w5-4b2bf4-openshift-default-59785c9d99-b52gb" event={"ID":"e21df403-3af7-4a07-8793-452782d57a3b","Type":"ContainerStarted","Data":"8e03d4d6805ad573b5dd4a47fc3ba1380e7ed057cac59b1280b3a354a3b3933e"} Nov 04 03:01:28.170077 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:01:28.169951 4405 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w5-4b2bf4-openshift-default-59785c9d99-b52gb" event={"ID":"e21df403-3af7-4a07-8793-452782d57a3b","Type":"ContainerStarted","Data":"a6fae144760f3a31dee73b22255cade2589b6bb3ad44f50bbfb0f2f88051abcd"} Nov 04 03:01:28.171465 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:01:28.171447 4405 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-57f116-599cbf5bbc-2fp4d" event={"ID":"449e5858-ec18-4e0e-ab17-520312115a05","Type":"ContainerStarted","Data":"9889ba24a9f3fba0e0546f8ead70aa4c6f50e01d14799a687618e48419675746"} Nov 04 03:01:28.171551 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:01:28.171487 4405 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-57f116-599cbf5bbc-2fp4d" event={"ID":"449e5858-ec18-4e0e-ab17-520312115a05","Type":"ContainerStarted","Data":"4ddf520445cffebb967548194bb4b68d6abd53c0d70c61c6178cf4b21c2b0eec"} Nov 04 03:01:28.171551 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:01:28.171521 4405 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-57f116-599cbf5bbc-2fp4d" Nov 04 03:01:28.285445 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:01:28.285394 4405 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-57f116-5b5dd4bd44-fzgp7" podStartSLOduration=2.285377119 podStartE2EDuration="2.285377119s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 03:01:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 03:01:28.282985679 +0000 UTC m=+1734.713685824" watchObservedRunningTime="2026-11-04 03:01:28.285377119 +0000 UTC m=+1734.716077240" Nov 04 03:01:28.419290 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:01:28.419161 4405 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w5-4b2bf4-openshift-default-59785c9d99-b52gb" podStartSLOduration=2.419134782 podStartE2EDuration="2.419134782s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 03:01:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 03:01:28.40424816 +0000 UTC m=+1734.834948283" watchObservedRunningTime="2026-11-04 03:01:28.419134782 +0000 UTC m=+1734.849834903" Nov 04 03:01:28.522494 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:01:28.522434 4405 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-57f116-599cbf5bbc-2fp4d" podStartSLOduration=2.522413862 podStartE2EDuration="2.522413862s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 03:01:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 03:01:28.518528112 +0000 UTC m=+1734.949228259" watchObservedRunningTime="2026-11-04 03:01:28.522413862 +0000 UTC m=+1734.953114003" Nov 04 03:01:28.765909 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:01:28.765829 4405 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w5-4b2bf4-openshift-default-59785c9d99-b52gb" Nov 04 03:01:28.786267 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:01:28.786236 4405 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w5-4b2bf4-openshift-default-59785c9d99-b52gb" Nov 04 03:01:29.178336 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:01:29.178303 4405 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w5-4b2bf4-openshift-default-59785c9d99-b52gb" Nov 04 03:01:29.179360 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:01:29.179337 4405 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w5-4b2bf4-openshift-default-59785c9d99-b52gb" Nov 04 03:01:34.565458 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:01:34.565416 4405 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w1-551e2b-79ddfccbf8-6rkbp"] Nov 04 03:01:34.648988 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:01:34.648936 4405 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w1-551e2b-79ddfccbf8-6rkbp"] Nov 04 03:01:34.649433 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:01:34.649270 4405 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-551e2b-79ddfccbf8-6rkbp" Nov 04 03:01:34.709166 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:01:34.709140 4405 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w1-551e2b-dockercfg-9gk7p\"" Nov 04 03:01:34.709592 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:01:34.709573 4405 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w1-551e2b\"" Nov 04 03:01:34.745006 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:01:34.744950 4405 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-55jkz\" (UniqueName: \"kubernetes.io/projected/85f4e570-eb96-4408-b9ed-12e96698c3fb-kube-api-access-55jkz\") pod \"payload-processing-e2e-shared-a-w1-551e2b-79ddfccbf8-6rkbp\" (UID: \"85f4e570-eb96-4408-b9ed-12e96698c3fb\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w1-551e2b-79ddfccbf8-6rkbp" Nov 04 03:01:34.745223 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:01:34.745201 4405 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/85f4e570-eb96-4408-b9ed-12e96698c3fb-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w1-551e2b-79ddfccbf8-6rkbp\" (UID: \"85f4e570-eb96-4408-b9ed-12e96698c3fb\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w1-551e2b-79ddfccbf8-6rkbp" Nov 04 03:01:34.854325 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:01:34.847351 4405 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/85f4e570-eb96-4408-b9ed-12e96698c3fb-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w1-551e2b-79ddfccbf8-6rkbp\" (UID: \"85f4e570-eb96-4408-b9ed-12e96698c3fb\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w1-551e2b-79ddfccbf8-6rkbp" Nov 04 03:01:34.854325 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:01:34.847609 4405 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-55jkz\" (UniqueName: \"kubernetes.io/projected/85f4e570-eb96-4408-b9ed-12e96698c3fb-kube-api-access-55jkz\") pod \"payload-processing-e2e-shared-a-w1-551e2b-79ddfccbf8-6rkbp\" (UID: \"85f4e570-eb96-4408-b9ed-12e96698c3fb\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w1-551e2b-79ddfccbf8-6rkbp" Nov 04 03:01:34.854325 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:01:34.849040 4405 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/85f4e570-eb96-4408-b9ed-12e96698c3fb-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w1-551e2b-79ddfccbf8-6rkbp\" (UID: \"85f4e570-eb96-4408-b9ed-12e96698c3fb\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w1-551e2b-79ddfccbf8-6rkbp" Nov 04 03:01:34.909002 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:01:34.908972 4405 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-55jkz\" (UniqueName: \"kubernetes.io/projected/85f4e570-eb96-4408-b9ed-12e96698c3fb-kube-api-access-55jkz\") pod \"payload-processing-e2e-shared-a-w1-551e2b-79ddfccbf8-6rkbp\" (UID: \"85f4e570-eb96-4408-b9ed-12e96698c3fb\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w1-551e2b-79ddfccbf8-6rkbp" Nov 04 03:01:34.985383 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:01:34.985343 4405 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-551e2b-79ddfccbf8-6rkbp" Nov 04 03:01:35.375103 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:01:35.375077 4405 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w1-551e2b-79ddfccbf8-6rkbp"] Nov 04 03:01:35.376049 ip-10-0-134-125 kubenswrapper[4405]: W1104 03:01:35.376026 4405 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod85f4e570_eb96_4408_b9ed_12e96698c3fb.slice/crio-696ecf2d574b10dd9f4e3cdf18b50deb0d445f676af0a11faf457f0fb719477d WatchSource:0}: Error finding container 696ecf2d574b10dd9f4e3cdf18b50deb0d445f676af0a11faf457f0fb719477d: Status 404 returned error can't find the container with id 696ecf2d574b10dd9f4e3cdf18b50deb0d445f676af0a11faf457f0fb719477d Nov 04 03:01:36.241636 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:01:36.241596 4405 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-551e2b-79ddfccbf8-6rkbp" event={"ID":"85f4e570-eb96-4408-b9ed-12e96698c3fb","Type":"ContainerStarted","Data":"9c3795342fd02f046adf5c19a2eff0e2539afcb640e9a67f31b2a0ebe0fa5dfc"} Nov 04 03:01:36.242095 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:01:36.241660 4405 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-551e2b-79ddfccbf8-6rkbp" event={"ID":"85f4e570-eb96-4408-b9ed-12e96698c3fb","Type":"ContainerStarted","Data":"696ecf2d574b10dd9f4e3cdf18b50deb0d445f676af0a11faf457f0fb719477d"} Nov 04 03:01:36.242095 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:01:36.241731 4405 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-551e2b-79ddfccbf8-6rkbp" Nov 04 03:01:36.407570 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:01:36.407512 4405 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-551e2b-79ddfccbf8-6rkbp" podStartSLOduration=2.407496261 podStartE2EDuration="2.407496261s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 03:01:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 03:01:36.356382288 +0000 UTC m=+1742.787082401" watchObservedRunningTime="2026-11-04 03:01:36.407496261 +0000 UTC m=+1742.838196381" Nov 04 03:01:36.884703 ip-10-0-134-125 kubenswrapper[4405]: E1104 03:01:36.884635 4405 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5f2vb" containerName="extractor" Nov 04 03:01:39.183981 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:01:39.183949 4405 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-57f116-599cbf5bbc-2fp4d" Nov 04 03:01:39.185473 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:01:39.185430 4405 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-57f116-5b5dd4bd44-fzgp7" Nov 04 03:01:47.325751 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:01:47.324881 4405 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-551e2b-79ddfccbf8-6rkbp" Nov 04 03:01:47.608695 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:01:47.585767 4405 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-ebb7fd/distinct-2-w6-ebb7fd-kserve-6f68c5db86-zrpjs" podUID="b7b786e0-9c94-4126-9949-cd354ebbfb2a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.65:8000/ready\": context deadline exceeded" Nov 04 03:01:47.720194 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:01:47.624476 4405 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-a0bcbc/premium-w2-a0bcbc-kserve-78dfd6d78c-lwth2" podUID="33b620c1-389c-48b4-b277-90252018c4b6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 03:01:47.985748 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:01:47.985125 4405 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-e516ff/distinct-w1-e516ff-kserve-f4474cb9c-glpzw" podUID="8842ad09-22ff-4492-ba5d-d4cacf4eac75" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 03:01:47.985748 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:01:47.985485 4405 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76597d6855-brntt container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.82:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 04 03:01:47.985748 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:01:47.985528 4405 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-brntt" podUID="423b62dd-792f-442c-b1ac-3248072565e8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.82:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 04 03:01:48.024690 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:01:48.023833 4405 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-cdcf67/distinct-w3-cdcf67-kserve-667cc7f7d7-5m2lq" podUID="52ed22cc-96e4-4ef0-b34d-ed58c89a9183" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 03:01:48.053857 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:01:48.053799 4405 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-ljhhw" podUID="f66b1861-d87b-4d57-ab27-f1a24544a534" containerName="main" probeResult="failure" output="Get \"https://10.134.0.26:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 03:01:48.115739 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:01:48.098162 4405 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-a0bcbc/distinct-w2-a0bcbc-kserve-596f94c7cd-szl4g" podUID="bac32934-8a57-4919-b435-13f367cbea3e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/health\": context deadline exceeded" Nov 04 03:01:48.118120 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:01:48.118086 4405 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76597d6855-7c95j container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.81:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 04 03:01:48.118392 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:01:48.118360 4405 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-7c95j" podUID="76df1601-21fb-40ab-a5ee-433f367453c7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.81:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 04 03:01:48.237799 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:01:48.237688 4405 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-ebb7fd/distinct-w6-ebb7fd-kserve-55c47d8865-7z9pz" podUID="ccb8e181-5610-46da-bb8b-e0b07fa02ff1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 03:01:48.745814 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:01:48.745763 4405 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-a0bcbc/premium-w2-a0bcbc-kserve-78dfd6d78c-lwth2" podUID="33b620c1-389c-48b4-b277-90252018c4b6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 03:01:48.747028 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:01:48.747000 4405 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-a0bcbc/premium-w2-a0bcbc-kserve-78dfd6d78c-lwth2" Nov 04 03:01:48.863028 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:01:48.862995 4405 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-a0bcbc/premium-w2-a0bcbc-kserve-78dfd6d78c-lwth2" Nov 04 03:01:53.257804 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:01:53.257759 4405 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-e516ff-openshift-default-6f4b957485-7pg2l"] Nov 04 03:01:53.258338 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:01:53.258030 4405 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-e516ff-openshift-default-6f4b957485-7pg2l" podUID="8fc7b14e-09f9-4630-bfc6-b98eab67753f" containerName="istio-proxy" containerID="cri-o://5ff33d8a8d5b1b7f130a605e52248650a506b2ee611d906f466d983b36b612ac" gracePeriod=30 Nov 04 03:01:53.312018 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:01:53.311988 4405 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-e516ff-openshift-default-6f4b957485-bjmn9"] Nov 04 03:01:53.312234 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:01:53.312213 4405 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-e516ff-openshift-default-6f4b957485-bjmn9" podUID="3fd4ae35-84c1-45c1-a49c-14f2a8ed1c82" containerName="istio-proxy" containerID="cri-o://1c64e020a17f41c8ad7c11a821b583d8becae3a4e813baa156a5e6f237a17ade" gracePeriod=30 Nov 04 03:01:53.315883 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:01:53.315798 4405 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-e516ff-openshift-default-6f4b957485-w2plx"] Nov 04 03:01:53.316090 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:01:53.316064 4405 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-e516ff-openshift-default-6f4b957485-w2plx" podUID="a525d6f7-79b5-4409-8137-319328626c10" containerName="istio-proxy" containerID="cri-o://48cd8b87ccdd12489dc788842979decee8e4e3f709d453c6c0c8b9d98ec655f4" gracePeriod=30 Nov 04 03:01:56.049411 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:01:56.049368 4405 patch_prober.go:29] interesting pod/e2e-worker-w1-e516ff-openshift-default-6f4b957485-7pg2l container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 04 03:01:56.049918 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:01:56.049444 4405 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-e516ff-openshift-default-6f4b957485-7pg2l" podUID="8fc7b14e-09f9-4630-bfc6-b98eab67753f" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 04 03:01:58.291871 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:01:58.291836 4405 patch_prober.go:29] interesting pod/e2e-worker-w1-e516ff-openshift-default-6f4b957485-w2plx container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 04 03:01:58.292172 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:01:58.291901 4405 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-e516ff-openshift-default-6f4b957485-w2plx" podUID="a525d6f7-79b5-4409-8137-319328626c10" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 04 03:01:58.685579 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:01:58.685539 4405 generic.go:356] "Generic (PLEG): container finished" podID="8fc7b14e-09f9-4630-bfc6-b98eab67753f" containerID="5ff33d8a8d5b1b7f130a605e52248650a506b2ee611d906f466d983b36b612ac" exitCode=0 Nov 04 03:01:58.685708 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:01:58.685597 4405 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-e516ff-openshift-default-6f4b957485-7pg2l" event={"ID":"8fc7b14e-09f9-4630-bfc6-b98eab67753f","Type":"ContainerDied","Data":"5ff33d8a8d5b1b7f130a605e52248650a506b2ee611d906f466d983b36b612ac"} Nov 04 03:01:58.687695 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:01:58.687659 4405 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-e516ff-openshift-default-6f4b957485-bjmn9" Nov 04 03:01:58.689211 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:01:58.689190 4405 generic.go:356] "Generic (PLEG): container finished" podID="3fd4ae35-84c1-45c1-a49c-14f2a8ed1c82" containerID="1c64e020a17f41c8ad7c11a821b583d8becae3a4e813baa156a5e6f237a17ade" exitCode=0 Nov 04 03:01:58.689285 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:01:58.689271 4405 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-e516ff-openshift-default-6f4b957485-bjmn9" event={"ID":"3fd4ae35-84c1-45c1-a49c-14f2a8ed1c82","Type":"ContainerDied","Data":"1c64e020a17f41c8ad7c11a821b583d8becae3a4e813baa156a5e6f237a17ade"} Nov 04 03:01:58.689325 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:01:58.689309 4405 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-e516ff-openshift-default-6f4b957485-bjmn9" event={"ID":"3fd4ae35-84c1-45c1-a49c-14f2a8ed1c82","Type":"ContainerDied","Data":"925d73507f5b928b84a3b79b8b6216df3c1d6a231ab744b7f472a3b626cd6d25"} Nov 04 03:01:58.689362 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:01:58.689329 4405 scope.go:122] "RemoveContainer" containerID="1c64e020a17f41c8ad7c11a821b583d8becae3a4e813baa156a5e6f237a17ade" Nov 04 03:01:58.692835 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:01:58.692806 4405 generic.go:356] "Generic (PLEG): container finished" podID="a525d6f7-79b5-4409-8137-319328626c10" containerID="48cd8b87ccdd12489dc788842979decee8e4e3f709d453c6c0c8b9d98ec655f4" exitCode=0 Nov 04 03:01:58.692924 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:01:58.692881 4405 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-e516ff-openshift-default-6f4b957485-w2plx" event={"ID":"a525d6f7-79b5-4409-8137-319328626c10","Type":"ContainerDied","Data":"48cd8b87ccdd12489dc788842979decee8e4e3f709d453c6c0c8b9d98ec655f4"} Nov 04 03:01:58.698694 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:01:58.697861 4405 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-e516ff-openshift-default-6f4b957485-7pg2l" Nov 04 03:01:58.702286 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:01:58.702268 4405 scope.go:122] "RemoveContainer" containerID="dc4e5d146d6b9dd6d75a3cb505756dee5d5552f16791b4deb553fc92ab758cc8" Nov 04 03:01:58.715384 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:01:58.715358 4405 scope.go:122] "RemoveContainer" containerID="1c64e020a17f41c8ad7c11a821b583d8becae3a4e813baa156a5e6f237a17ade" Nov 04 03:01:58.715786 ip-10-0-134-125 kubenswrapper[4405]: E1104 03:01:58.715761 4405 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1c64e020a17f41c8ad7c11a821b583d8becae3a4e813baa156a5e6f237a17ade\": container with ID starting with 1c64e020a17f41c8ad7c11a821b583d8becae3a4e813baa156a5e6f237a17ade not found: ID does not exist" containerID="1c64e020a17f41c8ad7c11a821b583d8becae3a4e813baa156a5e6f237a17ade" Nov 04 03:01:58.715895 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:01:58.715799 4405 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1c64e020a17f41c8ad7c11a821b583d8becae3a4e813baa156a5e6f237a17ade"} err="failed to get container status \"1c64e020a17f41c8ad7c11a821b583d8becae3a4e813baa156a5e6f237a17ade\": rpc error: code = NotFound desc = could not find container \"1c64e020a17f41c8ad7c11a821b583d8becae3a4e813baa156a5e6f237a17ade\": container with ID starting with 1c64e020a17f41c8ad7c11a821b583d8becae3a4e813baa156a5e6f237a17ade not found: ID does not exist" Nov 04 03:01:58.715895 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:01:58.715826 4405 scope.go:122] "RemoveContainer" containerID="dc4e5d146d6b9dd6d75a3cb505756dee5d5552f16791b4deb553fc92ab758cc8" Nov 04 03:01:58.716131 ip-10-0-134-125 kubenswrapper[4405]: E1104 03:01:58.716100 4405 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dc4e5d146d6b9dd6d75a3cb505756dee5d5552f16791b4deb553fc92ab758cc8\": container with ID starting with dc4e5d146d6b9dd6d75a3cb505756dee5d5552f16791b4deb553fc92ab758cc8 not found: ID does not exist" containerID="dc4e5d146d6b9dd6d75a3cb505756dee5d5552f16791b4deb553fc92ab758cc8" Nov 04 03:01:58.716224 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:01:58.716131 4405 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dc4e5d146d6b9dd6d75a3cb505756dee5d5552f16791b4deb553fc92ab758cc8"} err="failed to get container status \"dc4e5d146d6b9dd6d75a3cb505756dee5d5552f16791b4deb553fc92ab758cc8\": rpc error: code = NotFound desc = could not find container \"dc4e5d146d6b9dd6d75a3cb505756dee5d5552f16791b4deb553fc92ab758cc8\": container with ID starting with dc4e5d146d6b9dd6d75a3cb505756dee5d5552f16791b4deb553fc92ab758cc8 not found: ID does not exist" Nov 04 03:01:58.742482 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:01:58.742461 4405 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-e516ff-openshift-default-6f4b957485-w2plx" Nov 04 03:01:58.829657 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:01:58.829565 4405 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a525d6f7-79b5-4409-8137-319328626c10-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a525d6f7-79b5-4409-8137-319328626c10-workload-certs\") pod \"a525d6f7-79b5-4409-8137-319328626c10\" (UID: \"a525d6f7-79b5-4409-8137-319328626c10\") " Nov 04 03:01:58.829657 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:01:58.829607 4405 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a525d6f7-79b5-4409-8137-319328626c10-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a525d6f7-79b5-4409-8137-319328626c10-credential-socket\") pod \"a525d6f7-79b5-4409-8137-319328626c10\" (UID: \"a525d6f7-79b5-4409-8137-319328626c10\") " Nov 04 03:01:58.829935 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:01:58.829699 4405 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a525d6f7-79b5-4409-8137-319328626c10-istio-token\" (UniqueName: \"kubernetes.io/projected/a525d6f7-79b5-4409-8137-319328626c10-istio-token\") pod \"a525d6f7-79b5-4409-8137-319328626c10\" (UID: \"a525d6f7-79b5-4409-8137-319328626c10\") " Nov 04 03:01:58.829935 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:01:58.829739 4405 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3fd4ae35-84c1-45c1-a49c-14f2a8ed1c82-istio-token\" (UniqueName: \"kubernetes.io/projected/3fd4ae35-84c1-45c1-a49c-14f2a8ed1c82-istio-token\") pod \"3fd4ae35-84c1-45c1-a49c-14f2a8ed1c82\" (UID: \"3fd4ae35-84c1-45c1-a49c-14f2a8ed1c82\") " Nov 04 03:01:58.829935 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:01:58.829763 4405 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a525d6f7-79b5-4409-8137-319328626c10-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a525d6f7-79b5-4409-8137-319328626c10-istio-envoy\") pod \"a525d6f7-79b5-4409-8137-319328626c10\" (UID: \"a525d6f7-79b5-4409-8137-319328626c10\") " Nov 04 03:01:58.829935 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:01:58.829797 4405 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3fd4ae35-84c1-45c1-a49c-14f2a8ed1c82-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3fd4ae35-84c1-45c1-a49c-14f2a8ed1c82-workload-certs\") pod \"3fd4ae35-84c1-45c1-a49c-14f2a8ed1c82\" (UID: \"3fd4ae35-84c1-45c1-a49c-14f2a8ed1c82\") " Nov 04 03:01:58.829935 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:01:58.829823 4405 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3fd4ae35-84c1-45c1-a49c-14f2a8ed1c82-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3fd4ae35-84c1-45c1-a49c-14f2a8ed1c82-istio-envoy\") pod \"3fd4ae35-84c1-45c1-a49c-14f2a8ed1c82\" (UID: \"3fd4ae35-84c1-45c1-a49c-14f2a8ed1c82\") " Nov 04 03:01:58.829935 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:01:58.829891 4405 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/a525d6f7-79b5-4409-8137-319328626c10-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a525d6f7-79b5-4409-8137-319328626c10-istio-podinfo\") pod \"a525d6f7-79b5-4409-8137-319328626c10\" (UID: \"a525d6f7-79b5-4409-8137-319328626c10\") " Nov 04 03:01:58.830245 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:01:58.829900 4405 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a525d6f7-79b5-4409-8137-319328626c10-workload-certs" pod "a525d6f7-79b5-4409-8137-319328626c10" (UID: "a525d6f7-79b5-4409-8137-319328626c10"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 03:01:58.830245 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:01:58.829946 4405 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3fd4ae35-84c1-45c1-a49c-14f2a8ed1c82-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3fd4ae35-84c1-45c1-a49c-14f2a8ed1c82-istiod-ca-cert\") pod \"3fd4ae35-84c1-45c1-a49c-14f2a8ed1c82\" (UID: \"3fd4ae35-84c1-45c1-a49c-14f2a8ed1c82\") " Nov 04 03:01:58.830245 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:01:58.830000 4405 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a525d6f7-79b5-4409-8137-319328626c10-credential-socket" pod "a525d6f7-79b5-4409-8137-319328626c10" (UID: "a525d6f7-79b5-4409-8137-319328626c10"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 03:01:58.830245 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:01:58.830047 4405 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a525d6f7-79b5-4409-8137-319328626c10-kube-api-access-bnxc2\" (UniqueName: \"kubernetes.io/projected/a525d6f7-79b5-4409-8137-319328626c10-kube-api-access-bnxc2\") pod \"a525d6f7-79b5-4409-8137-319328626c10\" (UID: \"a525d6f7-79b5-4409-8137-319328626c10\") " Nov 04 03:01:58.830245 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:01:58.830099 4405 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/3fd4ae35-84c1-45c1-a49c-14f2a8ed1c82-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3fd4ae35-84c1-45c1-a49c-14f2a8ed1c82-istio-podinfo\") pod \"3fd4ae35-84c1-45c1-a49c-14f2a8ed1c82\" (UID: \"3fd4ae35-84c1-45c1-a49c-14f2a8ed1c82\") " Nov 04 03:01:58.830245 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:01:58.830113 4405 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3fd4ae35-84c1-45c1-a49c-14f2a8ed1c82-workload-certs" pod "3fd4ae35-84c1-45c1-a49c-14f2a8ed1c82" (UID: "3fd4ae35-84c1-45c1-a49c-14f2a8ed1c82"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 03:01:58.830245 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:01:58.830234 4405 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8fc7b14e-09f9-4630-bfc6-b98eab67753f-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8fc7b14e-09f9-4630-bfc6-b98eab67753f-workload-certs\") pod \"8fc7b14e-09f9-4630-bfc6-b98eab67753f\" (UID: \"8fc7b14e-09f9-4630-bfc6-b98eab67753f\") " Nov 04 03:01:58.831034 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:01:58.830270 4405 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3fd4ae35-84c1-45c1-a49c-14f2a8ed1c82-istiod-ca-cert" pod "3fd4ae35-84c1-45c1-a49c-14f2a8ed1c82" (UID: "3fd4ae35-84c1-45c1-a49c-14f2a8ed1c82"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 03:01:58.831034 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:01:58.830283 4405 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8fc7b14e-09f9-4630-bfc6-b98eab67753f-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8fc7b14e-09f9-4630-bfc6-b98eab67753f-workload-socket\") pod \"8fc7b14e-09f9-4630-bfc6-b98eab67753f\" (UID: \"8fc7b14e-09f9-4630-bfc6-b98eab67753f\") " Nov 04 03:01:58.831034 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:01:58.830315 4405 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a525d6f7-79b5-4409-8137-319328626c10-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a525d6f7-79b5-4409-8137-319328626c10-istio-data\") pod \"a525d6f7-79b5-4409-8137-319328626c10\" (UID: \"a525d6f7-79b5-4409-8137-319328626c10\") " Nov 04 03:01:58.831034 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:01:58.830358 4405 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8fc7b14e-09f9-4630-bfc6-b98eab67753f-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8fc7b14e-09f9-4630-bfc6-b98eab67753f-credential-socket\") pod \"8fc7b14e-09f9-4630-bfc6-b98eab67753f\" (UID: \"8fc7b14e-09f9-4630-bfc6-b98eab67753f\") " Nov 04 03:01:58.831034 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:01:58.830424 4405 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a525d6f7-79b5-4409-8137-319328626c10-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a525d6f7-79b5-4409-8137-319328626c10-istiod-ca-cert\") pod \"a525d6f7-79b5-4409-8137-319328626c10\" (UID: \"a525d6f7-79b5-4409-8137-319328626c10\") " Nov 04 03:01:58.831034 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:01:58.830457 4405 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3fd4ae35-84c1-45c1-a49c-14f2a8ed1c82-kube-api-access-v5n9f\" (UniqueName: \"kubernetes.io/projected/3fd4ae35-84c1-45c1-a49c-14f2a8ed1c82-kube-api-access-v5n9f\") pod \"3fd4ae35-84c1-45c1-a49c-14f2a8ed1c82\" (UID: \"3fd4ae35-84c1-45c1-a49c-14f2a8ed1c82\") " Nov 04 03:01:58.831034 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:01:58.830564 4405 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8fc7b14e-09f9-4630-bfc6-b98eab67753f-workload-certs" pod "8fc7b14e-09f9-4630-bfc6-b98eab67753f" (UID: "8fc7b14e-09f9-4630-bfc6-b98eab67753f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 03:01:58.831034 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:01:58.830531 4405 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8fc7b14e-09f9-4630-bfc6-b98eab67753f-kube-api-access-qh6vr\" (UniqueName: \"kubernetes.io/projected/8fc7b14e-09f9-4630-bfc6-b98eab67753f-kube-api-access-qh6vr\") pod \"8fc7b14e-09f9-4630-bfc6-b98eab67753f\" (UID: \"8fc7b14e-09f9-4630-bfc6-b98eab67753f\") " Nov 04 03:01:58.831034 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:01:58.830864 4405 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8fc7b14e-09f9-4630-bfc6-b98eab67753f-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8fc7b14e-09f9-4630-bfc6-b98eab67753f-istio-data\") pod \"8fc7b14e-09f9-4630-bfc6-b98eab67753f\" (UID: \"8fc7b14e-09f9-4630-bfc6-b98eab67753f\") " Nov 04 03:01:58.831034 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:01:58.830952 4405 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/8fc7b14e-09f9-4630-bfc6-b98eab67753f-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8fc7b14e-09f9-4630-bfc6-b98eab67753f-istio-podinfo\") pod \"8fc7b14e-09f9-4630-bfc6-b98eab67753f\" (UID: \"8fc7b14e-09f9-4630-bfc6-b98eab67753f\") " Nov 04 03:01:58.831034 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:01:58.831037 4405 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8fc7b14e-09f9-4630-bfc6-b98eab67753f-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8fc7b14e-09f9-4630-bfc6-b98eab67753f-istiod-ca-cert\") pod \"8fc7b14e-09f9-4630-bfc6-b98eab67753f\" (UID: \"8fc7b14e-09f9-4630-bfc6-b98eab67753f\") " Nov 04 03:01:58.831634 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:01:58.831050 4405 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8fc7b14e-09f9-4630-bfc6-b98eab67753f-credential-socket" pod "8fc7b14e-09f9-4630-bfc6-b98eab67753f" (UID: "8fc7b14e-09f9-4630-bfc6-b98eab67753f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 03:01:58.831634 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:01:58.831069 4405 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8fc7b14e-09f9-4630-bfc6-b98eab67753f-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8fc7b14e-09f9-4630-bfc6-b98eab67753f-istio-envoy\") pod \"8fc7b14e-09f9-4630-bfc6-b98eab67753f\" (UID: \"8fc7b14e-09f9-4630-bfc6-b98eab67753f\") " Nov 04 03:01:58.831634 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:01:58.831099 4405 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8fc7b14e-09f9-4630-bfc6-b98eab67753f-istio-token\" (UniqueName: \"kubernetes.io/projected/8fc7b14e-09f9-4630-bfc6-b98eab67753f-istio-token\") pod \"8fc7b14e-09f9-4630-bfc6-b98eab67753f\" (UID: \"8fc7b14e-09f9-4630-bfc6-b98eab67753f\") " Nov 04 03:01:58.831634 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:01:58.831189 4405 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3fd4ae35-84c1-45c1-a49c-14f2a8ed1c82-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3fd4ae35-84c1-45c1-a49c-14f2a8ed1c82-credential-socket\") pod \"3fd4ae35-84c1-45c1-a49c-14f2a8ed1c82\" (UID: \"3fd4ae35-84c1-45c1-a49c-14f2a8ed1c82\") " Nov 04 03:01:58.831634 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:01:58.831252 4405 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3fd4ae35-84c1-45c1-a49c-14f2a8ed1c82-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3fd4ae35-84c1-45c1-a49c-14f2a8ed1c82-workload-socket\") pod \"3fd4ae35-84c1-45c1-a49c-14f2a8ed1c82\" (UID: \"3fd4ae35-84c1-45c1-a49c-14f2a8ed1c82\") " Nov 04 03:01:58.831634 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:01:58.831275 4405 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8fc7b14e-09f9-4630-bfc6-b98eab67753f-workload-socket" pod "8fc7b14e-09f9-4630-bfc6-b98eab67753f" (UID: "8fc7b14e-09f9-4630-bfc6-b98eab67753f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 03:01:58.831634 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:01:58.831284 4405 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3fd4ae35-84c1-45c1-a49c-14f2a8ed1c82-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3fd4ae35-84c1-45c1-a49c-14f2a8ed1c82-istio-data\") pod \"3fd4ae35-84c1-45c1-a49c-14f2a8ed1c82\" (UID: \"3fd4ae35-84c1-45c1-a49c-14f2a8ed1c82\") " Nov 04 03:01:58.831634 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:01:58.831399 4405 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a525d6f7-79b5-4409-8137-319328626c10-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a525d6f7-79b5-4409-8137-319328626c10-workload-socket\") pod \"a525d6f7-79b5-4409-8137-319328626c10\" (UID: \"a525d6f7-79b5-4409-8137-319328626c10\") " Nov 04 03:01:58.832545 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:01:58.832520 4405 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8fc7b14e-09f9-4630-bfc6-b98eab67753f-credential-socket\") on node \"ip-10-0-134-125.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:01:58.832703 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:01:58.832547 4405 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a525d6f7-79b5-4409-8137-319328626c10-workload-certs\") on node \"ip-10-0-134-125.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:01:58.832703 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:01:58.832563 4405 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a525d6f7-79b5-4409-8137-319328626c10-credential-socket\") on node \"ip-10-0-134-125.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:01:58.832703 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:01:58.832581 4405 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3fd4ae35-84c1-45c1-a49c-14f2a8ed1c82-workload-certs\") on node \"ip-10-0-134-125.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:01:58.832703 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:01:58.832596 4405 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3fd4ae35-84c1-45c1-a49c-14f2a8ed1c82-istiod-ca-cert\") on node \"ip-10-0-134-125.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:01:58.832703 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:01:58.832610 4405 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8fc7b14e-09f9-4630-bfc6-b98eab67753f-workload-certs\") on node \"ip-10-0-134-125.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:01:58.832703 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:01:58.832625 4405 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8fc7b14e-09f9-4630-bfc6-b98eab67753f-workload-socket\") on node \"ip-10-0-134-125.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:01:58.833336 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:01:58.831279 4405 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a525d6f7-79b5-4409-8137-319328626c10-istio-data" pod "a525d6f7-79b5-4409-8137-319328626c10" (UID: "a525d6f7-79b5-4409-8137-319328626c10"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 03:01:58.833336 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:01:58.831865 4405 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8fc7b14e-09f9-4630-bfc6-b98eab67753f-istiod-ca-cert" pod "8fc7b14e-09f9-4630-bfc6-b98eab67753f" (UID: "8fc7b14e-09f9-4630-bfc6-b98eab67753f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 03:01:58.833336 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:01:58.832070 4405 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a525d6f7-79b5-4409-8137-319328626c10-istiod-ca-cert" pod "a525d6f7-79b5-4409-8137-319328626c10" (UID: "a525d6f7-79b5-4409-8137-319328626c10"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 03:01:58.833336 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:01:58.832107 4405 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8fc7b14e-09f9-4630-bfc6-b98eab67753f-istio-data" pod "8fc7b14e-09f9-4630-bfc6-b98eab67753f" (UID: "8fc7b14e-09f9-4630-bfc6-b98eab67753f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 03:01:58.833336 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:01:58.832727 4405 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3fd4ae35-84c1-45c1-a49c-14f2a8ed1c82-credential-socket" pod "3fd4ae35-84c1-45c1-a49c-14f2a8ed1c82" (UID: "3fd4ae35-84c1-45c1-a49c-14f2a8ed1c82"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 03:01:58.833336 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:01:58.833305 4405 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a525d6f7-79b5-4409-8137-319328626c10-workload-socket" pod "a525d6f7-79b5-4409-8137-319328626c10" (UID: "a525d6f7-79b5-4409-8137-319328626c10"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 03:01:58.833336 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:01:58.833326 4405 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3fd4ae35-84c1-45c1-a49c-14f2a8ed1c82-workload-socket" pod "3fd4ae35-84c1-45c1-a49c-14f2a8ed1c82" (UID: "3fd4ae35-84c1-45c1-a49c-14f2a8ed1c82"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 03:01:58.833773 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:01:58.833563 4405 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8fc7b14e-09f9-4630-bfc6-b98eab67753f-kube-api-access-qh6vr" pod "8fc7b14e-09f9-4630-bfc6-b98eab67753f" (UID: "8fc7b14e-09f9-4630-bfc6-b98eab67753f"). InnerVolumeSpecName "kube-api-access-qh6vr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 03:01:58.834175 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:01:58.834134 4405 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3fd4ae35-84c1-45c1-a49c-14f2a8ed1c82-istio-data" pod "3fd4ae35-84c1-45c1-a49c-14f2a8ed1c82" (UID: "3fd4ae35-84c1-45c1-a49c-14f2a8ed1c82"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 03:01:58.835881 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:01:58.835819 4405 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3fd4ae35-84c1-45c1-a49c-14f2a8ed1c82-kube-api-access-v5n9f" pod "3fd4ae35-84c1-45c1-a49c-14f2a8ed1c82" (UID: "3fd4ae35-84c1-45c1-a49c-14f2a8ed1c82"). InnerVolumeSpecName "kube-api-access-v5n9f". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 03:01:58.835979 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:01:58.835941 4405 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a525d6f7-79b5-4409-8137-319328626c10-kube-api-access-bnxc2" pod "a525d6f7-79b5-4409-8137-319328626c10" (UID: "a525d6f7-79b5-4409-8137-319328626c10"). InnerVolumeSpecName "kube-api-access-bnxc2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 03:01:58.836189 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:01:58.836049 4405 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3fd4ae35-84c1-45c1-a49c-14f2a8ed1c82-istio-envoy" pod "3fd4ae35-84c1-45c1-a49c-14f2a8ed1c82" (UID: "3fd4ae35-84c1-45c1-a49c-14f2a8ed1c82"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 03:01:58.836189 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:01:58.836093 4405 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a525d6f7-79b5-4409-8137-319328626c10-istio-token" pod "a525d6f7-79b5-4409-8137-319328626c10" (UID: "a525d6f7-79b5-4409-8137-319328626c10"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 03:01:58.836189 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:01:58.836154 4405 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3fd4ae35-84c1-45c1-a49c-14f2a8ed1c82-istio-token" pod "3fd4ae35-84c1-45c1-a49c-14f2a8ed1c82" (UID: "3fd4ae35-84c1-45c1-a49c-14f2a8ed1c82"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 03:01:58.836949 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:01:58.836917 4405 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a525d6f7-79b5-4409-8137-319328626c10-istio-podinfo" pod "a525d6f7-79b5-4409-8137-319328626c10" (UID: "a525d6f7-79b5-4409-8137-319328626c10"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 04 03:01:58.836949 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:01:58.836916 4405 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/3fd4ae35-84c1-45c1-a49c-14f2a8ed1c82-istio-podinfo" pod "3fd4ae35-84c1-45c1-a49c-14f2a8ed1c82" (UID: "3fd4ae35-84c1-45c1-a49c-14f2a8ed1c82"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 04 03:01:58.837389 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:01:58.837367 4405 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8fc7b14e-09f9-4630-bfc6-b98eab67753f-istio-envoy" pod "8fc7b14e-09f9-4630-bfc6-b98eab67753f" (UID: "8fc7b14e-09f9-4630-bfc6-b98eab67753f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 03:01:58.837549 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:01:58.837526 4405 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a525d6f7-79b5-4409-8137-319328626c10-istio-envoy" pod "a525d6f7-79b5-4409-8137-319328626c10" (UID: "a525d6f7-79b5-4409-8137-319328626c10"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 03:01:58.837783 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:01:58.837758 4405 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/8fc7b14e-09f9-4630-bfc6-b98eab67753f-istio-podinfo" pod "8fc7b14e-09f9-4630-bfc6-b98eab67753f" (UID: "8fc7b14e-09f9-4630-bfc6-b98eab67753f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 04 03:01:58.837916 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:01:58.837902 4405 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8fc7b14e-09f9-4630-bfc6-b98eab67753f-istio-token" pod "8fc7b14e-09f9-4630-bfc6-b98eab67753f" (UID: "8fc7b14e-09f9-4630-bfc6-b98eab67753f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 03:01:58.933566 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:01:58.933531 4405 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a525d6f7-79b5-4409-8137-319328626c10-istio-data\") on node \"ip-10-0-134-125.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:01:58.933566 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:01:58.933564 4405 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a525d6f7-79b5-4409-8137-319328626c10-istiod-ca-cert\") on node \"ip-10-0-134-125.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:01:58.933868 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:01:58.933579 4405 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v5n9f\" (UniqueName: \"kubernetes.io/projected/3fd4ae35-84c1-45c1-a49c-14f2a8ed1c82-kube-api-access-v5n9f\") on node \"ip-10-0-134-125.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:01:58.933868 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:01:58.933595 4405 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qh6vr\" (UniqueName: \"kubernetes.io/projected/8fc7b14e-09f9-4630-bfc6-b98eab67753f-kube-api-access-qh6vr\") on node \"ip-10-0-134-125.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:01:58.933868 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:01:58.933608 4405 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8fc7b14e-09f9-4630-bfc6-b98eab67753f-istio-data\") on node \"ip-10-0-134-125.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:01:58.933868 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:01:58.933618 4405 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8fc7b14e-09f9-4630-bfc6-b98eab67753f-istio-podinfo\") on node \"ip-10-0-134-125.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:01:58.933868 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:01:58.933630 4405 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8fc7b14e-09f9-4630-bfc6-b98eab67753f-istiod-ca-cert\") on node \"ip-10-0-134-125.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:01:58.933868 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:01:58.933643 4405 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8fc7b14e-09f9-4630-bfc6-b98eab67753f-istio-envoy\") on node \"ip-10-0-134-125.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:01:58.933868 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:01:58.933655 4405 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8fc7b14e-09f9-4630-bfc6-b98eab67753f-istio-token\") on node \"ip-10-0-134-125.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:01:58.933868 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:01:58.933705 4405 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3fd4ae35-84c1-45c1-a49c-14f2a8ed1c82-credential-socket\") on node \"ip-10-0-134-125.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:01:58.933868 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:01:58.933718 4405 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3fd4ae35-84c1-45c1-a49c-14f2a8ed1c82-workload-socket\") on node \"ip-10-0-134-125.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:01:58.933868 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:01:58.933729 4405 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3fd4ae35-84c1-45c1-a49c-14f2a8ed1c82-istio-data\") on node \"ip-10-0-134-125.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:01:58.933868 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:01:58.933741 4405 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a525d6f7-79b5-4409-8137-319328626c10-workload-socket\") on node \"ip-10-0-134-125.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:01:58.933868 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:01:58.933754 4405 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a525d6f7-79b5-4409-8137-319328626c10-istio-token\") on node \"ip-10-0-134-125.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:01:58.933868 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:01:58.933766 4405 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3fd4ae35-84c1-45c1-a49c-14f2a8ed1c82-istio-token\") on node \"ip-10-0-134-125.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:01:58.933868 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:01:58.933779 4405 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a525d6f7-79b5-4409-8137-319328626c10-istio-envoy\") on node \"ip-10-0-134-125.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:01:58.933868 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:01:58.933792 4405 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3fd4ae35-84c1-45c1-a49c-14f2a8ed1c82-istio-envoy\") on node \"ip-10-0-134-125.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:01:58.933868 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:01:58.933804 4405 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a525d6f7-79b5-4409-8137-319328626c10-istio-podinfo\") on node \"ip-10-0-134-125.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:01:58.933868 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:01:58.933817 4405 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bnxc2\" (UniqueName: \"kubernetes.io/projected/a525d6f7-79b5-4409-8137-319328626c10-kube-api-access-bnxc2\") on node \"ip-10-0-134-125.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:01:58.933868 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:01:58.933831 4405 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3fd4ae35-84c1-45c1-a49c-14f2a8ed1c82-istio-podinfo\") on node \"ip-10-0-134-125.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:01:59.702759 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:01:59.702728 4405 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-e516ff-openshift-default-6f4b957485-bjmn9" Nov 04 03:01:59.704313 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:01:59.704281 4405 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-e516ff-openshift-default-6f4b957485-w2plx" Nov 04 03:01:59.704732 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:01:59.704639 4405 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-e516ff-openshift-default-6f4b957485-w2plx" event={"ID":"a525d6f7-79b5-4409-8137-319328626c10","Type":"ContainerDied","Data":"1f083139fb3a8410adcf1be621e2a23358604bd623962c53393d36ffee309259"} Nov 04 03:01:59.704872 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:01:59.704756 4405 scope.go:122] "RemoveContainer" containerID="48cd8b87ccdd12489dc788842979decee8e4e3f709d453c6c0c8b9d98ec655f4" Nov 04 03:01:59.706218 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:01:59.705981 4405 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-e516ff-openshift-default-6f4b957485-7pg2l" Nov 04 03:01:59.706218 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:01:59.705983 4405 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-e516ff-openshift-default-6f4b957485-7pg2l" event={"ID":"8fc7b14e-09f9-4630-bfc6-b98eab67753f","Type":"ContainerDied","Data":"f7490e5285cf059ef915ad60914f6db9b77e4aeb088169a635e5fb82ef76b3da"} Nov 04 03:01:59.717174 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:01:59.717154 4405 scope.go:122] "RemoveContainer" containerID="5ff33d8a8d5b1b7f130a605e52248650a506b2ee611d906f466d983b36b612ac" Nov 04 03:01:59.868257 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:01:59.868223 4405 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-e516ff-openshift-default-6f4b957485-7pg2l"] Nov 04 03:01:59.917926 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:01:59.917895 4405 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-e516ff-openshift-default-6f4b957485-7pg2l"] Nov 04 03:02:00.028316 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:02:00.028125 4405 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-e516ff-openshift-default-6f4b957485-w2plx"] Nov 04 03:02:00.032960 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:02:00.032935 4405 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-e516ff-openshift-default-6f4b957485-w2plx"] Nov 04 03:02:00.142538 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:02:00.142490 4405 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-e516ff-openshift-default-6f4b957485-bjmn9"] Nov 04 03:02:00.148278 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:02:00.148247 4405 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-e516ff-openshift-default-6f4b957485-bjmn9"] Nov 04 03:02:01.702415 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:02:01.702382 4405 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3fd4ae35-84c1-45c1-a49c-14f2a8ed1c82" path="/var/lib/kubelet/pods/3fd4ae35-84c1-45c1-a49c-14f2a8ed1c82/volumes" Nov 04 03:02:01.702947 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:02:01.702932 4405 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8fc7b14e-09f9-4630-bfc6-b98eab67753f" path="/var/lib/kubelet/pods/8fc7b14e-09f9-4630-bfc6-b98eab67753f/volumes" Nov 04 03:02:01.703403 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:02:01.703390 4405 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a525d6f7-79b5-4409-8137-319328626c10" path="/var/lib/kubelet/pods/a525d6f7-79b5-4409-8137-319328626c10/volumes" Nov 04 03:02:06.885281 ip-10-0-134-125 kubenswrapper[4405]: E1104 03:02:06.885242 4405 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5f2vb" containerName="extractor" Nov 04 03:02:09.733984 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:02:09.733947 4405 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-35df3e-8675b47557-ms62p"] Nov 04 03:02:09.734622 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:02:09.734604 4405 cpu_manager.go:414] "containerMap: removing container" podUID="8fc7b14e-09f9-4630-bfc6-b98eab67753f" containerName="istio-proxy" Nov 04 03:02:09.734709 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:02:09.734624 4405 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8fc7b14e-09f9-4630-bfc6-b98eab67753f" containerName="istio-proxy" Nov 04 03:02:09.734709 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:02:09.734644 4405 cpu_manager.go:414] "containerMap: removing container" podUID="a525d6f7-79b5-4409-8137-319328626c10" containerName="istio-proxy" Nov 04 03:02:09.734709 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:02:09.734649 4405 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a525d6f7-79b5-4409-8137-319328626c10" containerName="istio-proxy" Nov 04 03:02:09.734709 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:02:09.734657 4405 cpu_manager.go:414] "containerMap: removing container" podUID="3fd4ae35-84c1-45c1-a49c-14f2a8ed1c82" containerName="istio-proxy" Nov 04 03:02:09.734709 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:02:09.734680 4405 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3fd4ae35-84c1-45c1-a49c-14f2a8ed1c82" containerName="istio-proxy" Nov 04 03:02:09.734709 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:02:09.734701 4405 cpu_manager.go:414] "containerMap: removing container" podUID="3fd4ae35-84c1-45c1-a49c-14f2a8ed1c82" containerName="istio-proxy" Nov 04 03:02:09.734709 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:02:09.734708 4405 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3fd4ae35-84c1-45c1-a49c-14f2a8ed1c82" containerName="istio-proxy" Nov 04 03:02:09.735002 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:02:09.734819 4405 memory_manager.go:365] "RemoveStaleState removing state" podUID="8fc7b14e-09f9-4630-bfc6-b98eab67753f" containerName="istio-proxy" Nov 04 03:02:09.735002 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:02:09.734831 4405 memory_manager.go:365] "RemoveStaleState removing state" podUID="a525d6f7-79b5-4409-8137-319328626c10" containerName="istio-proxy" Nov 04 03:02:09.735002 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:02:09.734837 4405 memory_manager.go:365] "RemoveStaleState removing state" podUID="3fd4ae35-84c1-45c1-a49c-14f2a8ed1c82" containerName="istio-proxy" Nov 04 03:02:09.735002 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:02:09.734844 4405 memory_manager.go:365] "RemoveStaleState removing state" podUID="3fd4ae35-84c1-45c1-a49c-14f2a8ed1c82" containerName="istio-proxy" Nov 04 03:02:09.739680 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:02:09.739638 4405 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-35df3e-8675b47557-ms62p" Nov 04 03:02:09.792397 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:02:09.792346 4405 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-w4-35df3e-serving-cert\"" Nov 04 03:02:09.795496 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:02:09.795462 4405 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kttb5\" (UniqueName: \"kubernetes.io/projected/c75d3a55-98c5-4af1-96ff-596d556597b9-kube-api-access-kttb5\") pod \"maas-api-e2e-shared-b-w4-35df3e-8675b47557-ms62p\" (UID: \"c75d3a55-98c5-4af1-96ff-596d556597b9\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-35df3e-8675b47557-ms62p" Nov 04 03:02:09.795778 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:02:09.795749 4405 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c75d3a55-98c5-4af1-96ff-596d556597b9-maas-api-tls\") pod \"maas-api-e2e-shared-b-w4-35df3e-8675b47557-ms62p\" (UID: \"c75d3a55-98c5-4af1-96ff-596d556597b9\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-35df3e-8675b47557-ms62p" Nov 04 03:02:09.795879 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:02:09.795801 4405 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c75d3a55-98c5-4af1-96ff-596d556597b9-metrics-certs\") pod \"maas-api-e2e-shared-b-w4-35df3e-8675b47557-ms62p\" (UID: \"c75d3a55-98c5-4af1-96ff-596d556597b9\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-35df3e-8675b47557-ms62p" Nov 04 03:02:09.795942 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:02:09.795886 4405 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c75d3a55-98c5-4af1-96ff-596d556597b9-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w4-35df3e-8675b47557-ms62p\" (UID: \"c75d3a55-98c5-4af1-96ff-596d556597b9\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-35df3e-8675b47557-ms62p" Nov 04 03:02:09.847904 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:02:09.847869 4405 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-35df3e-8675b47557-ms62p"] Nov 04 03:02:09.897680 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:02:09.897628 4405 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kttb5\" (UniqueName: \"kubernetes.io/projected/c75d3a55-98c5-4af1-96ff-596d556597b9-kube-api-access-kttb5\") pod \"maas-api-e2e-shared-b-w4-35df3e-8675b47557-ms62p\" (UID: \"c75d3a55-98c5-4af1-96ff-596d556597b9\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-35df3e-8675b47557-ms62p" Nov 04 03:02:09.897875 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:02:09.897706 4405 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c75d3a55-98c5-4af1-96ff-596d556597b9-maas-api-tls\") pod \"maas-api-e2e-shared-b-w4-35df3e-8675b47557-ms62p\" (UID: \"c75d3a55-98c5-4af1-96ff-596d556597b9\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-35df3e-8675b47557-ms62p" Nov 04 03:02:09.897875 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:02:09.897739 4405 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c75d3a55-98c5-4af1-96ff-596d556597b9-metrics-certs\") pod \"maas-api-e2e-shared-b-w4-35df3e-8675b47557-ms62p\" (UID: \"c75d3a55-98c5-4af1-96ff-596d556597b9\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-35df3e-8675b47557-ms62p" Nov 04 03:02:09.897875 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:02:09.897808 4405 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c75d3a55-98c5-4af1-96ff-596d556597b9-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w4-35df3e-8675b47557-ms62p\" (UID: \"c75d3a55-98c5-4af1-96ff-596d556597b9\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-35df3e-8675b47557-ms62p" Nov 04 03:02:09.898070 ip-10-0-134-125 kubenswrapper[4405]: E1104 03:02:09.897987 4405 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-35df3e-serving-cert: secret "maas-api-e2e-shared-b-w4-35df3e-serving-cert" not found Nov 04 03:02:09.898209 ip-10-0-134-125 kubenswrapper[4405]: E1104 03:02:09.898175 4405 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c75d3a55-98c5-4af1-96ff-596d556597b9-maas-api-tls podName:c75d3a55-98c5-4af1-96ff-596d556597b9 nodeName:}" failed. No retries permitted until 2026-11-04 03:02:10.398153626 +0000 UTC m=+1776.828853748 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/c75d3a55-98c5-4af1-96ff-596d556597b9-maas-api-tls") pod "maas-api-e2e-shared-b-w4-35df3e-8675b47557-ms62p" (UID: "c75d3a55-98c5-4af1-96ff-596d556597b9") : secret "maas-api-e2e-shared-b-w4-35df3e-serving-cert" not found Nov 04 03:02:09.899024 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:02:09.898986 4405 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c75d3a55-98c5-4af1-96ff-596d556597b9-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w4-35df3e-8675b47557-ms62p\" (UID: \"c75d3a55-98c5-4af1-96ff-596d556597b9\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-35df3e-8675b47557-ms62p" Nov 04 03:02:09.911552 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:02:09.911514 4405 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c75d3a55-98c5-4af1-96ff-596d556597b9-metrics-certs\") pod \"maas-api-e2e-shared-b-w4-35df3e-8675b47557-ms62p\" (UID: \"c75d3a55-98c5-4af1-96ff-596d556597b9\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-35df3e-8675b47557-ms62p" Nov 04 03:02:09.959588 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:02:09.959556 4405 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kttb5\" (UniqueName: \"kubernetes.io/projected/c75d3a55-98c5-4af1-96ff-596d556597b9-kube-api-access-kttb5\") pod \"maas-api-e2e-shared-b-w4-35df3e-8675b47557-ms62p\" (UID: \"c75d3a55-98c5-4af1-96ff-596d556597b9\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-35df3e-8675b47557-ms62p" Nov 04 03:02:10.406210 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:02:10.406158 4405 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c75d3a55-98c5-4af1-96ff-596d556597b9-maas-api-tls\") pod \"maas-api-e2e-shared-b-w4-35df3e-8675b47557-ms62p\" (UID: \"c75d3a55-98c5-4af1-96ff-596d556597b9\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-35df3e-8675b47557-ms62p" Nov 04 03:02:10.408913 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:02:10.408886 4405 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c75d3a55-98c5-4af1-96ff-596d556597b9-maas-api-tls\") pod \"maas-api-e2e-shared-b-w4-35df3e-8675b47557-ms62p\" (UID: \"c75d3a55-98c5-4af1-96ff-596d556597b9\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-35df3e-8675b47557-ms62p" Nov 04 03:02:10.655882 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:02:10.655838 4405 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-35df3e-8675b47557-ms62p" Nov 04 03:02:11.040152 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:02:11.040118 4405 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-35df3e-8675b47557-ms62p"] Nov 04 03:02:11.041914 ip-10-0-134-125 kubenswrapper[4405]: W1104 03:02:11.041873 4405 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc75d3a55_98c5_4af1_96ff_596d556597b9.slice/crio-3e7bd26893707ea04d49a05a2246db235c5a1567d4ae5ef0b97e573b927d58cf WatchSource:0}: Error finding container 3e7bd26893707ea04d49a05a2246db235c5a1567d4ae5ef0b97e573b927d58cf: Status 404 returned error can't find the container with id 3e7bd26893707ea04d49a05a2246db235c5a1567d4ae5ef0b97e573b927d58cf Nov 04 03:02:11.043529 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:02:11.043506 4405 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 04 03:02:11.793545 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:02:11.793510 4405 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-35df3e-8675b47557-ms62p" event={"ID":"c75d3a55-98c5-4af1-96ff-596d556597b9","Type":"ContainerStarted","Data":"3e7bd26893707ea04d49a05a2246db235c5a1567d4ae5ef0b97e573b927d58cf"} Nov 04 03:02:12.805945 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:02:12.805905 4405 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-35df3e-8675b47557-ms62p" event={"ID":"c75d3a55-98c5-4af1-96ff-596d556597b9","Type":"ContainerStarted","Data":"5edb540a5308ac2a9393bc43d1395c33fe254a732fd71a5cd37eceed82b49fdd"} Nov 04 03:02:12.806459 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:02:12.806113 4405 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-35df3e-8675b47557-ms62p" Nov 04 03:02:12.929708 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:02:12.929620 4405 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-35df3e-8675b47557-ms62p" podStartSLOduration=3.000541753 podStartE2EDuration="3.929600315s" totalImagesPullingTime="929.058562ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 03:02:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 03:02:12.923572154 +0000 UTC m=+1779.354272275" watchObservedRunningTime="2026-11-04 03:02:12.929600315 +0000 UTC m=+1779.360300436" Nov 04 03:02:18.023876 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:02:18.023843 4405 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-bl5mg"] Nov 04 03:02:18.818983 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:02:18.818949 4405 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-35df3e-8675b47557-ms62p" Nov 04 03:02:22.572704 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:02:22.571757 4405 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-ebb7fd/distinct-2-w6-ebb7fd-kserve-6f68c5db86-zrpjs" podUID="b7b786e0-9c94-4126-9949-cd354ebbfb2a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 03:02:25.135244 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:02:25.135202 4405 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-57f116-79648fc5c8-pg42j"] Nov 04 03:02:25.140292 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:02:25.140271 4405 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-57f116-79648fc5c8-pg42j" Nov 04 03:02:25.247991 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:02:25.247954 4405 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-w2-57f116-serving-cert\"" Nov 04 03:02:25.253726 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:02:25.253692 4405 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-57f116-79648fc5c8-pg42j"] Nov 04 03:02:25.307095 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:02:25.307057 4405 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d2227982-1e00-4ef1-9858-1abfe945bb84-metrics-certs\") pod \"maas-api-e2e-shared-a-w2-57f116-79648fc5c8-pg42j\" (UID: \"d2227982-1e00-4ef1-9858-1abfe945bb84\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-57f116-79648fc5c8-pg42j" Nov 04 03:02:25.307278 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:02:25.307256 4405 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d2227982-1e00-4ef1-9858-1abfe945bb84-maas-api-tls\") pod \"maas-api-e2e-shared-a-w2-57f116-79648fc5c8-pg42j\" (UID: \"d2227982-1e00-4ef1-9858-1abfe945bb84\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-57f116-79648fc5c8-pg42j" Nov 04 03:02:25.307324 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:02:25.307287 4405 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d2227982-1e00-4ef1-9858-1abfe945bb84-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w2-57f116-79648fc5c8-pg42j\" (UID: \"d2227982-1e00-4ef1-9858-1abfe945bb84\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-57f116-79648fc5c8-pg42j" Nov 04 03:02:25.307324 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:02:25.307312 4405 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6m4bx\" (UniqueName: \"kubernetes.io/projected/d2227982-1e00-4ef1-9858-1abfe945bb84-kube-api-access-6m4bx\") pod \"maas-api-e2e-shared-a-w2-57f116-79648fc5c8-pg42j\" (UID: \"d2227982-1e00-4ef1-9858-1abfe945bb84\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-57f116-79648fc5c8-pg42j" Nov 04 03:02:25.417343 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:02:25.413571 4405 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d2227982-1e00-4ef1-9858-1abfe945bb84-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w2-57f116-79648fc5c8-pg42j\" (UID: \"d2227982-1e00-4ef1-9858-1abfe945bb84\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-57f116-79648fc5c8-pg42j" Nov 04 03:02:25.417613 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:02:25.417575 4405 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6m4bx\" (UniqueName: \"kubernetes.io/projected/d2227982-1e00-4ef1-9858-1abfe945bb84-kube-api-access-6m4bx\") pod \"maas-api-e2e-shared-a-w2-57f116-79648fc5c8-pg42j\" (UID: \"d2227982-1e00-4ef1-9858-1abfe945bb84\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-57f116-79648fc5c8-pg42j" Nov 04 03:02:25.417822 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:02:25.417797 4405 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d2227982-1e00-4ef1-9858-1abfe945bb84-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w2-57f116-79648fc5c8-pg42j\" (UID: \"d2227982-1e00-4ef1-9858-1abfe945bb84\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-57f116-79648fc5c8-pg42j" Nov 04 03:02:25.417906 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:02:25.417839 4405 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d2227982-1e00-4ef1-9858-1abfe945bb84-metrics-certs\") pod \"maas-api-e2e-shared-a-w2-57f116-79648fc5c8-pg42j\" (UID: \"d2227982-1e00-4ef1-9858-1abfe945bb84\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-57f116-79648fc5c8-pg42j" Nov 04 03:02:25.418046 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:02:25.418028 4405 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d2227982-1e00-4ef1-9858-1abfe945bb84-maas-api-tls\") pod \"maas-api-e2e-shared-a-w2-57f116-79648fc5c8-pg42j\" (UID: \"d2227982-1e00-4ef1-9858-1abfe945bb84\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-57f116-79648fc5c8-pg42j" Nov 04 03:02:25.418234 ip-10-0-134-125 kubenswrapper[4405]: E1104 03:02:25.418215 4405 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-57f116-serving-cert: secret "maas-api-e2e-shared-a-w2-57f116-serving-cert" not found Nov 04 03:02:25.418312 ip-10-0-134-125 kubenswrapper[4405]: E1104 03:02:25.418279 4405 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d2227982-1e00-4ef1-9858-1abfe945bb84-maas-api-tls podName:d2227982-1e00-4ef1-9858-1abfe945bb84 nodeName:}" failed. No retries permitted until 2026-11-04 03:02:25.91826105 +0000 UTC m=+1792.348961152 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/d2227982-1e00-4ef1-9858-1abfe945bb84-maas-api-tls") pod "maas-api-e2e-shared-a-w2-57f116-79648fc5c8-pg42j" (UID: "d2227982-1e00-4ef1-9858-1abfe945bb84") : secret "maas-api-e2e-shared-a-w2-57f116-serving-cert" not found Nov 04 03:02:25.420822 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:02:25.420799 4405 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d2227982-1e00-4ef1-9858-1abfe945bb84-metrics-certs\") pod \"maas-api-e2e-shared-a-w2-57f116-79648fc5c8-pg42j\" (UID: \"d2227982-1e00-4ef1-9858-1abfe945bb84\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-57f116-79648fc5c8-pg42j" Nov 04 03:02:25.481690 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:02:25.481645 4405 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6m4bx\" (UniqueName: \"kubernetes.io/projected/d2227982-1e00-4ef1-9858-1abfe945bb84-kube-api-access-6m4bx\") pod \"maas-api-e2e-shared-a-w2-57f116-79648fc5c8-pg42j\" (UID: \"d2227982-1e00-4ef1-9858-1abfe945bb84\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-57f116-79648fc5c8-pg42j" Nov 04 03:02:25.927236 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:02:25.927202 4405 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d2227982-1e00-4ef1-9858-1abfe945bb84-maas-api-tls\") pod \"maas-api-e2e-shared-a-w2-57f116-79648fc5c8-pg42j\" (UID: \"d2227982-1e00-4ef1-9858-1abfe945bb84\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-57f116-79648fc5c8-pg42j" Nov 04 03:02:25.929769 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:02:25.929740 4405 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d2227982-1e00-4ef1-9858-1abfe945bb84-maas-api-tls\") pod \"maas-api-e2e-shared-a-w2-57f116-79648fc5c8-pg42j\" (UID: \"d2227982-1e00-4ef1-9858-1abfe945bb84\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-57f116-79648fc5c8-pg42j" Nov 04 03:02:26.054688 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:02:26.054644 4405 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-57f116-79648fc5c8-pg42j" Nov 04 03:02:26.438722 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:02:26.438694 4405 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-57f116-79648fc5c8-pg42j"] Nov 04 03:02:26.439464 ip-10-0-134-125 kubenswrapper[4405]: W1104 03:02:26.439430 4405 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd2227982_1e00_4ef1_9858_1abfe945bb84.slice/crio-97eb9647ef385aae404d394cfb76300e79c69d53f8480a8ffd9fe90a0b00b578 WatchSource:0}: Error finding container 97eb9647ef385aae404d394cfb76300e79c69d53f8480a8ffd9fe90a0b00b578: Status 404 returned error can't find the container with id 97eb9647ef385aae404d394cfb76300e79c69d53f8480a8ffd9fe90a0b00b578 Nov 04 03:02:27.162839 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:02:27.162799 4405 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-57f116-79648fc5c8-pg42j" event={"ID":"d2227982-1e00-4ef1-9858-1abfe945bb84","Type":"ContainerStarted","Data":"97eb9647ef385aae404d394cfb76300e79c69d53f8480a8ffd9fe90a0b00b578"} Nov 04 03:02:28.172869 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:02:28.172836 4405 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-57f116-79648fc5c8-pg42j" event={"ID":"d2227982-1e00-4ef1-9858-1abfe945bb84","Type":"ContainerStarted","Data":"34dd768ead4d27389133974c1b192d6e5118c0022b823eea38e38136c1babd2a"} Nov 04 03:02:28.173263 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:02:28.172967 4405 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-57f116-79648fc5c8-pg42j" Nov 04 03:02:28.293771 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:02:28.293721 4405 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-57f116-79648fc5c8-pg42j" podStartSLOduration=2.472045984 podStartE2EDuration="3.293706711s" totalImagesPullingTime="821.660727ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 03:02:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 03:02:28.289909703 +0000 UTC m=+1794.720609824" watchObservedRunningTime="2026-11-04 03:02:28.293706711 +0000 UTC m=+1794.724406831" Nov 04 03:02:33.918006 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:02:33.917971 4405 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-vwcx8_ffbb2d83-5a93-4a5b-bce3-e40afb8f58ea/ovn-acl-logging/0.log" Nov 04 03:02:33.919057 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:02:33.919034 4405 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-vwcx8_ffbb2d83-5a93-4a5b-bce3-e40afb8f58ea/ovn-acl-logging/0.log" Nov 04 03:02:34.187047 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:02:34.186971 4405 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-57f116-79648fc5c8-pg42j" Nov 04 03:02:36.885856 ip-10-0-134-125 kubenswrapper[4405]: E1104 03:02:36.885821 4405 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5f2vb" containerName="extractor" Nov 04 03:02:53.885859 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:02:53.885793 4405 patch_prober.go:29] interesting pod/e2e-shared-b-w4-35df3e-openshift-default-6c585b76f-vl6wz container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.96:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 04 03:02:53.885859 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:02:53.885860 4405 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-w4-35df3e-openshift-default-6c585b76f-vl6wz" podUID="18ffb552-4917-4c03-a3b5-aa9c6a89f0e7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.96:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 04 03:02:56.479034 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:02:56.478943 4405 generic.go:356] "Generic (PLEG): container finished" podID="18ffb552-4917-4c03-a3b5-aa9c6a89f0e7" containerID="9b9bed2e07dc5af1f69f981953d14d27e67d46f16bd7a3d70d9975fe219dfc75" exitCode=0 Nov 04 03:02:56.479034 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:02:56.479014 4405 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-35df3e-openshift-default-6c585b76f-vl6wz" event={"ID":"18ffb552-4917-4c03-a3b5-aa9c6a89f0e7","Type":"ContainerDied","Data":"9b9bed2e07dc5af1f69f981953d14d27e67d46f16bd7a3d70d9975fe219dfc75"} Nov 04 03:02:56.479526 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:02:56.479480 4405 scope.go:122] "RemoveContainer" containerID="9b9bed2e07dc5af1f69f981953d14d27e67d46f16bd7a3d70d9975fe219dfc75" Nov 04 03:02:56.481197 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:02:56.481172 4405 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 04 03:02:56.481268 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:02:56.481243 4405 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 04 03:02:56.481303 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:02:56.481274 4405 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 04 03:02:57.486887 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:02:57.486851 4405 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-35df3e-openshift-default-6c585b76f-vl6wz" event={"ID":"18ffb552-4917-4c03-a3b5-aa9c6a89f0e7","Type":"ContainerStarted","Data":"f135664a6bfe876b38d0757d58e9bcb5949fb9a20d7ffc8fad582c1c859c5ee7"} Nov 04 03:02:57.647986 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:02:57.647956 4405 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w4-35df3e-openshift-default-6c585b76f-vl6wz" Nov 04 03:02:57.648270 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:02:57.648246 4405 patch_prober.go:29] interesting pod/e2e-shared-b-w4-35df3e-openshift-default-6c585b76f-vl6wz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.96:15021/healthz/ready\": dial tcp 10.134.0.96:15021: connect: connection refused" start-of-body= Nov 04 03:02:57.648338 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:02:57.648317 4405 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-35df3e-openshift-default-6c585b76f-vl6wz" podUID="18ffb552-4917-4c03-a3b5-aa9c6a89f0e7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.96:15021/healthz/ready\": dial tcp 10.134.0.96:15021: connect: connection refused" Nov 04 03:02:58.647328 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:02:58.647294 4405 patch_prober.go:29] interesting pod/e2e-shared-b-w4-35df3e-openshift-default-6c585b76f-vl6wz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.96:15021/healthz/ready\": dial tcp 10.134.0.96:15021: connect: connection refused" start-of-body= Nov 04 03:02:58.647765 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:02:58.647353 4405 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-35df3e-openshift-default-6c585b76f-vl6wz" podUID="18ffb552-4917-4c03-a3b5-aa9c6a89f0e7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.96:15021/healthz/ready\": dial tcp 10.134.0.96:15021: connect: connection refused" Nov 04 03:02:59.647473 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:02:59.647436 4405 patch_prober.go:29] interesting pod/e2e-shared-b-w4-35df3e-openshift-default-6c585b76f-vl6wz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.96:15021/healthz/ready\": dial tcp 10.134.0.96:15021: connect: connection refused" start-of-body= Nov 04 03:02:59.647987 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:02:59.647514 4405 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-35df3e-openshift-default-6c585b76f-vl6wz" podUID="18ffb552-4917-4c03-a3b5-aa9c6a89f0e7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.96:15021/healthz/ready\": dial tcp 10.134.0.96:15021: connect: connection refused" Nov 04 03:03:00.647654 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:03:00.647618 4405 patch_prober.go:29] interesting pod/e2e-shared-b-w4-35df3e-openshift-default-6c585b76f-vl6wz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.96:15021/healthz/ready\": dial tcp 10.134.0.96:15021: connect: connection refused" start-of-body= Nov 04 03:03:00.648155 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:03:00.648123 4405 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-35df3e-openshift-default-6c585b76f-vl6wz" podUID="18ffb552-4917-4c03-a3b5-aa9c6a89f0e7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.96:15021/healthz/ready\": dial tcp 10.134.0.96:15021: connect: connection refused" Nov 04 03:03:01.647424 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:03:01.647389 4405 patch_prober.go:29] interesting pod/e2e-shared-b-w4-35df3e-openshift-default-6c585b76f-vl6wz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.96:15021/healthz/ready\": dial tcp 10.134.0.96:15021: connect: connection refused" start-of-body= Nov 04 03:03:01.647740 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:03:01.647710 4405 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-35df3e-openshift-default-6c585b76f-vl6wz" podUID="18ffb552-4917-4c03-a3b5-aa9c6a89f0e7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.96:15021/healthz/ready\": dial tcp 10.134.0.96:15021: connect: connection refused" Nov 04 03:03:02.648356 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:03:02.648315 4405 patch_prober.go:29] interesting pod/e2e-shared-b-w4-35df3e-openshift-default-6c585b76f-vl6wz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.96:15021/healthz/ready\": dial tcp 10.134.0.96:15021: connect: connection refused" start-of-body= Nov 04 03:03:02.648813 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:03:02.648372 4405 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-35df3e-openshift-default-6c585b76f-vl6wz" podUID="18ffb552-4917-4c03-a3b5-aa9c6a89f0e7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.96:15021/healthz/ready\": dial tcp 10.134.0.96:15021: connect: connection refused" Nov 04 03:03:03.652560 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:03:03.652523 4405 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w4-35df3e-openshift-default-6c585b76f-vl6wz" Nov 04 03:03:03.653025 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:03:03.652968 4405 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w4-35df3e-openshift-default-6c585b76f-vl6wz" Nov 04 03:03:03.653867 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:03:03.653844 4405 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w4-35df3e-openshift-default-6c585b76f-vl6wz" Nov 04 03:03:06.886137 ip-10-0-134-125 kubenswrapper[4405]: E1104 03:03:06.886094 4405 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5f2vb" containerName="extractor" Nov 04 03:03:07.788446 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:03:07.788406 4405 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-a021b3-775855fc55-hd92k"] Nov 04 03:03:07.793348 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:03:07.793326 4405 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-a021b3-775855fc55-hd92k" Nov 04 03:03:07.849312 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:03:07.849264 4405 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-a-a021b3\"" Nov 04 03:03:07.909728 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:03:07.897573 4405 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8e29627f-f096-4fe6-b5a9-d66ed0631172-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-a021b3-775855fc55-hd92k\" (UID: \"8e29627f-f096-4fe6-b5a9-d66ed0631172\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-a021b3-775855fc55-hd92k" Nov 04 03:03:07.911298 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:03:07.911150 4405 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x68k9\" (UniqueName: \"kubernetes.io/projected/8e29627f-f096-4fe6-b5a9-d66ed0631172-kube-api-access-x68k9\") pod \"payload-pre-processing-e2e-api-a-a021b3-775855fc55-hd92k\" (UID: \"8e29627f-f096-4fe6-b5a9-d66ed0631172\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-a021b3-775855fc55-hd92k" Nov 04 03:03:07.927278 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:03:07.927237 4405 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-a-a021b3-dockercfg-rc599\"" Nov 04 03:03:08.009301 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:03:08.009270 4405 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-a021b3-775855fc55-hd92k"] Nov 04 03:03:08.029589 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:03:08.029551 4405 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8e29627f-f096-4fe6-b5a9-d66ed0631172-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-a021b3-775855fc55-hd92k\" (UID: \"8e29627f-f096-4fe6-b5a9-d66ed0631172\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-a021b3-775855fc55-hd92k" Nov 04 03:03:08.029969 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:03:08.029944 4405 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x68k9\" (UniqueName: \"kubernetes.io/projected/8e29627f-f096-4fe6-b5a9-d66ed0631172-kube-api-access-x68k9\") pod \"payload-pre-processing-e2e-api-a-a021b3-775855fc55-hd92k\" (UID: \"8e29627f-f096-4fe6-b5a9-d66ed0631172\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-a021b3-775855fc55-hd92k" Nov 04 03:03:08.043947 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:03:08.031578 4405 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8e29627f-f096-4fe6-b5a9-d66ed0631172-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-a021b3-775855fc55-hd92k\" (UID: \"8e29627f-f096-4fe6-b5a9-d66ed0631172\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-a021b3-775855fc55-hd92k" Nov 04 03:03:08.100097 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:03:08.100019 4405 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x68k9\" (UniqueName: \"kubernetes.io/projected/8e29627f-f096-4fe6-b5a9-d66ed0631172-kube-api-access-x68k9\") pod \"payload-pre-processing-e2e-api-a-a021b3-775855fc55-hd92k\" (UID: \"8e29627f-f096-4fe6-b5a9-d66ed0631172\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-a021b3-775855fc55-hd92k" Nov 04 03:03:08.123690 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:03:08.116275 4405 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-a021b3-775855fc55-hd92k" Nov 04 03:03:08.138739 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:03:08.137402 4405 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-a021b3-6556664496-wmwrg"] Nov 04 03:03:08.142575 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:03:08.142552 4405 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-a021b3-6556664496-wmwrg" Nov 04 03:03:08.145143 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:03:08.145116 4405 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-a021b3-6556664496-wmwrg"] Nov 04 03:03:08.242609 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:03:08.242345 4405 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hzpcj\" (UniqueName: \"kubernetes.io/projected/596af222-bf6a-44fe-9e2b-5ced60a615c0-kube-api-access-hzpcj\") pod \"payload-processing-e2e-api-a-a021b3-6556664496-wmwrg\" (UID: \"596af222-bf6a-44fe-9e2b-5ced60a615c0\") " pod="openshift-ingress/payload-processing-e2e-api-a-a021b3-6556664496-wmwrg" Nov 04 03:03:08.242609 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:03:08.242473 4405 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/596af222-bf6a-44fe-9e2b-5ced60a615c0-plugins-config-volume\") pod \"payload-processing-e2e-api-a-a021b3-6556664496-wmwrg\" (UID: \"596af222-bf6a-44fe-9e2b-5ced60a615c0\") " pod="openshift-ingress/payload-processing-e2e-api-a-a021b3-6556664496-wmwrg" Nov 04 03:03:08.343439 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:03:08.343407 4405 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/596af222-bf6a-44fe-9e2b-5ced60a615c0-plugins-config-volume\") pod \"payload-processing-e2e-api-a-a021b3-6556664496-wmwrg\" (UID: \"596af222-bf6a-44fe-9e2b-5ced60a615c0\") " pod="openshift-ingress/payload-processing-e2e-api-a-a021b3-6556664496-wmwrg" Nov 04 03:03:08.343608 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:03:08.343594 4405 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hzpcj\" (UniqueName: \"kubernetes.io/projected/596af222-bf6a-44fe-9e2b-5ced60a615c0-kube-api-access-hzpcj\") pod \"payload-processing-e2e-api-a-a021b3-6556664496-wmwrg\" (UID: \"596af222-bf6a-44fe-9e2b-5ced60a615c0\") " pod="openshift-ingress/payload-processing-e2e-api-a-a021b3-6556664496-wmwrg" Nov 04 03:03:08.344162 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:03:08.344143 4405 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/596af222-bf6a-44fe-9e2b-5ced60a615c0-plugins-config-volume\") pod \"payload-processing-e2e-api-a-a021b3-6556664496-wmwrg\" (UID: \"596af222-bf6a-44fe-9e2b-5ced60a615c0\") " pod="openshift-ingress/payload-processing-e2e-api-a-a021b3-6556664496-wmwrg" Nov 04 03:03:08.408177 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:03:08.408141 4405 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hzpcj\" (UniqueName: \"kubernetes.io/projected/596af222-bf6a-44fe-9e2b-5ced60a615c0-kube-api-access-hzpcj\") pod \"payload-processing-e2e-api-a-a021b3-6556664496-wmwrg\" (UID: \"596af222-bf6a-44fe-9e2b-5ced60a615c0\") " pod="openshift-ingress/payload-processing-e2e-api-a-a021b3-6556664496-wmwrg" Nov 04 03:03:08.496137 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:03:08.496099 4405 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-a021b3-6556664496-wmwrg" Nov 04 03:03:08.541895 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:03:08.541849 4405 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-a021b3-775855fc55-hd92k"] Nov 04 03:03:08.544951 ip-10-0-134-125 kubenswrapper[4405]: W1104 03:03:08.544912 4405 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8e29627f_f096_4fe6_b5a9_d66ed0631172.slice/crio-1e99d697f83beeb7e52b68d95d9c6c9ca0ab48f670c22dce67abcc33fc7a4f97 WatchSource:0}: Error finding container 1e99d697f83beeb7e52b68d95d9c6c9ca0ab48f670c22dce67abcc33fc7a4f97: Status 404 returned error can't find the container with id 1e99d697f83beeb7e52b68d95d9c6c9ca0ab48f670c22dce67abcc33fc7a4f97 Nov 04 03:03:08.604540 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:03:08.604454 4405 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-a021b3-775855fc55-hd92k" event={"ID":"8e29627f-f096-4fe6-b5a9-d66ed0631172","Type":"ContainerStarted","Data":"1e99d697f83beeb7e52b68d95d9c6c9ca0ab48f670c22dce67abcc33fc7a4f97"} Nov 04 03:03:08.880798 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:03:08.880769 4405 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-a021b3-6556664496-wmwrg"] Nov 04 03:03:08.881712 ip-10-0-134-125 kubenswrapper[4405]: W1104 03:03:08.881652 4405 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod596af222_bf6a_44fe_9e2b_5ced60a615c0.slice/crio-6e047f069d0d39bdedb8d01bc8d732b52313feeb0194a666eb30e457e48034d5 WatchSource:0}: Error finding container 6e047f069d0d39bdedb8d01bc8d732b52313feeb0194a666eb30e457e48034d5: Status 404 returned error can't find the container with id 6e047f069d0d39bdedb8d01bc8d732b52313feeb0194a666eb30e457e48034d5 Nov 04 03:03:09.614239 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:03:09.614187 4405 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-a021b3-775855fc55-hd92k" event={"ID":"8e29627f-f096-4fe6-b5a9-d66ed0631172","Type":"ContainerStarted","Data":"0b819818c87e4fb3a0d65c83a4390740669fd23e2809aaf65b2e0358badb0014"} Nov 04 03:03:09.614737 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:03:09.614494 4405 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-a021b3-775855fc55-hd92k" Nov 04 03:03:09.616326 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:03:09.616299 4405 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-a021b3-6556664496-wmwrg" event={"ID":"596af222-bf6a-44fe-9e2b-5ced60a615c0","Type":"ContainerStarted","Data":"2db06180baa75ad13abc50a8deed840c02d6079bec73f5d17bcd2ddf80eed2c2"} Nov 04 03:03:09.616454 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:03:09.616349 4405 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-a021b3-6556664496-wmwrg" event={"ID":"596af222-bf6a-44fe-9e2b-5ced60a615c0","Type":"ContainerStarted","Data":"6e047f069d0d39bdedb8d01bc8d732b52313feeb0194a666eb30e457e48034d5"} Nov 04 03:03:09.616454 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:03:09.616381 4405 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-a-a021b3-6556664496-wmwrg" Nov 04 03:03:09.739348 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:03:09.739295 4405 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-a-a021b3-775855fc55-hd92k" podStartSLOduration=2.739281756 podStartE2EDuration="2.739281756s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 03:03:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 03:03:09.736124655 +0000 UTC m=+1836.166824776" watchObservedRunningTime="2026-11-04 03:03:09.739281756 +0000 UTC m=+1836.169981875" Nov 04 03:03:09.854654 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:03:09.854595 4405 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-a-a021b3-6556664496-wmwrg" podStartSLOduration=2.854578578 podStartE2EDuration="2.854578578s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 03:03:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 03:03:09.851190494 +0000 UTC m=+1836.281890614" watchObservedRunningTime="2026-11-04 03:03:09.854578578 +0000 UTC m=+1836.285278698" Nov 04 03:03:20.626252 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:03:20.626218 4405 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-a-a021b3-6556664496-wmwrg" Nov 04 03:03:20.626802 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:03:20.626308 4405 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-a021b3-775855fc55-hd92k" Nov 04 03:03:25.315772 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:03:25.315725 4405 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-bl5mg"] Nov 04 03:03:36.817268 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:03:36.817177 4405 generic.go:356] "Generic (PLEG): container finished" podID="e21df403-3af7-4a07-8793-452782d57a3b" containerID="8e03d4d6805ad573b5dd4a47fc3ba1380e7ed057cac59b1280b3a354a3b3933e" exitCode=0 Nov 04 03:03:36.817268 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:03:36.817226 4405 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w5-4b2bf4-openshift-default-59785c9d99-b52gb" event={"ID":"e21df403-3af7-4a07-8793-452782d57a3b","Type":"ContainerDied","Data":"8e03d4d6805ad573b5dd4a47fc3ba1380e7ed057cac59b1280b3a354a3b3933e"} Nov 04 03:03:36.817898 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:03:36.817761 4405 scope.go:122] "RemoveContainer" containerID="8e03d4d6805ad573b5dd4a47fc3ba1380e7ed057cac59b1280b3a354a3b3933e" Nov 04 03:03:36.819613 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:03:36.819584 4405 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 04 03:03:36.819721 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:03:36.819659 4405 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 04 03:03:36.819721 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:03:36.819715 4405 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 04 03:03:36.921537 ip-10-0-134-125 kubenswrapper[4405]: E1104 03:03:36.921502 4405 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5f2vb" containerName="extractor" Nov 04 03:03:37.827754 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:03:37.827720 4405 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w5-4b2bf4-openshift-default-59785c9d99-b52gb" event={"ID":"e21df403-3af7-4a07-8793-452782d57a3b","Type":"ContainerStarted","Data":"d34e8eb65f5fcad3a97427105b2cda652ec0581ac447867457d6f0460d4c7e6b"} Nov 04 03:03:38.766072 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:03:38.766035 4405 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w5-4b2bf4-openshift-default-59785c9d99-b52gb" Nov 04 03:03:38.766439 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:03:38.766415 4405 patch_prober.go:29] interesting pod/e2e-shared-b-w5-4b2bf4-openshift-default-59785c9d99-b52gb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.100:15021/healthz/ready\": dial tcp 10.134.0.100:15021: connect: connection refused" start-of-body= Nov 04 03:03:38.766510 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:03:38.766472 4405 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w5-4b2bf4-openshift-default-59785c9d99-b52gb" podUID="e21df403-3af7-4a07-8793-452782d57a3b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.100:15021/healthz/ready\": dial tcp 10.134.0.100:15021: connect: connection refused" Nov 04 03:03:39.765553 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:03:39.765518 4405 patch_prober.go:29] interesting pod/e2e-shared-b-w5-4b2bf4-openshift-default-59785c9d99-b52gb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.100:15021/healthz/ready\": dial tcp 10.134.0.100:15021: connect: connection refused" start-of-body= Nov 04 03:03:39.765971 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:03:39.765569 4405 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w5-4b2bf4-openshift-default-59785c9d99-b52gb" podUID="e21df403-3af7-4a07-8793-452782d57a3b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.100:15021/healthz/ready\": dial tcp 10.134.0.100:15021: connect: connection refused" Nov 04 03:03:39.790118 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:03:39.790083 4405 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-ebb7fd-openshift-default-544f5768f-47dj7"] Nov 04 03:03:39.790330 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:03:39.790305 4405 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w6-ebb7fd-openshift-default-544f5768f-47dj7" podUID="8762bab6-0f89-4b6b-a87a-dc1555608862" containerName="istio-proxy" containerID="cri-o://fc4bfde43f7d0c1abd2d9b987a382e6165987a5b3dd18701500856f6b51184aa" gracePeriod=30 Nov 04 03:03:40.766255 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:03:40.766217 4405 patch_prober.go:29] interesting pod/e2e-shared-b-w5-4b2bf4-openshift-default-59785c9d99-b52gb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.100:15021/healthz/ready\": dial tcp 10.134.0.100:15021: connect: connection refused" start-of-body= Nov 04 03:03:40.766699 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:03:40.766285 4405 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w5-4b2bf4-openshift-default-59785c9d99-b52gb" podUID="e21df403-3af7-4a07-8793-452782d57a3b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.100:15021/healthz/ready\": dial tcp 10.134.0.100:15021: connect: connection refused" Nov 04 03:03:41.766280 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:03:41.766239 4405 patch_prober.go:29] interesting pod/e2e-shared-b-w5-4b2bf4-openshift-default-59785c9d99-b52gb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.100:15021/healthz/ready\": dial tcp 10.134.0.100:15021: connect: connection refused" start-of-body= Nov 04 03:03:41.766714 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:03:41.766303 4405 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w5-4b2bf4-openshift-default-59785c9d99-b52gb" podUID="e21df403-3af7-4a07-8793-452782d57a3b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.100:15021/healthz/ready\": dial tcp 10.134.0.100:15021: connect: connection refused" Nov 04 03:03:42.766075 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:03:42.766040 4405 patch_prober.go:29] interesting pod/e2e-shared-b-w5-4b2bf4-openshift-default-59785c9d99-b52gb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.100:15021/healthz/ready\": dial tcp 10.134.0.100:15021: connect: connection refused" start-of-body= Nov 04 03:03:42.766275 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:03:42.766095 4405 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w5-4b2bf4-openshift-default-59785c9d99-b52gb" podUID="e21df403-3af7-4a07-8793-452782d57a3b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.100:15021/healthz/ready\": dial tcp 10.134.0.100:15021: connect: connection refused" Nov 04 03:03:43.770850 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:03:43.770821 4405 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w5-4b2bf4-openshift-default-59785c9d99-b52gb" Nov 04 03:03:43.771343 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:03:43.771329 4405 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w5-4b2bf4-openshift-default-59785c9d99-b52gb" Nov 04 03:03:43.772526 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:03:43.772496 4405 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w5-4b2bf4-openshift-default-59785c9d99-b52gb" Nov 04 03:03:44.267066 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:03:44.267027 4405 patch_prober.go:29] interesting pod/e2e-worker-w6-ebb7fd-openshift-default-544f5768f-47dj7 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 04 03:03:44.267238 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:03:44.267111 4405 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w6-ebb7fd-openshift-default-544f5768f-47dj7" podUID="8762bab6-0f89-4b6b-a87a-dc1555608862" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 04 03:03:44.906364 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:03:44.906269 4405 generic.go:356] "Generic (PLEG): container finished" podID="8762bab6-0f89-4b6b-a87a-dc1555608862" containerID="fc4bfde43f7d0c1abd2d9b987a382e6165987a5b3dd18701500856f6b51184aa" exitCode=0 Nov 04 03:03:44.906947 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:03:44.906726 4405 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-ebb7fd-openshift-default-544f5768f-47dj7" event={"ID":"8762bab6-0f89-4b6b-a87a-dc1555608862","Type":"ContainerDied","Data":"fc4bfde43f7d0c1abd2d9b987a382e6165987a5b3dd18701500856f6b51184aa"} Nov 04 03:03:45.062275 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:03:45.062210 4405 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-ebb7fd-openshift-default-544f5768f-47dj7" Nov 04 03:03:45.194909 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:03:45.194865 4405 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8762bab6-0f89-4b6b-a87a-dc1555608862-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8762bab6-0f89-4b6b-a87a-dc1555608862-istio-envoy\") pod \"8762bab6-0f89-4b6b-a87a-dc1555608862\" (UID: \"8762bab6-0f89-4b6b-a87a-dc1555608862\") " Nov 04 03:03:45.195113 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:03:45.194980 4405 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8762bab6-0f89-4b6b-a87a-dc1555608862-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8762bab6-0f89-4b6b-a87a-dc1555608862-istio-data\") pod \"8762bab6-0f89-4b6b-a87a-dc1555608862\" (UID: \"8762bab6-0f89-4b6b-a87a-dc1555608862\") " Nov 04 03:03:45.195113 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:03:45.195012 4405 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/8762bab6-0f89-4b6b-a87a-dc1555608862-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8762bab6-0f89-4b6b-a87a-dc1555608862-istio-podinfo\") pod \"8762bab6-0f89-4b6b-a87a-dc1555608862\" (UID: \"8762bab6-0f89-4b6b-a87a-dc1555608862\") " Nov 04 03:03:45.195113 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:03:45.195073 4405 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8762bab6-0f89-4b6b-a87a-dc1555608862-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8762bab6-0f89-4b6b-a87a-dc1555608862-credential-socket\") pod \"8762bab6-0f89-4b6b-a87a-dc1555608862\" (UID: \"8762bab6-0f89-4b6b-a87a-dc1555608862\") " Nov 04 03:03:45.195113 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:03:45.195099 4405 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8762bab6-0f89-4b6b-a87a-dc1555608862-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8762bab6-0f89-4b6b-a87a-dc1555608862-workload-certs\") pod \"8762bab6-0f89-4b6b-a87a-dc1555608862\" (UID: \"8762bab6-0f89-4b6b-a87a-dc1555608862\") " Nov 04 03:03:45.195343 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:03:45.195233 4405 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8762bab6-0f89-4b6b-a87a-dc1555608862-istio-token\" (UniqueName: \"kubernetes.io/projected/8762bab6-0f89-4b6b-a87a-dc1555608862-istio-token\") pod \"8762bab6-0f89-4b6b-a87a-dc1555608862\" (UID: \"8762bab6-0f89-4b6b-a87a-dc1555608862\") " Nov 04 03:03:45.195343 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:03:45.195267 4405 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8762bab6-0f89-4b6b-a87a-dc1555608862-istio-data" pod "8762bab6-0f89-4b6b-a87a-dc1555608862" (UID: "8762bab6-0f89-4b6b-a87a-dc1555608862"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 03:03:45.195343 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:03:45.195333 4405 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8762bab6-0f89-4b6b-a87a-dc1555608862-kube-api-access-rfsbz\" (UniqueName: \"kubernetes.io/projected/8762bab6-0f89-4b6b-a87a-dc1555608862-kube-api-access-rfsbz\") pod \"8762bab6-0f89-4b6b-a87a-dc1555608862\" (UID: \"8762bab6-0f89-4b6b-a87a-dc1555608862\") " Nov 04 03:03:45.195497 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:03:45.195388 4405 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8762bab6-0f89-4b6b-a87a-dc1555608862-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8762bab6-0f89-4b6b-a87a-dc1555608862-istiod-ca-cert\") pod \"8762bab6-0f89-4b6b-a87a-dc1555608862\" (UID: \"8762bab6-0f89-4b6b-a87a-dc1555608862\") " Nov 04 03:03:45.195497 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:03:45.195418 4405 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8762bab6-0f89-4b6b-a87a-dc1555608862-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8762bab6-0f89-4b6b-a87a-dc1555608862-workload-socket\") pod \"8762bab6-0f89-4b6b-a87a-dc1555608862\" (UID: \"8762bab6-0f89-4b6b-a87a-dc1555608862\") " Nov 04 03:03:45.195497 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:03:45.195433 4405 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8762bab6-0f89-4b6b-a87a-dc1555608862-workload-certs" pod "8762bab6-0f89-4b6b-a87a-dc1555608862" (UID: "8762bab6-0f89-4b6b-a87a-dc1555608862"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 03:03:45.195652 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:03:45.195615 4405 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8762bab6-0f89-4b6b-a87a-dc1555608862-workload-socket" pod "8762bab6-0f89-4b6b-a87a-dc1555608862" (UID: "8762bab6-0f89-4b6b-a87a-dc1555608862"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 03:03:45.195757 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:03:45.195644 4405 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8762bab6-0f89-4b6b-a87a-dc1555608862-istiod-ca-cert" pod "8762bab6-0f89-4b6b-a87a-dc1555608862" (UID: "8762bab6-0f89-4b6b-a87a-dc1555608862"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 03:03:45.195757 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:03:45.195689 4405 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8762bab6-0f89-4b6b-a87a-dc1555608862-credential-socket" pod "8762bab6-0f89-4b6b-a87a-dc1555608862" (UID: "8762bab6-0f89-4b6b-a87a-dc1555608862"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 03:03:45.196172 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:03:45.196146 4405 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8762bab6-0f89-4b6b-a87a-dc1555608862-istiod-ca-cert\") on node \"ip-10-0-134-125.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:03:45.196172 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:03:45.196173 4405 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8762bab6-0f89-4b6b-a87a-dc1555608862-workload-socket\") on node \"ip-10-0-134-125.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:03:45.196341 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:03:45.196184 4405 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8762bab6-0f89-4b6b-a87a-dc1555608862-istio-data\") on node \"ip-10-0-134-125.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:03:45.196341 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:03:45.196193 4405 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8762bab6-0f89-4b6b-a87a-dc1555608862-credential-socket\") on node \"ip-10-0-134-125.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:03:45.196341 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:03:45.196201 4405 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8762bab6-0f89-4b6b-a87a-dc1555608862-workload-certs\") on node \"ip-10-0-134-125.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:03:45.198280 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:03:45.198252 4405 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/8762bab6-0f89-4b6b-a87a-dc1555608862-istio-podinfo" pod "8762bab6-0f89-4b6b-a87a-dc1555608862" (UID: "8762bab6-0f89-4b6b-a87a-dc1555608862"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 04 03:03:45.198280 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:03:45.198259 4405 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8762bab6-0f89-4b6b-a87a-dc1555608862-istio-envoy" pod "8762bab6-0f89-4b6b-a87a-dc1555608862" (UID: "8762bab6-0f89-4b6b-a87a-dc1555608862"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 03:03:45.198475 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:03:45.198377 4405 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8762bab6-0f89-4b6b-a87a-dc1555608862-istio-token" pod "8762bab6-0f89-4b6b-a87a-dc1555608862" (UID: "8762bab6-0f89-4b6b-a87a-dc1555608862"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 03:03:45.198475 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:03:45.198432 4405 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8762bab6-0f89-4b6b-a87a-dc1555608862-kube-api-access-rfsbz" pod "8762bab6-0f89-4b6b-a87a-dc1555608862" (UID: "8762bab6-0f89-4b6b-a87a-dc1555608862"). InnerVolumeSpecName "kube-api-access-rfsbz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 03:03:45.297932 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:03:45.297889 4405 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8762bab6-0f89-4b6b-a87a-dc1555608862-istio-podinfo\") on node \"ip-10-0-134-125.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:03:45.297932 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:03:45.297930 4405 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8762bab6-0f89-4b6b-a87a-dc1555608862-istio-token\") on node \"ip-10-0-134-125.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:03:45.298150 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:03:45.297946 4405 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rfsbz\" (UniqueName: \"kubernetes.io/projected/8762bab6-0f89-4b6b-a87a-dc1555608862-kube-api-access-rfsbz\") on node \"ip-10-0-134-125.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:03:45.298150 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:03:45.297963 4405 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8762bab6-0f89-4b6b-a87a-dc1555608862-istio-envoy\") on node \"ip-10-0-134-125.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:03:45.916952 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:03:45.916915 4405 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-ebb7fd-openshift-default-544f5768f-47dj7" event={"ID":"8762bab6-0f89-4b6b-a87a-dc1555608862","Type":"ContainerDied","Data":"7cf83655ba6da668b44f80c5d84840fac97b28ad4e7f11f43159ec3ddd6802a4"} Nov 04 03:03:45.916952 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:03:45.916942 4405 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-ebb7fd-openshift-default-544f5768f-47dj7" Nov 04 03:03:45.917552 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:03:45.917019 4405 scope.go:122] "RemoveContainer" containerID="fc4bfde43f7d0c1abd2d9b987a382e6165987a5b3dd18701500856f6b51184aa" Nov 04 03:03:46.084802 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:03:46.084772 4405 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-ebb7fd-openshift-default-544f5768f-47dj7"] Nov 04 03:03:46.136252 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:03:46.136214 4405 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w6-ebb7fd-openshift-default-544f5768f-47dj7"] Nov 04 03:03:47.705095 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:03:47.705056 4405 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8762bab6-0f89-4b6b-a87a-dc1555608862" path="/var/lib/kubelet/pods/8762bab6-0f89-4b6b-a87a-dc1555608862/volumes" Nov 04 03:03:48.445540 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:03:48.445498 4405 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-cdb11d-87b64cbd4-bvfpd"] Nov 04 03:03:48.446264 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:03:48.446243 4405 cpu_manager.go:414] "containerMap: removing container" podUID="8762bab6-0f89-4b6b-a87a-dc1555608862" containerName="istio-proxy" Nov 04 03:03:48.446406 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:03:48.446265 4405 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8762bab6-0f89-4b6b-a87a-dc1555608862" containerName="istio-proxy" Nov 04 03:03:48.446466 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:03:48.446458 4405 memory_manager.go:365] "RemoveStaleState removing state" podUID="8762bab6-0f89-4b6b-a87a-dc1555608862" containerName="istio-proxy" Nov 04 03:03:48.451506 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:03:48.451481 4405 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-cdb11d-87b64cbd4-bvfpd" Nov 04 03:03:48.504606 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:03:48.504571 4405 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-a-cdb11d\"" Nov 04 03:03:48.555037 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:03:48.555002 4405 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-a-cdb11d-dockercfg-vxtff\"" Nov 04 03:03:48.561460 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:03:48.561412 4405 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-cdb11d-87b64cbd4-bvfpd"] Nov 04 03:03:48.566013 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:03:48.565039 4405 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7acf0cd1-800b-4796-9b10-a2041f971104-plugins-config-volume\") pod \"payload-processing-e2e-api-a-cdb11d-87b64cbd4-bvfpd\" (UID: \"7acf0cd1-800b-4796-9b10-a2041f971104\") " pod="openshift-ingress/payload-processing-e2e-api-a-cdb11d-87b64cbd4-bvfpd" Nov 04 03:03:48.566013 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:03:48.565425 4405 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gmqnc\" (UniqueName: \"kubernetes.io/projected/7acf0cd1-800b-4796-9b10-a2041f971104-kube-api-access-gmqnc\") pod \"payload-processing-e2e-api-a-cdb11d-87b64cbd4-bvfpd\" (UID: \"7acf0cd1-800b-4796-9b10-a2041f971104\") " pod="openshift-ingress/payload-processing-e2e-api-a-cdb11d-87b64cbd4-bvfpd" Nov 04 03:03:48.632545 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:03:48.632511 4405 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-cdb11d-585c6fc6dc-dtbnf"] Nov 04 03:03:48.636761 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:03:48.636740 4405 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-cdb11d-585c6fc6dc-dtbnf" Nov 04 03:03:48.667736 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:03:48.667653 4405 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gmqnc\" (UniqueName: \"kubernetes.io/projected/7acf0cd1-800b-4796-9b10-a2041f971104-kube-api-access-gmqnc\") pod \"payload-processing-e2e-api-a-cdb11d-87b64cbd4-bvfpd\" (UID: \"7acf0cd1-800b-4796-9b10-a2041f971104\") " pod="openshift-ingress/payload-processing-e2e-api-a-cdb11d-87b64cbd4-bvfpd" Nov 04 03:03:48.668248 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:03:48.668221 4405 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7acf0cd1-800b-4796-9b10-a2041f971104-plugins-config-volume\") pod \"payload-processing-e2e-api-a-cdb11d-87b64cbd4-bvfpd\" (UID: \"7acf0cd1-800b-4796-9b10-a2041f971104\") " pod="openshift-ingress/payload-processing-e2e-api-a-cdb11d-87b64cbd4-bvfpd" Nov 04 03:03:48.669425 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:03:48.669388 4405 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7acf0cd1-800b-4796-9b10-a2041f971104-plugins-config-volume\") pod \"payload-processing-e2e-api-a-cdb11d-87b64cbd4-bvfpd\" (UID: \"7acf0cd1-800b-4796-9b10-a2041f971104\") " pod="openshift-ingress/payload-processing-e2e-api-a-cdb11d-87b64cbd4-bvfpd" Nov 04 03:03:48.763020 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:03:48.755634 4405 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-cdb11d-585c6fc6dc-dtbnf"] Nov 04 03:03:48.763781 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:03:48.763751 4405 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gmqnc\" (UniqueName: \"kubernetes.io/projected/7acf0cd1-800b-4796-9b10-a2041f971104-kube-api-access-gmqnc\") pod \"payload-processing-e2e-api-a-cdb11d-87b64cbd4-bvfpd\" (UID: \"7acf0cd1-800b-4796-9b10-a2041f971104\") " pod="openshift-ingress/payload-processing-e2e-api-a-cdb11d-87b64cbd4-bvfpd" Nov 04 03:03:48.768628 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:03:48.768588 4405 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-cdb11d-87b64cbd4-bvfpd" Nov 04 03:03:48.769596 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:03:48.769572 4405 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-glhr5\" (UniqueName: \"kubernetes.io/projected/b7f09efb-5b70-457a-93c0-9125cae4c2dc-kube-api-access-glhr5\") pod \"payload-pre-processing-e2e-api-a-cdb11d-585c6fc6dc-dtbnf\" (UID: \"b7f09efb-5b70-457a-93c0-9125cae4c2dc\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-cdb11d-585c6fc6dc-dtbnf" Nov 04 03:03:48.769749 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:03:48.769728 4405 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b7f09efb-5b70-457a-93c0-9125cae4c2dc-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-cdb11d-585c6fc6dc-dtbnf\" (UID: \"b7f09efb-5b70-457a-93c0-9125cae4c2dc\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-cdb11d-585c6fc6dc-dtbnf" Nov 04 03:03:48.871105 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:03:48.871072 4405 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-glhr5\" (UniqueName: \"kubernetes.io/projected/b7f09efb-5b70-457a-93c0-9125cae4c2dc-kube-api-access-glhr5\") pod \"payload-pre-processing-e2e-api-a-cdb11d-585c6fc6dc-dtbnf\" (UID: \"b7f09efb-5b70-457a-93c0-9125cae4c2dc\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-cdb11d-585c6fc6dc-dtbnf" Nov 04 03:03:48.871305 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:03:48.871147 4405 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b7f09efb-5b70-457a-93c0-9125cae4c2dc-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-cdb11d-585c6fc6dc-dtbnf\" (UID: \"b7f09efb-5b70-457a-93c0-9125cae4c2dc\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-cdb11d-585c6fc6dc-dtbnf" Nov 04 03:03:48.871949 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:03:48.871929 4405 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b7f09efb-5b70-457a-93c0-9125cae4c2dc-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-cdb11d-585c6fc6dc-dtbnf\" (UID: \"b7f09efb-5b70-457a-93c0-9125cae4c2dc\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-cdb11d-585c6fc6dc-dtbnf" Nov 04 03:03:48.943009 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:03:48.932394 4405 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-glhr5\" (UniqueName: \"kubernetes.io/projected/b7f09efb-5b70-457a-93c0-9125cae4c2dc-kube-api-access-glhr5\") pod \"payload-pre-processing-e2e-api-a-cdb11d-585c6fc6dc-dtbnf\" (UID: \"b7f09efb-5b70-457a-93c0-9125cae4c2dc\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-cdb11d-585c6fc6dc-dtbnf" Nov 04 03:03:48.959579 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:03:48.959539 4405 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-cdb11d-585c6fc6dc-dtbnf" Nov 04 03:03:49.166862 ip-10-0-134-125 kubenswrapper[4405]: W1104 03:03:49.166829 4405 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7acf0cd1_800b_4796_9b10_a2041f971104.slice/crio-86099ed10dce718f0e598ca98b5eb9aa2d625911c6a40428a511771e16a2022c WatchSource:0}: Error finding container 86099ed10dce718f0e598ca98b5eb9aa2d625911c6a40428a511771e16a2022c: Status 404 returned error can't find the container with id 86099ed10dce718f0e598ca98b5eb9aa2d625911c6a40428a511771e16a2022c Nov 04 03:03:49.167294 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:03:49.167268 4405 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-cdb11d-87b64cbd4-bvfpd"] Nov 04 03:03:49.343158 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:03:49.343131 4405 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-cdb11d-585c6fc6dc-dtbnf"] Nov 04 03:03:49.343324 ip-10-0-134-125 kubenswrapper[4405]: W1104 03:03:49.343293 4405 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb7f09efb_5b70_457a_93c0_9125cae4c2dc.slice/crio-7fcf14dd15fea81fe8a3cc7735eadcec747ffc8baa65df0bc0d502a9b5bb26f2 WatchSource:0}: Error finding container 7fcf14dd15fea81fe8a3cc7735eadcec747ffc8baa65df0bc0d502a9b5bb26f2: Status 404 returned error can't find the container with id 7fcf14dd15fea81fe8a3cc7735eadcec747ffc8baa65df0bc0d502a9b5bb26f2 Nov 04 03:03:49.950550 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:03:49.950499 4405 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-cdb11d-585c6fc6dc-dtbnf" event={"ID":"b7f09efb-5b70-457a-93c0-9125cae4c2dc","Type":"ContainerStarted","Data":"2e551a9e5311105bb05d1c69ee90c56d56381ee1193ec9e88484c06c96164099"} Nov 04 03:03:49.951020 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:03:49.950573 4405 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-cdb11d-585c6fc6dc-dtbnf" Nov 04 03:03:49.951020 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:03:49.950597 4405 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-cdb11d-585c6fc6dc-dtbnf" event={"ID":"b7f09efb-5b70-457a-93c0-9125cae4c2dc","Type":"ContainerStarted","Data":"7fcf14dd15fea81fe8a3cc7735eadcec747ffc8baa65df0bc0d502a9b5bb26f2"} Nov 04 03:03:49.952300 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:03:49.952272 4405 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-cdb11d-87b64cbd4-bvfpd" event={"ID":"7acf0cd1-800b-4796-9b10-a2041f971104","Type":"ContainerStarted","Data":"8bb720159944d524388822d75ceeefccdf1a72e4c5a6e2e792b273bbe538c932"} Nov 04 03:03:49.952431 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:03:49.952322 4405 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-cdb11d-87b64cbd4-bvfpd" event={"ID":"7acf0cd1-800b-4796-9b10-a2041f971104","Type":"ContainerStarted","Data":"86099ed10dce718f0e598ca98b5eb9aa2d625911c6a40428a511771e16a2022c"} Nov 04 03:03:49.952431 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:03:49.952345 4405 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-a-cdb11d-87b64cbd4-bvfpd" Nov 04 03:03:50.071339 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:03:50.071288 4405 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-a-cdb11d-585c6fc6dc-dtbnf" podStartSLOduration=2.071274082 podStartE2EDuration="2.071274082s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 03:03:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 03:03:50.069651924 +0000 UTC m=+1876.500352046" watchObservedRunningTime="2026-11-04 03:03:50.071274082 +0000 UTC m=+1876.501974202" Nov 04 03:03:50.185121 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:03:50.185059 4405 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-a-cdb11d-87b64cbd4-bvfpd" podStartSLOduration=2.185039425 podStartE2EDuration="2.185039425s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 03:03:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 03:03:50.18454116 +0000 UTC m=+1876.615241279" watchObservedRunningTime="2026-11-04 03:03:50.185039425 +0000 UTC m=+1876.615739551" Nov 04 03:03:53.866818 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:03:53.866763 4405 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-a0bcbc/premium-w2-a0bcbc-kserve-78dfd6d78c-lwth2" podUID="33b620c1-389c-48b4-b277-90252018c4b6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 03:03:53.894834 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:03:53.894781 4405 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-cdcf67/distinct-w3-cdcf67-kserve-667cc7f7d7-5m2lq" podUID="52ed22cc-96e4-4ef0-b34d-ed58c89a9183" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 03:03:53.948828 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:03:53.948783 4405 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-203c35/simulator-w4-203c35-kserve-f9c4bd695-999ns" podUID="eb0d8bdd-6808-4fd0-b9b5-567dff3e7ca5" containerName="main" probeResult="failure" output="Get \"https://10.134.0.46:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 04 03:03:54.238754 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:03:54.225760 4405 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-ebb7fd/distinct-w6-ebb7fd-kserve-55c47d8865-7z9pz" podUID="ccb8e181-5610-46da-bb8b-e0b07fa02ff1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 03:03:54.461824 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:03:54.461771 4405 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-fe4e0c/distinct-2-w0-fe4e0c-kserve-579bf4cc77-8pbq8" podUID="2b2ab722-d1bb-4376-bede-f5f4ee4562aa" containerName="main" probeResult="failure" output="Get \"https://10.134.0.68:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 03:03:54.463641 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:03:54.463605 4405 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-a0bcbc/distinct-w2-a0bcbc-kserve-596f94c7cd-szl4g" podUID="bac32934-8a57-4919-b435-13f367cbea3e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 03:03:54.575047 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:03:54.573778 4405 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-ebb7fd/distinct-2-w6-ebb7fd-kserve-6f68c5db86-zrpjs" podUID="b7b786e0-9c94-4126-9949-cd354ebbfb2a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 03:03:54.811693 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:03:54.803773 4405 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-e516ff/distinct-w1-e516ff-kserve-f4474cb9c-glpzw" podUID="8842ad09-22ff-4492-ba5d-d4cacf4eac75" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 03:03:54.949692 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:03:54.949279 4405 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-203c35/simulator-w4-203c35-kserve-f9c4bd695-999ns" podUID="eb0d8bdd-6808-4fd0-b9b5-567dff3e7ca5" containerName="main" probeResult="failure" output="Get \"https://10.134.0.46:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 03:03:54.950238 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:03:54.949872 4405 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-203c35/simulator-w4-203c35-kserve-f9c4bd695-999ns" Nov 04 03:03:54.967570 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:03:54.967529 4405 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-203c35/simulator-w4-203c35-kserve-f9c4bd695-999ns" Nov 04 03:04:00.970610 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:04:00.970560 4405 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-cdb11d-585c6fc6dc-dtbnf" Nov 04 03:04:00.971170 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:04:00.970775 4405 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-a-cdb11d-87b64cbd4-bvfpd" Nov 04 03:04:03.083845 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:04:03.083802 4405 patch_prober.go:29] interesting pod/e2e-shared-a-w2-57f116-openshift-default-75f4fc499-znglh container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.99:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 04 03:04:03.084271 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:04:03.083881 4405 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-w2-57f116-openshift-default-75f4fc499-znglh" podUID="fd2b4daf-d65f-44ca-8939-67778fe8ef5a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.99:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 04 03:04:04.485150 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:04:04.485117 4405 generic.go:356] "Generic (PLEG): container finished" podID="fd2b4daf-d65f-44ca-8939-67778fe8ef5a" containerID="7192f537f98158d936b5e94e6c021bd5b8a8ba642b9a06dea35cddcc897bf35f" exitCode=0 Nov 04 03:04:04.485604 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:04:04.485201 4405 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-57f116-openshift-default-75f4fc499-znglh" event={"ID":"fd2b4daf-d65f-44ca-8939-67778fe8ef5a","Type":"ContainerDied","Data":"7192f537f98158d936b5e94e6c021bd5b8a8ba642b9a06dea35cddcc897bf35f"} Nov 04 03:04:04.485784 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:04:04.485766 4405 scope.go:122] "RemoveContainer" containerID="7192f537f98158d936b5e94e6c021bd5b8a8ba642b9a06dea35cddcc897bf35f" Nov 04 03:04:04.487874 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:04:04.487840 4405 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 04 03:04:04.487995 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:04:04.487927 4405 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 04 03:04:04.487995 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:04:04.487976 4405 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 04 03:04:05.495692 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:04:05.494301 4405 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-57f116-openshift-default-75f4fc499-znglh" event={"ID":"fd2b4daf-d65f-44ca-8939-67778fe8ef5a","Type":"ContainerStarted","Data":"8990f651f55bcfb5f4dd6208c0a5730f4a31e23bad1d255db7883f28f1975764"} Nov 04 03:04:06.083646 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:04:06.083610 4405 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w2-57f116-openshift-default-75f4fc499-znglh" Nov 04 03:04:06.083867 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:04:06.083789 4405 patch_prober.go:29] interesting pod/e2e-shared-a-w2-57f116-openshift-default-75f4fc499-znglh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.99:15021/healthz/ready\": dial tcp 10.134.0.99:15021: connect: connection refused" start-of-body= Nov 04 03:04:06.083867 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:04:06.083843 4405 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-57f116-openshift-default-75f4fc499-znglh" podUID="fd2b4daf-d65f-44ca-8939-67778fe8ef5a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.99:15021/healthz/ready\": dial tcp 10.134.0.99:15021: connect: connection refused" Nov 04 03:04:06.885275 ip-10-0-134-125 kubenswrapper[4405]: E1104 03:04:06.885238 4405 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5f2vb" containerName="extractor" Nov 04 03:04:07.084481 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:04:07.084444 4405 patch_prober.go:29] interesting pod/e2e-shared-a-w2-57f116-openshift-default-75f4fc499-znglh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.99:15021/healthz/ready\": dial tcp 10.134.0.99:15021: connect: connection refused" start-of-body= Nov 04 03:04:07.084706 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:04:07.084508 4405 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-57f116-openshift-default-75f4fc499-znglh" podUID="fd2b4daf-d65f-44ca-8939-67778fe8ef5a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.99:15021/healthz/ready\": dial tcp 10.134.0.99:15021: connect: connection refused" Nov 04 03:04:08.084098 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:04:08.084065 4405 patch_prober.go:29] interesting pod/e2e-shared-a-w2-57f116-openshift-default-75f4fc499-znglh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.99:15021/healthz/ready\": dial tcp 10.134.0.99:15021: connect: connection refused" start-of-body= Nov 04 03:04:08.084478 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:04:08.084129 4405 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-57f116-openshift-default-75f4fc499-znglh" podUID="fd2b4daf-d65f-44ca-8939-67778fe8ef5a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.99:15021/healthz/ready\": dial tcp 10.134.0.99:15021: connect: connection refused" Nov 04 03:04:08.554752 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:04:08.553760 4405 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-00c846-5d7b447995-p8qhc"] Nov 04 03:04:08.564692 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:04:08.558201 4405 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-00c846-5d7b447995-p8qhc" Nov 04 03:04:08.614690 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:04:08.609661 4405 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-00c846-5d7b447995-p8qhc"] Nov 04 03:04:08.614690 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:04:08.610584 4405 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-97n2r\" (UniqueName: \"kubernetes.io/projected/10ad282e-7474-4938-8dee-81c65db8f4ae-kube-api-access-97n2r\") pod \"payload-pre-processing-e2e-shared-b-w2-00c846-5d7b447995-p8qhc\" (UID: \"10ad282e-7474-4938-8dee-81c65db8f4ae\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-00c846-5d7b447995-p8qhc" Nov 04 03:04:08.614690 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:04:08.610843 4405 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/10ad282e-7474-4938-8dee-81c65db8f4ae-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w2-00c846-5d7b447995-p8qhc\" (UID: \"10ad282e-7474-4938-8dee-81c65db8f4ae\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-00c846-5d7b447995-p8qhc" Nov 04 03:04:08.614690 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:04:08.611397 4405 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w2-00c846\"" Nov 04 03:04:08.614690 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:04:08.611591 4405 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w2-00c846-dockercfg-zj6wc\"" Nov 04 03:04:08.682014 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:04:08.681979 4405 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-00c846-6865789c84-c24hp"] Nov 04 03:04:08.685144 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:04:08.685119 4405 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-00c846-6865789c84-c24hp" Nov 04 03:04:08.729726 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:04:08.726532 4405 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-97n2r\" (UniqueName: \"kubernetes.io/projected/10ad282e-7474-4938-8dee-81c65db8f4ae-kube-api-access-97n2r\") pod \"payload-pre-processing-e2e-shared-b-w2-00c846-5d7b447995-p8qhc\" (UID: \"10ad282e-7474-4938-8dee-81c65db8f4ae\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-00c846-5d7b447995-p8qhc" Nov 04 03:04:08.729726 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:04:08.726723 4405 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/10ad282e-7474-4938-8dee-81c65db8f4ae-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w2-00c846-5d7b447995-p8qhc\" (UID: \"10ad282e-7474-4938-8dee-81c65db8f4ae\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-00c846-5d7b447995-p8qhc" Nov 04 03:04:08.730521 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:04:08.730493 4405 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/10ad282e-7474-4938-8dee-81c65db8f4ae-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w2-00c846-5d7b447995-p8qhc\" (UID: \"10ad282e-7474-4938-8dee-81c65db8f4ae\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-00c846-5d7b447995-p8qhc" Nov 04 03:04:08.746211 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:04:08.746179 4405 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-00c846-6865789c84-c24hp"] Nov 04 03:04:08.839703 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:04:08.835392 4405 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-97n2r\" (UniqueName: \"kubernetes.io/projected/10ad282e-7474-4938-8dee-81c65db8f4ae-kube-api-access-97n2r\") pod \"payload-pre-processing-e2e-shared-b-w2-00c846-5d7b447995-p8qhc\" (UID: \"10ad282e-7474-4938-8dee-81c65db8f4ae\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-00c846-5d7b447995-p8qhc" Nov 04 03:04:08.839953 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:04:08.839775 4405 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vbcd5\" (UniqueName: \"kubernetes.io/projected/f30fbe83-4589-49c6-bdd9-3e994b1a1d4d-kube-api-access-vbcd5\") pod \"payload-processing-e2e-shared-b-w2-00c846-6865789c84-c24hp\" (UID: \"f30fbe83-4589-49c6-bdd9-3e994b1a1d4d\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-00c846-6865789c84-c24hp" Nov 04 03:04:08.840117 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:04:08.840093 4405 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f30fbe83-4589-49c6-bdd9-3e994b1a1d4d-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w2-00c846-6865789c84-c24hp\" (UID: \"f30fbe83-4589-49c6-bdd9-3e994b1a1d4d\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-00c846-6865789c84-c24hp" Nov 04 03:04:08.946905 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:04:08.943779 4405 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vbcd5\" (UniqueName: \"kubernetes.io/projected/f30fbe83-4589-49c6-bdd9-3e994b1a1d4d-kube-api-access-vbcd5\") pod \"payload-processing-e2e-shared-b-w2-00c846-6865789c84-c24hp\" (UID: \"f30fbe83-4589-49c6-bdd9-3e994b1a1d4d\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-00c846-6865789c84-c24hp" Nov 04 03:04:08.946905 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:04:08.943845 4405 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f30fbe83-4589-49c6-bdd9-3e994b1a1d4d-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w2-00c846-6865789c84-c24hp\" (UID: \"f30fbe83-4589-49c6-bdd9-3e994b1a1d4d\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-00c846-6865789c84-c24hp" Nov 04 03:04:08.946905 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:04:08.945033 4405 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f30fbe83-4589-49c6-bdd9-3e994b1a1d4d-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w2-00c846-6865789c84-c24hp\" (UID: \"f30fbe83-4589-49c6-bdd9-3e994b1a1d4d\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-00c846-6865789c84-c24hp" Nov 04 03:04:08.966724 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:04:08.966378 4405 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-00c846-5d7b447995-p8qhc" Nov 04 03:04:09.034597 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:04:09.034474 4405 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vbcd5\" (UniqueName: \"kubernetes.io/projected/f30fbe83-4589-49c6-bdd9-3e994b1a1d4d-kube-api-access-vbcd5\") pod \"payload-processing-e2e-shared-b-w2-00c846-6865789c84-c24hp\" (UID: \"f30fbe83-4589-49c6-bdd9-3e994b1a1d4d\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-00c846-6865789c84-c24hp" Nov 04 03:04:09.046017 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:04:09.045981 4405 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-00c846-6865789c84-c24hp" Nov 04 03:04:09.086282 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:04:09.084003 4405 patch_prober.go:29] interesting pod/e2e-shared-a-w2-57f116-openshift-default-75f4fc499-znglh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.99:15021/healthz/ready\": dial tcp 10.134.0.99:15021: connect: connection refused" start-of-body= Nov 04 03:04:09.086282 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:04:09.084062 4405 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-57f116-openshift-default-75f4fc499-znglh" podUID="fd2b4daf-d65f-44ca-8939-67778fe8ef5a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.99:15021/healthz/ready\": dial tcp 10.134.0.99:15021: connect: connection refused" Nov 04 03:04:09.418747 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:04:09.408877 4405 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-00c846-5d7b447995-p8qhc"] Nov 04 03:04:09.530045 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:04:09.528440 4405 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-00c846-6865789c84-c24hp"] Nov 04 03:04:09.565909 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:04:09.565422 4405 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-00c846-5d7b447995-p8qhc" event={"ID":"10ad282e-7474-4938-8dee-81c65db8f4ae","Type":"ContainerStarted","Data":"e7275d6a058359b7202bdfb6abb769d805a8392045b32389e04c7c2c02a39681"} Nov 04 03:04:09.565909 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:04:09.565487 4405 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-00c846-5d7b447995-p8qhc" event={"ID":"10ad282e-7474-4938-8dee-81c65db8f4ae","Type":"ContainerStarted","Data":"9b2bf6d510325c789cedde851d20ebe281921f13afc6af2ff5efc349d3bb93a8"} Nov 04 03:04:09.566798 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:04:09.566754 4405 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-00c846-5d7b447995-p8qhc" Nov 04 03:04:09.569110 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:04:09.569088 4405 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-00c846-6865789c84-c24hp" event={"ID":"f30fbe83-4589-49c6-bdd9-3e994b1a1d4d","Type":"ContainerStarted","Data":"c1edee9b7bb51c538d0b4baa897d010abdabb3b0f283a21455f67916d965df8c"} Nov 04 03:04:09.720762 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:04:09.706613 4405 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-00c846-5d7b447995-p8qhc" podStartSLOduration=1.706591194 podStartE2EDuration="1.706591194s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 03:04:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 03:04:09.703162214 +0000 UTC m=+1896.133862336" watchObservedRunningTime="2026-11-04 03:04:09.706591194 +0000 UTC m=+1896.137291317" Nov 04 03:04:10.083846 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:04:10.083808 4405 patch_prober.go:29] interesting pod/e2e-shared-a-w2-57f116-openshift-default-75f4fc499-znglh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.99:15021/healthz/ready\": dial tcp 10.134.0.99:15021: connect: connection refused" start-of-body= Nov 04 03:04:10.084062 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:04:10.083885 4405 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-57f116-openshift-default-75f4fc499-znglh" podUID="fd2b4daf-d65f-44ca-8939-67778fe8ef5a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.99:15021/healthz/ready\": dial tcp 10.134.0.99:15021: connect: connection refused" Nov 04 03:04:10.578155 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:04:10.578060 4405 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-00c846-6865789c84-c24hp" event={"ID":"f30fbe83-4589-49c6-bdd9-3e994b1a1d4d","Type":"ContainerStarted","Data":"8ab91cade19aa24aa2f656d0bb1ca1713f4db2280c4b685520ac1f8ca946c11d"} Nov 04 03:04:11.083881 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:04:11.083843 4405 patch_prober.go:29] interesting pod/e2e-shared-a-w2-57f116-openshift-default-75f4fc499-znglh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.99:15021/healthz/ready\": dial tcp 10.134.0.99:15021: connect: connection refused" start-of-body= Nov 04 03:04:11.084077 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:04:11.083896 4405 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-57f116-openshift-default-75f4fc499-znglh" podUID="fd2b4daf-d65f-44ca-8939-67778fe8ef5a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.99:15021/healthz/ready\": dial tcp 10.134.0.99:15021: connect: connection refused" Nov 04 03:04:11.588301 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:04:11.588260 4405 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-00c846-6865789c84-c24hp" Nov 04 03:04:12.089625 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:04:12.089598 4405 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w2-57f116-openshift-default-75f4fc499-znglh" Nov 04 03:04:12.090101 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:04:12.090081 4405 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w2-57f116-openshift-default-75f4fc499-znglh" Nov 04 03:04:12.090897 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:04:12.090877 4405 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w2-57f116-openshift-default-75f4fc499-znglh" Nov 04 03:04:12.261084 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:04:12.261029 4405 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-00c846-6865789c84-c24hp" podStartSLOduration=4.261014511 podStartE2EDuration="4.261014511s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 03:04:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 03:04:10.719982818 +0000 UTC m=+1897.150682939" watchObservedRunningTime="2026-11-04 03:04:12.261014511 +0000 UTC m=+1898.691714677" Nov 04 03:04:18.311421 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:04:18.311376 4405 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w1-d2b8ee-56b894f96d-dbtft"] Nov 04 03:04:18.315239 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:04:18.315207 4405 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-d2b8ee-56b894f96d-dbtft" Nov 04 03:04:18.404698 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:04:18.398354 4405 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w1-d2b8ee-56b894f96d-dbtft"] Nov 04 03:04:18.426701 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:04:18.426647 4405 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w1-d2b8ee-dockercfg-685q6\"" Nov 04 03:04:18.426879 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:04:18.426804 4405 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w1-d2b8ee\"" Nov 04 03:04:18.444846 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:04:18.444796 4405 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/595ef150-1fae-478f-8974-0b0efc667be8-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w1-d2b8ee-56b894f96d-dbtft\" (UID: \"595ef150-1fae-478f-8974-0b0efc667be8\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w1-d2b8ee-56b894f96d-dbtft" Nov 04 03:04:18.445052 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:04:18.444925 4405 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qdg2g\" (UniqueName: \"kubernetes.io/projected/595ef150-1fae-478f-8974-0b0efc667be8-kube-api-access-qdg2g\") pod \"payload-processing-e2e-shared-b-w1-d2b8ee-56b894f96d-dbtft\" (UID: \"595ef150-1fae-478f-8974-0b0efc667be8\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w1-d2b8ee-56b894f96d-dbtft" Nov 04 03:04:18.547611 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:04:18.547562 4405 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/595ef150-1fae-478f-8974-0b0efc667be8-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w1-d2b8ee-56b894f96d-dbtft\" (UID: \"595ef150-1fae-478f-8974-0b0efc667be8\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w1-d2b8ee-56b894f96d-dbtft" Nov 04 03:04:18.547885 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:04:18.547712 4405 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qdg2g\" (UniqueName: \"kubernetes.io/projected/595ef150-1fae-478f-8974-0b0efc667be8-kube-api-access-qdg2g\") pod \"payload-processing-e2e-shared-b-w1-d2b8ee-56b894f96d-dbtft\" (UID: \"595ef150-1fae-478f-8974-0b0efc667be8\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w1-d2b8ee-56b894f96d-dbtft" Nov 04 03:04:18.548782 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:04:18.548751 4405 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/595ef150-1fae-478f-8974-0b0efc667be8-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w1-d2b8ee-56b894f96d-dbtft\" (UID: \"595ef150-1fae-478f-8974-0b0efc667be8\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w1-d2b8ee-56b894f96d-dbtft" Nov 04 03:04:18.609223 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:04:18.609188 4405 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qdg2g\" (UniqueName: \"kubernetes.io/projected/595ef150-1fae-478f-8974-0b0efc667be8-kube-api-access-qdg2g\") pod \"payload-processing-e2e-shared-b-w1-d2b8ee-56b894f96d-dbtft\" (UID: \"595ef150-1fae-478f-8974-0b0efc667be8\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w1-d2b8ee-56b894f96d-dbtft" Nov 04 03:04:18.645691 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:04:18.642956 4405 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-d2b8ee-56b894f96d-dbtft" Nov 04 03:04:19.060235 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:04:19.060204 4405 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w1-d2b8ee-56b894f96d-dbtft"] Nov 04 03:04:19.061418 ip-10-0-134-125 kubenswrapper[4405]: W1104 03:04:19.061391 4405 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod595ef150_1fae_478f_8974_0b0efc667be8.slice/crio-962418f2bc79ca6c614af94cf20064d0a02500034635449af92de3a7b78e990c WatchSource:0}: Error finding container 962418f2bc79ca6c614af94cf20064d0a02500034635449af92de3a7b78e990c: Status 404 returned error can't find the container with id 962418f2bc79ca6c614af94cf20064d0a02500034635449af92de3a7b78e990c Nov 04 03:04:19.656309 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:04:19.656261 4405 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-d2b8ee-56b894f96d-dbtft" event={"ID":"595ef150-1fae-478f-8974-0b0efc667be8","Type":"ContainerStarted","Data":"52c79ac172eee50bf33e12a0951aba93690dab4022df88924db189888deef78c"} Nov 04 03:04:19.656802 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:04:19.656378 4405 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-d2b8ee-56b894f96d-dbtft" Nov 04 03:04:19.656802 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:04:19.656398 4405 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-d2b8ee-56b894f96d-dbtft" event={"ID":"595ef150-1fae-478f-8974-0b0efc667be8","Type":"ContainerStarted","Data":"962418f2bc79ca6c614af94cf20064d0a02500034635449af92de3a7b78e990c"} Nov 04 03:04:19.821918 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:04:19.821860 4405 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-d2b8ee-56b894f96d-dbtft" podStartSLOduration=1.82184798 podStartE2EDuration="1.82184798s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 03:04:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 03:04:19.821144347 +0000 UTC m=+1906.251844468" watchObservedRunningTime="2026-11-04 03:04:19.82184798 +0000 UTC m=+1906.252548100" Nov 04 03:04:20.322713 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:04:20.322680 4405 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-bl5mg"] Nov 04 03:04:21.592588 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:04:21.592560 4405 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-00c846-5d7b447995-p8qhc" Nov 04 03:04:22.597890 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:04:22.597859 4405 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-00c846-6865789c84-c24hp" Nov 04 03:04:26.842374 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:04:26.842343 4405 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-a021b3-6c445c7c48-hpzbn"] Nov 04 03:04:26.846749 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:04:26.846649 4405 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-a021b3-6c445c7c48-hpzbn" Nov 04 03:04:26.902263 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:04:26.902012 4405 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-a-a021b3-serving-cert\"" Nov 04 03:04:26.955454 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:04:26.955410 4405 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-a021b3-6c445c7c48-hpzbn"] Nov 04 03:04:26.981229 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:04:26.981198 4405 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c2b4760c-84fe-4d11-a0d8-43e19f1b0a20-maas-api-tls\") pod \"maas-api-e2e-api-a-a021b3-6c445c7c48-hpzbn\" (UID: \"c2b4760c-84fe-4d11-a0d8-43e19f1b0a20\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-a021b3-6c445c7c48-hpzbn" Nov 04 03:04:26.981414 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:04:26.981382 4405 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c2b4760c-84fe-4d11-a0d8-43e19f1b0a20-metrics-certs\") pod \"maas-api-e2e-api-a-a021b3-6c445c7c48-hpzbn\" (UID: \"c2b4760c-84fe-4d11-a0d8-43e19f1b0a20\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-a021b3-6c445c7c48-hpzbn" Nov 04 03:04:26.981464 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:04:26.981421 4405 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6ndk5\" (UniqueName: \"kubernetes.io/projected/c2b4760c-84fe-4d11-a0d8-43e19f1b0a20-kube-api-access-6ndk5\") pod \"maas-api-e2e-api-a-a021b3-6c445c7c48-hpzbn\" (UID: \"c2b4760c-84fe-4d11-a0d8-43e19f1b0a20\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-a021b3-6c445c7c48-hpzbn" Nov 04 03:04:26.981529 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:04:26.981511 4405 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c2b4760c-84fe-4d11-a0d8-43e19f1b0a20-openshift-service-ca\") pod \"maas-api-e2e-api-a-a021b3-6c445c7c48-hpzbn\" (UID: \"c2b4760c-84fe-4d11-a0d8-43e19f1b0a20\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-a021b3-6c445c7c48-hpzbn" Nov 04 03:04:27.082760 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:04:27.082716 4405 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c2b4760c-84fe-4d11-a0d8-43e19f1b0a20-metrics-certs\") pod \"maas-api-e2e-api-a-a021b3-6c445c7c48-hpzbn\" (UID: \"c2b4760c-84fe-4d11-a0d8-43e19f1b0a20\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-a021b3-6c445c7c48-hpzbn" Nov 04 03:04:27.082760 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:04:27.082767 4405 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6ndk5\" (UniqueName: \"kubernetes.io/projected/c2b4760c-84fe-4d11-a0d8-43e19f1b0a20-kube-api-access-6ndk5\") pod \"maas-api-e2e-api-a-a021b3-6c445c7c48-hpzbn\" (UID: \"c2b4760c-84fe-4d11-a0d8-43e19f1b0a20\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-a021b3-6c445c7c48-hpzbn" Nov 04 03:04:27.083029 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:04:27.082807 4405 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c2b4760c-84fe-4d11-a0d8-43e19f1b0a20-openshift-service-ca\") pod \"maas-api-e2e-api-a-a021b3-6c445c7c48-hpzbn\" (UID: \"c2b4760c-84fe-4d11-a0d8-43e19f1b0a20\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-a021b3-6c445c7c48-hpzbn" Nov 04 03:04:27.083029 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:04:27.082978 4405 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c2b4760c-84fe-4d11-a0d8-43e19f1b0a20-maas-api-tls\") pod \"maas-api-e2e-api-a-a021b3-6c445c7c48-hpzbn\" (UID: \"c2b4760c-84fe-4d11-a0d8-43e19f1b0a20\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-a021b3-6c445c7c48-hpzbn" Nov 04 03:04:27.083138 ip-10-0-134-125 kubenswrapper[4405]: E1104 03:04:27.083102 4405 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-api-a-a021b3-serving-cert: secret "maas-api-e2e-api-a-a021b3-serving-cert" not found Nov 04 03:04:27.083193 ip-10-0-134-125 kubenswrapper[4405]: E1104 03:04:27.083180 4405 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c2b4760c-84fe-4d11-a0d8-43e19f1b0a20-maas-api-tls podName:c2b4760c-84fe-4d11-a0d8-43e19f1b0a20 nodeName:}" failed. No retries permitted until 2026-11-04 03:04:27.583162839 +0000 UTC m=+1914.013862959 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/c2b4760c-84fe-4d11-a0d8-43e19f1b0a20-maas-api-tls") pod "maas-api-e2e-api-a-a021b3-6c445c7c48-hpzbn" (UID: "c2b4760c-84fe-4d11-a0d8-43e19f1b0a20") : secret "maas-api-e2e-api-a-a021b3-serving-cert" not found Nov 04 03:04:27.083563 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:04:27.083538 4405 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c2b4760c-84fe-4d11-a0d8-43e19f1b0a20-openshift-service-ca\") pod \"maas-api-e2e-api-a-a021b3-6c445c7c48-hpzbn\" (UID: \"c2b4760c-84fe-4d11-a0d8-43e19f1b0a20\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-a021b3-6c445c7c48-hpzbn" Nov 04 03:04:27.085887 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:04:27.085867 4405 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c2b4760c-84fe-4d11-a0d8-43e19f1b0a20-metrics-certs\") pod \"maas-api-e2e-api-a-a021b3-6c445c7c48-hpzbn\" (UID: \"c2b4760c-84fe-4d11-a0d8-43e19f1b0a20\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-a021b3-6c445c7c48-hpzbn" Nov 04 03:04:27.158601 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:04:27.158508 4405 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6ndk5\" (UniqueName: \"kubernetes.io/projected/c2b4760c-84fe-4d11-a0d8-43e19f1b0a20-kube-api-access-6ndk5\") pod \"maas-api-e2e-api-a-a021b3-6c445c7c48-hpzbn\" (UID: \"c2b4760c-84fe-4d11-a0d8-43e19f1b0a20\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-a021b3-6c445c7c48-hpzbn" Nov 04 03:04:27.590651 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:04:27.590610 4405 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c2b4760c-84fe-4d11-a0d8-43e19f1b0a20-maas-api-tls\") pod \"maas-api-e2e-api-a-a021b3-6c445c7c48-hpzbn\" (UID: \"c2b4760c-84fe-4d11-a0d8-43e19f1b0a20\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-a021b3-6c445c7c48-hpzbn" Nov 04 03:04:27.593330 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:04:27.593302 4405 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c2b4760c-84fe-4d11-a0d8-43e19f1b0a20-maas-api-tls\") pod \"maas-api-e2e-api-a-a021b3-6c445c7c48-hpzbn\" (UID: \"c2b4760c-84fe-4d11-a0d8-43e19f1b0a20\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-a021b3-6c445c7c48-hpzbn" Nov 04 03:04:27.761638 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:04:27.761605 4405 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-a021b3-6c445c7c48-hpzbn" Nov 04 03:04:28.192690 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:04:28.188162 4405 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-a021b3-6c445c7c48-hpzbn"] Nov 04 03:04:28.739009 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:04:28.738972 4405 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-a021b3-6c445c7c48-hpzbn" event={"ID":"c2b4760c-84fe-4d11-a0d8-43e19f1b0a20","Type":"ContainerStarted","Data":"e59ba3a0dc8e7e86deb53427706f11a0d17110bca9f2f6ec4cfd560eaa70e425"} Nov 04 03:04:29.748571 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:04:29.748526 4405 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-a021b3-6c445c7c48-hpzbn" event={"ID":"c2b4760c-84fe-4d11-a0d8-43e19f1b0a20","Type":"ContainerStarted","Data":"f47101ea6881e097da1900bdb4896f976ac517fea6bddf415c842b0291a31316"} Nov 04 03:04:29.749036 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:04:29.748606 4405 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-a021b3-6c445c7c48-hpzbn" Nov 04 03:04:30.678121 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:04:30.678092 4405 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-d2b8ee-56b894f96d-dbtft" Nov 04 03:04:30.797638 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:04:30.797562 4405 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-a021b3-6c445c7c48-hpzbn" podStartSLOduration=3.837252424 podStartE2EDuration="4.797543927s" totalImagesPullingTime="960.291503ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 03:04:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 03:04:29.869859594 +0000 UTC m=+1916.300559715" watchObservedRunningTime="2026-11-04 03:04:30.797543927 +0000 UTC m=+1917.228244050" Nov 04 03:04:34.274027 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:04:34.273996 4405 scope.go:122] "RemoveContainer" containerID="37dfe03c937c3c23aad39a11e25a7dbca9e449ae2b01a890d6664f9bbc3e96f5" Nov 04 03:04:35.762359 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:04:35.762329 4405 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-a021b3-6c445c7c48-hpzbn" Nov 04 03:04:36.885339 ip-10-0-134-125 kubenswrapper[4405]: E1104 03:04:36.885298 4405 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5f2vb" containerName="extractor" Nov 04 03:05:06.886857 ip-10-0-134-125 kubenswrapper[4405]: E1104 03:05:06.886749 4405 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5f2vb" containerName="extractor" Nov 04 03:05:22.183930 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:05:22.183889 4405 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-bl5mg"] Nov 04 03:05:27.257153 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:05:27.257112 4405 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-9d2b96-7d8dd46f5b-7rbgg"] Nov 04 03:05:27.263081 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:05:27.263052 4405 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-9d2b96-7d8dd46f5b-7rbgg" Nov 04 03:05:27.344716 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:05:27.340251 4405 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-9d2b96-7d8dd46f5b-7rbgg"] Nov 04 03:05:27.348050 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:05:27.348001 4405 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/24a56d15-dba0-46bc-aca3-f6fabdf7c354-plugins-config-volume\") pod \"payload-processing-e2e-api-b-9d2b96-7d8dd46f5b-7rbgg\" (UID: \"24a56d15-dba0-46bc-aca3-f6fabdf7c354\") " pod="openshift-ingress/payload-processing-e2e-api-b-9d2b96-7d8dd46f5b-7rbgg" Nov 04 03:05:27.348588 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:05:27.348562 4405 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hmbw8\" (UniqueName: \"kubernetes.io/projected/24a56d15-dba0-46bc-aca3-f6fabdf7c354-kube-api-access-hmbw8\") pod \"payload-processing-e2e-api-b-9d2b96-7d8dd46f5b-7rbgg\" (UID: \"24a56d15-dba0-46bc-aca3-f6fabdf7c354\") " pod="openshift-ingress/payload-processing-e2e-api-b-9d2b96-7d8dd46f5b-7rbgg" Nov 04 03:05:27.359173 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:05:27.359144 4405 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-b-9d2b96\"" Nov 04 03:05:27.359324 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:05:27.359203 4405 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-b-9d2b96-dockercfg-2w27p\"" Nov 04 03:05:27.451145 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:05:27.450340 4405 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hmbw8\" (UniqueName: \"kubernetes.io/projected/24a56d15-dba0-46bc-aca3-f6fabdf7c354-kube-api-access-hmbw8\") pod \"payload-processing-e2e-api-b-9d2b96-7d8dd46f5b-7rbgg\" (UID: \"24a56d15-dba0-46bc-aca3-f6fabdf7c354\") " pod="openshift-ingress/payload-processing-e2e-api-b-9d2b96-7d8dd46f5b-7rbgg" Nov 04 03:05:27.451145 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:05:27.450519 4405 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/24a56d15-dba0-46bc-aca3-f6fabdf7c354-plugins-config-volume\") pod \"payload-processing-e2e-api-b-9d2b96-7d8dd46f5b-7rbgg\" (UID: \"24a56d15-dba0-46bc-aca3-f6fabdf7c354\") " pod="openshift-ingress/payload-processing-e2e-api-b-9d2b96-7d8dd46f5b-7rbgg" Nov 04 03:05:27.451473 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:05:27.451453 4405 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/24a56d15-dba0-46bc-aca3-f6fabdf7c354-plugins-config-volume\") pod \"payload-processing-e2e-api-b-9d2b96-7d8dd46f5b-7rbgg\" (UID: \"24a56d15-dba0-46bc-aca3-f6fabdf7c354\") " pod="openshift-ingress/payload-processing-e2e-api-b-9d2b96-7d8dd46f5b-7rbgg" Nov 04 03:05:27.511010 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:05:27.510937 4405 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hmbw8\" (UniqueName: \"kubernetes.io/projected/24a56d15-dba0-46bc-aca3-f6fabdf7c354-kube-api-access-hmbw8\") pod \"payload-processing-e2e-api-b-9d2b96-7d8dd46f5b-7rbgg\" (UID: \"24a56d15-dba0-46bc-aca3-f6fabdf7c354\") " pod="openshift-ingress/payload-processing-e2e-api-b-9d2b96-7d8dd46f5b-7rbgg" Nov 04 03:05:27.581159 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:05:27.581129 4405 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-9d2b96-7d8dd46f5b-7rbgg" Nov 04 03:05:27.961588 ip-10-0-134-125 kubenswrapper[4405]: W1104 03:05:27.961553 4405 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod24a56d15_dba0_46bc_aca3_f6fabdf7c354.slice/crio-93927ced226f2285e3d490d2a8c9cf79d121fbbf6ee31245235760129d6ba624 WatchSource:0}: Error finding container 93927ced226f2285e3d490d2a8c9cf79d121fbbf6ee31245235760129d6ba624: Status 404 returned error can't find the container with id 93927ced226f2285e3d490d2a8c9cf79d121fbbf6ee31245235760129d6ba624 Nov 04 03:05:27.961808 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:05:27.961744 4405 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-9d2b96-7d8dd46f5b-7rbgg"] Nov 04 03:05:28.265834 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:05:28.265740 4405 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-9d2b96-7d8dd46f5b-7rbgg" event={"ID":"24a56d15-dba0-46bc-aca3-f6fabdf7c354","Type":"ContainerStarted","Data":"b7442d178bb44a6ff44f2c82bb2146204a34f48330ffe0999fb2aac2ce7cd281"} Nov 04 03:05:28.265834 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:05:28.265802 4405 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-9d2b96-7d8dd46f5b-7rbgg" event={"ID":"24a56d15-dba0-46bc-aca3-f6fabdf7c354","Type":"ContainerStarted","Data":"93927ced226f2285e3d490d2a8c9cf79d121fbbf6ee31245235760129d6ba624"} Nov 04 03:05:28.266256 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:05:28.265843 4405 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-b-9d2b96-7d8dd46f5b-7rbgg" Nov 04 03:05:28.387565 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:05:28.387508 4405 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-b-9d2b96-7d8dd46f5b-7rbgg" podStartSLOduration=1.387494392 podStartE2EDuration="1.387494392s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 03:05:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 03:05:28.382296439 +0000 UTC m=+1974.812996561" watchObservedRunningTime="2026-11-04 03:05:28.387494392 +0000 UTC m=+1974.818194511" Nov 04 03:05:36.896285 ip-10-0-134-125 kubenswrapper[4405]: E1104 03:05:36.896236 4405 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5f2vb" containerName="extractor" Nov 04 03:05:39.279054 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:05:39.278558 4405 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-b-9d2b96-7d8dd46f5b-7rbgg" Nov 04 03:05:43.940061 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:05:43.940015 4405 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-d2b8ee-6ffd864944-9fgxg"] Nov 04 03:05:43.944886 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:05:43.944860 4405 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-d2b8ee-6ffd864944-9fgxg" Nov 04 03:05:44.024013 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:05:44.023970 4405 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/541d536e-5ac3-4f98-9f6b-a827a5ef340a-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w1-d2b8ee-6ffd864944-9fgxg\" (UID: \"541d536e-5ac3-4f98-9f6b-a827a5ef340a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-d2b8ee-6ffd864944-9fgxg" Nov 04 03:05:44.024202 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:05:44.024047 4405 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/541d536e-5ac3-4f98-9f6b-a827a5ef340a-maas-api-tls\") pod \"maas-api-e2e-shared-b-w1-d2b8ee-6ffd864944-9fgxg\" (UID: \"541d536e-5ac3-4f98-9f6b-a827a5ef340a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-d2b8ee-6ffd864944-9fgxg" Nov 04 03:05:44.024202 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:05:44.024092 4405 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/541d536e-5ac3-4f98-9f6b-a827a5ef340a-metrics-certs\") pod \"maas-api-e2e-shared-b-w1-d2b8ee-6ffd864944-9fgxg\" (UID: \"541d536e-5ac3-4f98-9f6b-a827a5ef340a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-d2b8ee-6ffd864944-9fgxg" Nov 04 03:05:44.024202 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:05:44.024179 4405 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gphgn\" (UniqueName: \"kubernetes.io/projected/541d536e-5ac3-4f98-9f6b-a827a5ef340a-kube-api-access-gphgn\") pod \"maas-api-e2e-shared-b-w1-d2b8ee-6ffd864944-9fgxg\" (UID: \"541d536e-5ac3-4f98-9f6b-a827a5ef340a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-d2b8ee-6ffd864944-9fgxg" Nov 04 03:05:44.048411 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:05:44.048376 4405 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-w1-d2b8ee-serving-cert\"" Nov 04 03:05:44.055154 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:05:44.055121 4405 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-d2b8ee-6ffd864944-9fgxg"] Nov 04 03:05:44.125566 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:05:44.125535 4405 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/541d536e-5ac3-4f98-9f6b-a827a5ef340a-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w1-d2b8ee-6ffd864944-9fgxg\" (UID: \"541d536e-5ac3-4f98-9f6b-a827a5ef340a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-d2b8ee-6ffd864944-9fgxg" Nov 04 03:05:44.125803 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:05:44.125586 4405 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/541d536e-5ac3-4f98-9f6b-a827a5ef340a-maas-api-tls\") pod \"maas-api-e2e-shared-b-w1-d2b8ee-6ffd864944-9fgxg\" (UID: \"541d536e-5ac3-4f98-9f6b-a827a5ef340a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-d2b8ee-6ffd864944-9fgxg" Nov 04 03:05:44.125803 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:05:44.125613 4405 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/541d536e-5ac3-4f98-9f6b-a827a5ef340a-metrics-certs\") pod \"maas-api-e2e-shared-b-w1-d2b8ee-6ffd864944-9fgxg\" (UID: \"541d536e-5ac3-4f98-9f6b-a827a5ef340a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-d2b8ee-6ffd864944-9fgxg" Nov 04 03:05:44.125803 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:05:44.125708 4405 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gphgn\" (UniqueName: \"kubernetes.io/projected/541d536e-5ac3-4f98-9f6b-a827a5ef340a-kube-api-access-gphgn\") pod \"maas-api-e2e-shared-b-w1-d2b8ee-6ffd864944-9fgxg\" (UID: \"541d536e-5ac3-4f98-9f6b-a827a5ef340a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-d2b8ee-6ffd864944-9fgxg" Nov 04 03:05:44.125803 ip-10-0-134-125 kubenswrapper[4405]: E1104 03:05:44.125770 4405 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-d2b8ee-serving-cert: secret "maas-api-e2e-shared-b-w1-d2b8ee-serving-cert" not found Nov 04 03:05:44.126101 ip-10-0-134-125 kubenswrapper[4405]: E1104 03:05:44.125838 4405 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/541d536e-5ac3-4f98-9f6b-a827a5ef340a-maas-api-tls podName:541d536e-5ac3-4f98-9f6b-a827a5ef340a nodeName:}" failed. No retries permitted until 2026-11-04 03:05:44.625819566 +0000 UTC m=+1991.056519681 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/541d536e-5ac3-4f98-9f6b-a827a5ef340a-maas-api-tls") pod "maas-api-e2e-shared-b-w1-d2b8ee-6ffd864944-9fgxg" (UID: "541d536e-5ac3-4f98-9f6b-a827a5ef340a") : secret "maas-api-e2e-shared-b-w1-d2b8ee-serving-cert" not found Nov 04 03:05:44.126364 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:05:44.126339 4405 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/541d536e-5ac3-4f98-9f6b-a827a5ef340a-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w1-d2b8ee-6ffd864944-9fgxg\" (UID: \"541d536e-5ac3-4f98-9f6b-a827a5ef340a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-d2b8ee-6ffd864944-9fgxg" Nov 04 03:05:44.128309 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:05:44.128290 4405 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/541d536e-5ac3-4f98-9f6b-a827a5ef340a-metrics-certs\") pod \"maas-api-e2e-shared-b-w1-d2b8ee-6ffd864944-9fgxg\" (UID: \"541d536e-5ac3-4f98-9f6b-a827a5ef340a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-d2b8ee-6ffd864944-9fgxg" Nov 04 03:05:44.210893 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:05:44.210808 4405 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gphgn\" (UniqueName: \"kubernetes.io/projected/541d536e-5ac3-4f98-9f6b-a827a5ef340a-kube-api-access-gphgn\") pod \"maas-api-e2e-shared-b-w1-d2b8ee-6ffd864944-9fgxg\" (UID: \"541d536e-5ac3-4f98-9f6b-a827a5ef340a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-d2b8ee-6ffd864944-9fgxg" Nov 04 03:05:44.639025 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:05:44.638988 4405 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/541d536e-5ac3-4f98-9f6b-a827a5ef340a-maas-api-tls\") pod \"maas-api-e2e-shared-b-w1-d2b8ee-6ffd864944-9fgxg\" (UID: \"541d536e-5ac3-4f98-9f6b-a827a5ef340a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-d2b8ee-6ffd864944-9fgxg" Nov 04 03:05:44.641571 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:05:44.641544 4405 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/541d536e-5ac3-4f98-9f6b-a827a5ef340a-maas-api-tls\") pod \"maas-api-e2e-shared-b-w1-d2b8ee-6ffd864944-9fgxg\" (UID: \"541d536e-5ac3-4f98-9f6b-a827a5ef340a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-d2b8ee-6ffd864944-9fgxg" Nov 04 03:05:44.859928 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:05:44.859891 4405 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-d2b8ee-6ffd864944-9fgxg" Nov 04 03:05:45.246850 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:05:45.246819 4405 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-d2b8ee-6ffd864944-9fgxg"] Nov 04 03:05:45.247263 ip-10-0-134-125 kubenswrapper[4405]: W1104 03:05:45.247190 4405 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod541d536e_5ac3_4f98_9f6b_a827a5ef340a.slice/crio-8aff5795d4a2cc8c7e8a7570cd20c3d122e5f6f260f8145e042a434fa4aa6c8a WatchSource:0}: Error finding container 8aff5795d4a2cc8c7e8a7570cd20c3d122e5f6f260f8145e042a434fa4aa6c8a: Status 404 returned error can't find the container with id 8aff5795d4a2cc8c7e8a7570cd20c3d122e5f6f260f8145e042a434fa4aa6c8a Nov 04 03:05:45.412714 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:05:45.412655 4405 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-d2b8ee-6ffd864944-9fgxg" event={"ID":"541d536e-5ac3-4f98-9f6b-a827a5ef340a","Type":"ContainerStarted","Data":"8aff5795d4a2cc8c7e8a7570cd20c3d122e5f6f260f8145e042a434fa4aa6c8a"} Nov 04 03:05:46.421825 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:05:46.421725 4405 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-d2b8ee-6ffd864944-9fgxg" event={"ID":"541d536e-5ac3-4f98-9f6b-a827a5ef340a","Type":"ContainerStarted","Data":"e80ef986f4158c0dcab4283cfdfcc6f2dc2314c4c99e1619955564af2eda604b"} Nov 04 03:05:46.422298 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:05:46.421828 4405 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-d2b8ee-6ffd864944-9fgxg" Nov 04 03:05:46.542120 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:05:46.542063 4405 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-d2b8ee-6ffd864944-9fgxg" podStartSLOduration=2.664629218 podStartE2EDuration="3.542048182s" totalImagesPullingTime="877.418964ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 03:05:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 03:05:46.537992087 +0000 UTC m=+1992.968692207" watchObservedRunningTime="2026-11-04 03:05:46.542048182 +0000 UTC m=+1992.972748301" Nov 04 03:05:52.435592 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:05:52.435555 4405 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-d2b8ee-6ffd864944-9fgxg" Nov 04 03:06:06.886735 ip-10-0-134-125 kubenswrapper[4405]: E1104 03:06:06.886690 4405 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5f2vb" containerName="extractor" Nov 04 03:06:08.352021 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:06:08.351983 4405 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-a021b3-6c445c7c48-hpzbn"] Nov 04 03:06:08.352485 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:06:08.352196 4405 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-a021b3-6c445c7c48-hpzbn" podUID="c2b4760c-84fe-4d11-a0d8-43e19f1b0a20" containerName="maas-api" containerID="cri-o://f47101ea6881e097da1900bdb4896f976ac517fea6bddf415c842b0291a31316" gracePeriod=30 Nov 04 03:06:08.630768 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:06:08.630603 4405 generic.go:356] "Generic (PLEG): container finished" podID="c2b4760c-84fe-4d11-a0d8-43e19f1b0a20" containerID="f47101ea6881e097da1900bdb4896f976ac517fea6bddf415c842b0291a31316" exitCode=0 Nov 04 03:06:08.630988 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:06:08.630933 4405 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-a021b3-6c445c7c48-hpzbn" event={"ID":"c2b4760c-84fe-4d11-a0d8-43e19f1b0a20","Type":"ContainerDied","Data":"f47101ea6881e097da1900bdb4896f976ac517fea6bddf415c842b0291a31316"} Nov 04 03:06:08.676112 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:06:08.676045 4405 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-a021b3-6c445c7c48-hpzbn" Nov 04 03:06:08.707765 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:06:08.707735 4405 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c2b4760c-84fe-4d11-a0d8-43e19f1b0a20-kube-api-access-6ndk5\" (UniqueName: \"kubernetes.io/projected/c2b4760c-84fe-4d11-a0d8-43e19f1b0a20-kube-api-access-6ndk5\") pod \"c2b4760c-84fe-4d11-a0d8-43e19f1b0a20\" (UID: \"c2b4760c-84fe-4d11-a0d8-43e19f1b0a20\") " Nov 04 03:06:08.707972 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:06:08.707804 4405 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c2b4760c-84fe-4d11-a0d8-43e19f1b0a20-metrics-certs\" (UniqueName: \"kubernetes.io/secret/c2b4760c-84fe-4d11-a0d8-43e19f1b0a20-metrics-certs\") pod \"c2b4760c-84fe-4d11-a0d8-43e19f1b0a20\" (UID: \"c2b4760c-84fe-4d11-a0d8-43e19f1b0a20\") " Nov 04 03:06:08.708060 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:06:08.708025 4405 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c2b4760c-84fe-4d11-a0d8-43e19f1b0a20-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c2b4760c-84fe-4d11-a0d8-43e19f1b0a20-openshift-service-ca\") pod \"c2b4760c-84fe-4d11-a0d8-43e19f1b0a20\" (UID: \"c2b4760c-84fe-4d11-a0d8-43e19f1b0a20\") " Nov 04 03:06:08.708123 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:06:08.708097 4405 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c2b4760c-84fe-4d11-a0d8-43e19f1b0a20-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c2b4760c-84fe-4d11-a0d8-43e19f1b0a20-maas-api-tls\") pod \"c2b4760c-84fe-4d11-a0d8-43e19f1b0a20\" (UID: \"c2b4760c-84fe-4d11-a0d8-43e19f1b0a20\") " Nov 04 03:06:08.708896 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:06:08.708868 4405 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c2b4760c-84fe-4d11-a0d8-43e19f1b0a20-openshift-service-ca" pod "c2b4760c-84fe-4d11-a0d8-43e19f1b0a20" (UID: "c2b4760c-84fe-4d11-a0d8-43e19f1b0a20"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 03:06:08.710373 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:06:08.710344 4405 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c2b4760c-84fe-4d11-a0d8-43e19f1b0a20-kube-api-access-6ndk5" pod "c2b4760c-84fe-4d11-a0d8-43e19f1b0a20" (UID: "c2b4760c-84fe-4d11-a0d8-43e19f1b0a20"). InnerVolumeSpecName "kube-api-access-6ndk5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 03:06:08.710869 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:06:08.710846 4405 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c2b4760c-84fe-4d11-a0d8-43e19f1b0a20-metrics-certs" pod "c2b4760c-84fe-4d11-a0d8-43e19f1b0a20" (UID: "c2b4760c-84fe-4d11-a0d8-43e19f1b0a20"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 04 03:06:08.711403 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:06:08.711380 4405 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c2b4760c-84fe-4d11-a0d8-43e19f1b0a20-maas-api-tls" pod "c2b4760c-84fe-4d11-a0d8-43e19f1b0a20" (UID: "c2b4760c-84fe-4d11-a0d8-43e19f1b0a20"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 04 03:06:08.815960 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:06:08.815867 4405 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6ndk5\" (UniqueName: \"kubernetes.io/projected/c2b4760c-84fe-4d11-a0d8-43e19f1b0a20-kube-api-access-6ndk5\") on node \"ip-10-0-134-125.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:06:08.815960 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:06:08.815902 4405 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c2b4760c-84fe-4d11-a0d8-43e19f1b0a20-metrics-certs\") on node \"ip-10-0-134-125.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:06:08.815960 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:06:08.815917 4405 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c2b4760c-84fe-4d11-a0d8-43e19f1b0a20-openshift-service-ca\") on node \"ip-10-0-134-125.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:06:08.815960 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:06:08.815934 4405 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c2b4760c-84fe-4d11-a0d8-43e19f1b0a20-maas-api-tls\") on node \"ip-10-0-134-125.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:06:08.829856 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:06:08.828638 4405 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-a021b3-6556664496-wmwrg"] Nov 04 03:06:08.894456 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:06:08.894372 4405 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-a021b3-775855fc55-hd92k"] Nov 04 03:06:09.640943 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:06:09.640913 4405 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-a021b3-6c445c7c48-hpzbn" Nov 04 03:06:09.641436 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:06:09.640904 4405 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-a021b3-6c445c7c48-hpzbn" event={"ID":"c2b4760c-84fe-4d11-a0d8-43e19f1b0a20","Type":"ContainerDied","Data":"e59ba3a0dc8e7e86deb53427706f11a0d17110bca9f2f6ec4cfd560eaa70e425"} Nov 04 03:06:09.641436 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:06:09.641114 4405 scope.go:122] "RemoveContainer" containerID="f47101ea6881e097da1900bdb4896f976ac517fea6bddf415c842b0291a31316" Nov 04 03:06:09.922960 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:06:09.922862 4405 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-a021b3-6c445c7c48-hpzbn"] Nov 04 03:06:09.927340 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:06:09.927312 4405 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-a021b3-6c445c7c48-hpzbn"] Nov 04 03:06:11.702753 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:06:11.702703 4405 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c2b4760c-84fe-4d11-a0d8-43e19f1b0a20" path="/var/lib/kubelet/pods/c2b4760c-84fe-4d11-a0d8-43e19f1b0a20/volumes" Nov 04 03:06:13.284023 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:06:13.283986 4405 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-bl5mg"] Nov 04 03:06:13.895013 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:06:13.894972 4405 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-a-a021b3-775855fc55-hd92k" podUID="8e29627f-f096-4fe6-b5a9-d66ed0631172" containerName="payload-pre-processing" containerID="cri-o://0b819818c87e4fb3a0d65c83a4390740669fd23e2809aaf65b2e0358badb0014" gracePeriod=25 Nov 04 03:06:14.202459 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:06:14.202431 4405 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-a021b3-775855fc55-hd92k" Nov 04 03:06:14.334283 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:06:14.334245 4405 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8e29627f-f096-4fe6-b5a9-d66ed0631172-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8e29627f-f096-4fe6-b5a9-d66ed0631172-plugins-config-volume\") pod \"8e29627f-f096-4fe6-b5a9-d66ed0631172\" (UID: \"8e29627f-f096-4fe6-b5a9-d66ed0631172\") " Nov 04 03:06:14.334828 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:06:14.334307 4405 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8e29627f-f096-4fe6-b5a9-d66ed0631172-kube-api-access-x68k9\" (UniqueName: \"kubernetes.io/projected/8e29627f-f096-4fe6-b5a9-d66ed0631172-kube-api-access-x68k9\") pod \"8e29627f-f096-4fe6-b5a9-d66ed0631172\" (UID: \"8e29627f-f096-4fe6-b5a9-d66ed0631172\") " Nov 04 03:06:14.334828 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:06:14.334651 4405 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8e29627f-f096-4fe6-b5a9-d66ed0631172-plugins-config-volume" pod "8e29627f-f096-4fe6-b5a9-d66ed0631172" (UID: "8e29627f-f096-4fe6-b5a9-d66ed0631172"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 03:06:14.335124 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:06:14.335091 4405 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8e29627f-f096-4fe6-b5a9-d66ed0631172-plugins-config-volume\") on node \"ip-10-0-134-125.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:06:14.336939 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:06:14.336901 4405 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8e29627f-f096-4fe6-b5a9-d66ed0631172-kube-api-access-x68k9" pod "8e29627f-f096-4fe6-b5a9-d66ed0631172" (UID: "8e29627f-f096-4fe6-b5a9-d66ed0631172"). InnerVolumeSpecName "kube-api-access-x68k9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 03:06:14.437146 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:06:14.437041 4405 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x68k9\" (UniqueName: \"kubernetes.io/projected/8e29627f-f096-4fe6-b5a9-d66ed0631172-kube-api-access-x68k9\") on node \"ip-10-0-134-125.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:06:14.641785 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:06:14.641739 4405 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-a-a021b3-6556664496-wmwrg" podUID="596af222-bf6a-44fe-9e2b-5ced60a615c0" containerName="payload-processing" containerID="cri-o://2db06180baa75ad13abc50a8deed840c02d6079bec73f5d17bcd2ddf80eed2c2" gracePeriod=25 Nov 04 03:06:14.679921 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:06:14.679882 4405 generic.go:356] "Generic (PLEG): container finished" podID="8e29627f-f096-4fe6-b5a9-d66ed0631172" containerID="0b819818c87e4fb3a0d65c83a4390740669fd23e2809aaf65b2e0358badb0014" exitCode=0 Nov 04 03:06:14.680114 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:06:14.679934 4405 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-a021b3-775855fc55-hd92k" Nov 04 03:06:14.680114 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:06:14.679970 4405 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-a021b3-775855fc55-hd92k" event={"ID":"8e29627f-f096-4fe6-b5a9-d66ed0631172","Type":"ContainerDied","Data":"0b819818c87e4fb3a0d65c83a4390740669fd23e2809aaf65b2e0358badb0014"} Nov 04 03:06:14.680114 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:06:14.680069 4405 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-a021b3-775855fc55-hd92k" event={"ID":"8e29627f-f096-4fe6-b5a9-d66ed0631172","Type":"ContainerDied","Data":"1e99d697f83beeb7e52b68d95d9c6c9ca0ab48f670c22dce67abcc33fc7a4f97"} Nov 04 03:06:14.680114 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:06:14.680101 4405 scope.go:122] "RemoveContainer" containerID="0b819818c87e4fb3a0d65c83a4390740669fd23e2809aaf65b2e0358badb0014" Nov 04 03:06:14.760595 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:06:14.760448 4405 scope.go:122] "RemoveContainer" containerID="0b819818c87e4fb3a0d65c83a4390740669fd23e2809aaf65b2e0358badb0014" Nov 04 03:06:14.762385 ip-10-0-134-125 kubenswrapper[4405]: E1104 03:06:14.761165 4405 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0b819818c87e4fb3a0d65c83a4390740669fd23e2809aaf65b2e0358badb0014\": container with ID starting with 0b819818c87e4fb3a0d65c83a4390740669fd23e2809aaf65b2e0358badb0014 not found: ID does not exist" containerID="0b819818c87e4fb3a0d65c83a4390740669fd23e2809aaf65b2e0358badb0014" Nov 04 03:06:14.762385 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:06:14.761216 4405 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0b819818c87e4fb3a0d65c83a4390740669fd23e2809aaf65b2e0358badb0014"} err="failed to get container status \"0b819818c87e4fb3a0d65c83a4390740669fd23e2809aaf65b2e0358badb0014\": rpc error: code = NotFound desc = could not find container \"0b819818c87e4fb3a0d65c83a4390740669fd23e2809aaf65b2e0358badb0014\": container with ID starting with 0b819818c87e4fb3a0d65c83a4390740669fd23e2809aaf65b2e0358badb0014 not found: ID does not exist" Nov 04 03:06:14.901738 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:06:14.901660 4405 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-a021b3-775855fc55-hd92k"] Nov 04 03:06:14.907786 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:06:14.907747 4405 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-a021b3-775855fc55-hd92k"] Nov 04 03:06:15.078692 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:06:15.078042 4405 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-a021b3-6556664496-wmwrg" Nov 04 03:06:15.208554 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:06:15.208452 4405 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/596af222-bf6a-44fe-9e2b-5ced60a615c0-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/596af222-bf6a-44fe-9e2b-5ced60a615c0-plugins-config-volume\") pod \"596af222-bf6a-44fe-9e2b-5ced60a615c0\" (UID: \"596af222-bf6a-44fe-9e2b-5ced60a615c0\") " Nov 04 03:06:15.208748 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:06:15.208586 4405 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/596af222-bf6a-44fe-9e2b-5ced60a615c0-kube-api-access-hzpcj\" (UniqueName: \"kubernetes.io/projected/596af222-bf6a-44fe-9e2b-5ced60a615c0-kube-api-access-hzpcj\") pod \"596af222-bf6a-44fe-9e2b-5ced60a615c0\" (UID: \"596af222-bf6a-44fe-9e2b-5ced60a615c0\") " Nov 04 03:06:15.208833 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:06:15.208762 4405 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/596af222-bf6a-44fe-9e2b-5ced60a615c0-plugins-config-volume" pod "596af222-bf6a-44fe-9e2b-5ced60a615c0" (UID: "596af222-bf6a-44fe-9e2b-5ced60a615c0"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 03:06:15.209536 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:06:15.209506 4405 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/596af222-bf6a-44fe-9e2b-5ced60a615c0-plugins-config-volume\") on node \"ip-10-0-134-125.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:06:15.211076 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:06:15.211047 4405 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/596af222-bf6a-44fe-9e2b-5ced60a615c0-kube-api-access-hzpcj" pod "596af222-bf6a-44fe-9e2b-5ced60a615c0" (UID: "596af222-bf6a-44fe-9e2b-5ced60a615c0"). InnerVolumeSpecName "kube-api-access-hzpcj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 03:06:15.311096 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:06:15.310995 4405 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hzpcj\" (UniqueName: \"kubernetes.io/projected/596af222-bf6a-44fe-9e2b-5ced60a615c0-kube-api-access-hzpcj\") on node \"ip-10-0-134-125.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:06:15.688377 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:06:15.688340 4405 generic.go:356] "Generic (PLEG): container finished" podID="596af222-bf6a-44fe-9e2b-5ced60a615c0" containerID="2db06180baa75ad13abc50a8deed840c02d6079bec73f5d17bcd2ddf80eed2c2" exitCode=0 Nov 04 03:06:15.688977 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:06:15.688390 4405 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-a021b3-6556664496-wmwrg" Nov 04 03:06:15.688977 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:06:15.688418 4405 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-a021b3-6556664496-wmwrg" event={"ID":"596af222-bf6a-44fe-9e2b-5ced60a615c0","Type":"ContainerDied","Data":"2db06180baa75ad13abc50a8deed840c02d6079bec73f5d17bcd2ddf80eed2c2"} Nov 04 03:06:15.688977 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:06:15.688473 4405 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-a021b3-6556664496-wmwrg" event={"ID":"596af222-bf6a-44fe-9e2b-5ced60a615c0","Type":"ContainerDied","Data":"6e047f069d0d39bdedb8d01bc8d732b52313feeb0194a666eb30e457e48034d5"} Nov 04 03:06:15.688977 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:06:15.688498 4405 scope.go:122] "RemoveContainer" containerID="2db06180baa75ad13abc50a8deed840c02d6079bec73f5d17bcd2ddf80eed2c2" Nov 04 03:06:15.701916 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:06:15.701887 4405 scope.go:122] "RemoveContainer" containerID="2db06180baa75ad13abc50a8deed840c02d6079bec73f5d17bcd2ddf80eed2c2" Nov 04 03:06:15.702389 ip-10-0-134-125 kubenswrapper[4405]: E1104 03:06:15.702354 4405 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2db06180baa75ad13abc50a8deed840c02d6079bec73f5d17bcd2ddf80eed2c2\": container with ID starting with 2db06180baa75ad13abc50a8deed840c02d6079bec73f5d17bcd2ddf80eed2c2 not found: ID does not exist" containerID="2db06180baa75ad13abc50a8deed840c02d6079bec73f5d17bcd2ddf80eed2c2" Nov 04 03:06:15.702554 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:06:15.702398 4405 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2db06180baa75ad13abc50a8deed840c02d6079bec73f5d17bcd2ddf80eed2c2"} err="failed to get container status \"2db06180baa75ad13abc50a8deed840c02d6079bec73f5d17bcd2ddf80eed2c2\": rpc error: code = NotFound desc = could not find container \"2db06180baa75ad13abc50a8deed840c02d6079bec73f5d17bcd2ddf80eed2c2\": container with ID starting with 2db06180baa75ad13abc50a8deed840c02d6079bec73f5d17bcd2ddf80eed2c2 not found: ID does not exist" Nov 04 03:06:15.706201 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:06:15.706172 4405 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8e29627f-f096-4fe6-b5a9-d66ed0631172" path="/var/lib/kubelet/pods/8e29627f-f096-4fe6-b5a9-d66ed0631172/volumes" Nov 04 03:06:15.853922 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:06:15.853888 4405 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-a021b3-6556664496-wmwrg"] Nov 04 03:06:15.857266 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:06:15.857246 4405 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-a021b3-6556664496-wmwrg"] Nov 04 03:06:17.702881 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:06:17.702847 4405 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="596af222-bf6a-44fe-9e2b-5ced60a615c0" path="/var/lib/kubelet/pods/596af222-bf6a-44fe-9e2b-5ced60a615c0/volumes" Nov 04 03:06:20.756118 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:06:20.756086 4405 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w1-d2b8ee-openshift-default-6f9c87f479-qpj8z"] Nov 04 03:06:20.756843 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:06:20.756825 4405 cpu_manager.go:414] "containerMap: removing container" podUID="596af222-bf6a-44fe-9e2b-5ced60a615c0" containerName="payload-processing" Nov 04 03:06:20.756900 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:06:20.756845 4405 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="596af222-bf6a-44fe-9e2b-5ced60a615c0" containerName="payload-processing" Nov 04 03:06:20.756900 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:06:20.756865 4405 cpu_manager.go:414] "containerMap: removing container" podUID="c2b4760c-84fe-4d11-a0d8-43e19f1b0a20" containerName="maas-api" Nov 04 03:06:20.756900 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:06:20.756870 4405 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c2b4760c-84fe-4d11-a0d8-43e19f1b0a20" containerName="maas-api" Nov 04 03:06:20.756900 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:06:20.756884 4405 cpu_manager.go:414] "containerMap: removing container" podUID="8e29627f-f096-4fe6-b5a9-d66ed0631172" containerName="payload-pre-processing" Nov 04 03:06:20.756900 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:06:20.756889 4405 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8e29627f-f096-4fe6-b5a9-d66ed0631172" containerName="payload-pre-processing" Nov 04 03:06:20.757057 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:06:20.757005 4405 memory_manager.go:365] "RemoveStaleState removing state" podUID="596af222-bf6a-44fe-9e2b-5ced60a615c0" containerName="payload-processing" Nov 04 03:06:20.757057 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:06:20.757015 4405 memory_manager.go:365] "RemoveStaleState removing state" podUID="c2b4760c-84fe-4d11-a0d8-43e19f1b0a20" containerName="maas-api" Nov 04 03:06:20.757057 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:06:20.757022 4405 memory_manager.go:365] "RemoveStaleState removing state" podUID="8e29627f-f096-4fe6-b5a9-d66ed0631172" containerName="payload-pre-processing" Nov 04 03:06:20.761967 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:06:20.761948 4405 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w1-d2b8ee-openshift-default-6f9c87f479-qpj8z" Nov 04 03:06:20.814007 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:06:20.813973 4405 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w1-d2b8ee-openshift-default-dockercfg-bd27j\"" Nov 04 03:06:20.832584 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:06:20.832552 4405 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/05f6d23b-ab70-4e62-9065-41f075c26ab6-istio-data\") pod \"e2e-shared-b-w1-d2b8ee-openshift-default-6f9c87f479-qpj8z\" (UID: \"05f6d23b-ab70-4e62-9065-41f075c26ab6\") " pod="openshift-ingress/e2e-shared-b-w1-d2b8ee-openshift-default-6f9c87f479-qpj8z" Nov 04 03:06:20.832758 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:06:20.832602 4405 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/05f6d23b-ab70-4e62-9065-41f075c26ab6-workload-socket\") pod \"e2e-shared-b-w1-d2b8ee-openshift-default-6f9c87f479-qpj8z\" (UID: \"05f6d23b-ab70-4e62-9065-41f075c26ab6\") " pod="openshift-ingress/e2e-shared-b-w1-d2b8ee-openshift-default-6f9c87f479-qpj8z" Nov 04 03:06:20.832758 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:06:20.832702 4405 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/05f6d23b-ab70-4e62-9065-41f075c26ab6-workload-certs\") pod \"e2e-shared-b-w1-d2b8ee-openshift-default-6f9c87f479-qpj8z\" (UID: \"05f6d23b-ab70-4e62-9065-41f075c26ab6\") " pod="openshift-ingress/e2e-shared-b-w1-d2b8ee-openshift-default-6f9c87f479-qpj8z" Nov 04 03:06:20.832758 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:06:20.832742 4405 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q6qv9\" (UniqueName: \"kubernetes.io/projected/05f6d23b-ab70-4e62-9065-41f075c26ab6-kube-api-access-q6qv9\") pod \"e2e-shared-b-w1-d2b8ee-openshift-default-6f9c87f479-qpj8z\" (UID: \"05f6d23b-ab70-4e62-9065-41f075c26ab6\") " pod="openshift-ingress/e2e-shared-b-w1-d2b8ee-openshift-default-6f9c87f479-qpj8z" Nov 04 03:06:20.832937 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:06:20.832762 4405 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/05f6d23b-ab70-4e62-9065-41f075c26ab6-istio-envoy\") pod \"e2e-shared-b-w1-d2b8ee-openshift-default-6f9c87f479-qpj8z\" (UID: \"05f6d23b-ab70-4e62-9065-41f075c26ab6\") " pod="openshift-ingress/e2e-shared-b-w1-d2b8ee-openshift-default-6f9c87f479-qpj8z" Nov 04 03:06:20.832937 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:06:20.832884 4405 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/05f6d23b-ab70-4e62-9065-41f075c26ab6-istiod-ca-cert\") pod \"e2e-shared-b-w1-d2b8ee-openshift-default-6f9c87f479-qpj8z\" (UID: \"05f6d23b-ab70-4e62-9065-41f075c26ab6\") " pod="openshift-ingress/e2e-shared-b-w1-d2b8ee-openshift-default-6f9c87f479-qpj8z" Nov 04 03:06:20.833027 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:06:20.832971 4405 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/05f6d23b-ab70-4e62-9065-41f075c26ab6-istio-token\") pod \"e2e-shared-b-w1-d2b8ee-openshift-default-6f9c87f479-qpj8z\" (UID: \"05f6d23b-ab70-4e62-9065-41f075c26ab6\") " pod="openshift-ingress/e2e-shared-b-w1-d2b8ee-openshift-default-6f9c87f479-qpj8z" Nov 04 03:06:20.833027 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:06:20.832996 4405 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/05f6d23b-ab70-4e62-9065-41f075c26ab6-credential-socket\") pod \"e2e-shared-b-w1-d2b8ee-openshift-default-6f9c87f479-qpj8z\" (UID: \"05f6d23b-ab70-4e62-9065-41f075c26ab6\") " pod="openshift-ingress/e2e-shared-b-w1-d2b8ee-openshift-default-6f9c87f479-qpj8z" Nov 04 03:06:20.833139 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:06:20.833125 4405 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/05f6d23b-ab70-4e62-9065-41f075c26ab6-istio-podinfo\") pod \"e2e-shared-b-w1-d2b8ee-openshift-default-6f9c87f479-qpj8z\" (UID: \"05f6d23b-ab70-4e62-9065-41f075c26ab6\") " pod="openshift-ingress/e2e-shared-b-w1-d2b8ee-openshift-default-6f9c87f479-qpj8z" Nov 04 03:06:20.899661 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:06:20.899626 4405 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-d2b8ee-openshift-default-6f9c87f479-qpj8z"] Nov 04 03:06:20.934775 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:06:20.934734 4405 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/05f6d23b-ab70-4e62-9065-41f075c26ab6-istio-podinfo\") pod \"e2e-shared-b-w1-d2b8ee-openshift-default-6f9c87f479-qpj8z\" (UID: \"05f6d23b-ab70-4e62-9065-41f075c26ab6\") " pod="openshift-ingress/e2e-shared-b-w1-d2b8ee-openshift-default-6f9c87f479-qpj8z" Nov 04 03:06:20.934975 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:06:20.934888 4405 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/05f6d23b-ab70-4e62-9065-41f075c26ab6-istio-data\") pod \"e2e-shared-b-w1-d2b8ee-openshift-default-6f9c87f479-qpj8z\" (UID: \"05f6d23b-ab70-4e62-9065-41f075c26ab6\") " pod="openshift-ingress/e2e-shared-b-w1-d2b8ee-openshift-default-6f9c87f479-qpj8z" Nov 04 03:06:20.934975 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:06:20.934922 4405 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/05f6d23b-ab70-4e62-9065-41f075c26ab6-workload-socket\") pod \"e2e-shared-b-w1-d2b8ee-openshift-default-6f9c87f479-qpj8z\" (UID: \"05f6d23b-ab70-4e62-9065-41f075c26ab6\") " pod="openshift-ingress/e2e-shared-b-w1-d2b8ee-openshift-default-6f9c87f479-qpj8z" Nov 04 03:06:20.934975 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:06:20.934958 4405 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/05f6d23b-ab70-4e62-9065-41f075c26ab6-workload-certs\") pod \"e2e-shared-b-w1-d2b8ee-openshift-default-6f9c87f479-qpj8z\" (UID: \"05f6d23b-ab70-4e62-9065-41f075c26ab6\") " pod="openshift-ingress/e2e-shared-b-w1-d2b8ee-openshift-default-6f9c87f479-qpj8z" Nov 04 03:06:20.935151 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:06:20.935019 4405 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q6qv9\" (UniqueName: \"kubernetes.io/projected/05f6d23b-ab70-4e62-9065-41f075c26ab6-kube-api-access-q6qv9\") pod \"e2e-shared-b-w1-d2b8ee-openshift-default-6f9c87f479-qpj8z\" (UID: \"05f6d23b-ab70-4e62-9065-41f075c26ab6\") " pod="openshift-ingress/e2e-shared-b-w1-d2b8ee-openshift-default-6f9c87f479-qpj8z" Nov 04 03:06:20.935151 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:06:20.935066 4405 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/05f6d23b-ab70-4e62-9065-41f075c26ab6-istio-envoy\") pod \"e2e-shared-b-w1-d2b8ee-openshift-default-6f9c87f479-qpj8z\" (UID: \"05f6d23b-ab70-4e62-9065-41f075c26ab6\") " pod="openshift-ingress/e2e-shared-b-w1-d2b8ee-openshift-default-6f9c87f479-qpj8z" Nov 04 03:06:20.935151 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:06:20.935116 4405 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/05f6d23b-ab70-4e62-9065-41f075c26ab6-istiod-ca-cert\") pod \"e2e-shared-b-w1-d2b8ee-openshift-default-6f9c87f479-qpj8z\" (UID: \"05f6d23b-ab70-4e62-9065-41f075c26ab6\") " pod="openshift-ingress/e2e-shared-b-w1-d2b8ee-openshift-default-6f9c87f479-qpj8z" Nov 04 03:06:20.935344 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:06:20.935322 4405 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/05f6d23b-ab70-4e62-9065-41f075c26ab6-workload-certs\") pod \"e2e-shared-b-w1-d2b8ee-openshift-default-6f9c87f479-qpj8z\" (UID: \"05f6d23b-ab70-4e62-9065-41f075c26ab6\") " pod="openshift-ingress/e2e-shared-b-w1-d2b8ee-openshift-default-6f9c87f479-qpj8z" Nov 04 03:06:20.935413 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:06:20.935337 4405 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/05f6d23b-ab70-4e62-9065-41f075c26ab6-workload-socket\") pod \"e2e-shared-b-w1-d2b8ee-openshift-default-6f9c87f479-qpj8z\" (UID: \"05f6d23b-ab70-4e62-9065-41f075c26ab6\") " pod="openshift-ingress/e2e-shared-b-w1-d2b8ee-openshift-default-6f9c87f479-qpj8z" Nov 04 03:06:20.935858 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:06:20.935567 4405 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/05f6d23b-ab70-4e62-9065-41f075c26ab6-istio-token\") pod \"e2e-shared-b-w1-d2b8ee-openshift-default-6f9c87f479-qpj8z\" (UID: \"05f6d23b-ab70-4e62-9065-41f075c26ab6\") " pod="openshift-ingress/e2e-shared-b-w1-d2b8ee-openshift-default-6f9c87f479-qpj8z" Nov 04 03:06:20.935858 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:06:20.935589 4405 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/05f6d23b-ab70-4e62-9065-41f075c26ab6-istio-data\") pod \"e2e-shared-b-w1-d2b8ee-openshift-default-6f9c87f479-qpj8z\" (UID: \"05f6d23b-ab70-4e62-9065-41f075c26ab6\") " pod="openshift-ingress/e2e-shared-b-w1-d2b8ee-openshift-default-6f9c87f479-qpj8z" Nov 04 03:06:20.935858 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:06:20.935608 4405 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/05f6d23b-ab70-4e62-9065-41f075c26ab6-credential-socket\") pod \"e2e-shared-b-w1-d2b8ee-openshift-default-6f9c87f479-qpj8z\" (UID: \"05f6d23b-ab70-4e62-9065-41f075c26ab6\") " pod="openshift-ingress/e2e-shared-b-w1-d2b8ee-openshift-default-6f9c87f479-qpj8z" Nov 04 03:06:20.936097 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:06:20.936037 4405 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/05f6d23b-ab70-4e62-9065-41f075c26ab6-credential-socket\") pod \"e2e-shared-b-w1-d2b8ee-openshift-default-6f9c87f479-qpj8z\" (UID: \"05f6d23b-ab70-4e62-9065-41f075c26ab6\") " pod="openshift-ingress/e2e-shared-b-w1-d2b8ee-openshift-default-6f9c87f479-qpj8z" Nov 04 03:06:20.936197 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:06:20.936170 4405 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/05f6d23b-ab70-4e62-9065-41f075c26ab6-istiod-ca-cert\") pod \"e2e-shared-b-w1-d2b8ee-openshift-default-6f9c87f479-qpj8z\" (UID: \"05f6d23b-ab70-4e62-9065-41f075c26ab6\") " pod="openshift-ingress/e2e-shared-b-w1-d2b8ee-openshift-default-6f9c87f479-qpj8z" Nov 04 03:06:20.938045 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:06:20.938024 4405 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/05f6d23b-ab70-4e62-9065-41f075c26ab6-istio-envoy\") pod \"e2e-shared-b-w1-d2b8ee-openshift-default-6f9c87f479-qpj8z\" (UID: \"05f6d23b-ab70-4e62-9065-41f075c26ab6\") " pod="openshift-ingress/e2e-shared-b-w1-d2b8ee-openshift-default-6f9c87f479-qpj8z" Nov 04 03:06:20.938367 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:06:20.938340 4405 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/05f6d23b-ab70-4e62-9065-41f075c26ab6-istio-podinfo\") pod \"e2e-shared-b-w1-d2b8ee-openshift-default-6f9c87f479-qpj8z\" (UID: \"05f6d23b-ab70-4e62-9065-41f075c26ab6\") " pod="openshift-ingress/e2e-shared-b-w1-d2b8ee-openshift-default-6f9c87f479-qpj8z" Nov 04 03:06:21.004987 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:06:21.004955 4405 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/05f6d23b-ab70-4e62-9065-41f075c26ab6-istio-token\") pod \"e2e-shared-b-w1-d2b8ee-openshift-default-6f9c87f479-qpj8z\" (UID: \"05f6d23b-ab70-4e62-9065-41f075c26ab6\") " pod="openshift-ingress/e2e-shared-b-w1-d2b8ee-openshift-default-6f9c87f479-qpj8z" Nov 04 03:06:21.005146 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:06:21.004963 4405 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q6qv9\" (UniqueName: \"kubernetes.io/projected/05f6d23b-ab70-4e62-9065-41f075c26ab6-kube-api-access-q6qv9\") pod \"e2e-shared-b-w1-d2b8ee-openshift-default-6f9c87f479-qpj8z\" (UID: \"05f6d23b-ab70-4e62-9065-41f075c26ab6\") " pod="openshift-ingress/e2e-shared-b-w1-d2b8ee-openshift-default-6f9c87f479-qpj8z" Nov 04 03:06:21.077533 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:06:21.077452 4405 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w1-d2b8ee-openshift-default-6f9c87f479-qpj8z" Nov 04 03:06:21.526227 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:06:21.526201 4405 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-d2b8ee-openshift-default-6f9c87f479-qpj8z"] Nov 04 03:06:21.527700 ip-10-0-134-125 kubenswrapper[4405]: W1104 03:06:21.527653 4405 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod05f6d23b_ab70_4e62_9065_41f075c26ab6.slice/crio-b3d139c51befb99571e70a5e3f8e93424c03363bc0855a7a6639dcaf39d729c8 WatchSource:0}: Error finding container b3d139c51befb99571e70a5e3f8e93424c03363bc0855a7a6639dcaf39d729c8: Status 404 returned error can't find the container with id b3d139c51befb99571e70a5e3f8e93424c03363bc0855a7a6639dcaf39d729c8 Nov 04 03:06:21.530831 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:06:21.530796 4405 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 04 03:06:21.530941 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:06:21.530871 4405 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 04 03:06:21.530941 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:06:21.530901 4405 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 04 03:06:21.736639 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:06:21.736596 4405 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-d2b8ee-openshift-default-6f9c87f479-qpj8z" event={"ID":"05f6d23b-ab70-4e62-9065-41f075c26ab6","Type":"ContainerStarted","Data":"53d64ae561768b772001dc3e78a94d2c43c3af8d3eb2c22d4f465a47376c8d4b"} Nov 04 03:06:21.736851 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:06:21.736701 4405 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-d2b8ee-openshift-default-6f9c87f479-qpj8z" event={"ID":"05f6d23b-ab70-4e62-9065-41f075c26ab6","Type":"ContainerStarted","Data":"b3d139c51befb99571e70a5e3f8e93424c03363bc0855a7a6639dcaf39d729c8"} Nov 04 03:06:21.912197 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:06:21.912129 4405 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w1-d2b8ee-openshift-default-6f9c87f479-qpj8z" podStartSLOduration=1.912115606 podStartE2EDuration="1.912115606s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 03:06:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 03:06:21.909902452 +0000 UTC m=+2028.340602611" watchObservedRunningTime="2026-11-04 03:06:21.912115606 +0000 UTC m=+2028.342816093" Nov 04 03:06:22.078684 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:06:22.078636 4405 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w1-d2b8ee-openshift-default-6f9c87f479-qpj8z" Nov 04 03:06:22.080761 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:06:22.080706 4405 patch_prober.go:29] interesting pod/e2e-shared-b-w1-d2b8ee-openshift-default-6f9c87f479-qpj8z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.116:15021/healthz/ready\": dial tcp 10.134.0.116:15021: connect: connection refused" start-of-body= Nov 04 03:06:22.081084 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:06:22.081040 4405 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-d2b8ee-openshift-default-6f9c87f479-qpj8z" podUID="05f6d23b-ab70-4e62-9065-41f075c26ab6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.116:15021/healthz/ready\": dial tcp 10.134.0.116:15021: connect: connection refused" Nov 04 03:06:23.078216 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:06:23.078180 4405 patch_prober.go:29] interesting pod/e2e-shared-b-w1-d2b8ee-openshift-default-6f9c87f479-qpj8z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.116:15021/healthz/ready\": dial tcp 10.134.0.116:15021: connect: connection refused" start-of-body= Nov 04 03:06:23.078656 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:06:23.078233 4405 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-d2b8ee-openshift-default-6f9c87f479-qpj8z" podUID="05f6d23b-ab70-4e62-9065-41f075c26ab6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.116:15021/healthz/ready\": dial tcp 10.134.0.116:15021: connect: connection refused" Nov 04 03:06:24.078522 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:06:24.078479 4405 patch_prober.go:29] interesting pod/e2e-shared-b-w1-d2b8ee-openshift-default-6f9c87f479-qpj8z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.116:15021/healthz/ready\": dial tcp 10.134.0.116:15021: connect: connection refused" start-of-body= Nov 04 03:06:24.079084 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:06:24.078563 4405 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-d2b8ee-openshift-default-6f9c87f479-qpj8z" podUID="05f6d23b-ab70-4e62-9065-41f075c26ab6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.116:15021/healthz/ready\": dial tcp 10.134.0.116:15021: connect: connection refused" Nov 04 03:06:25.078039 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:06:25.078003 4405 patch_prober.go:29] interesting pod/e2e-shared-b-w1-d2b8ee-openshift-default-6f9c87f479-qpj8z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.116:15021/healthz/ready\": dial tcp 10.134.0.116:15021: connect: connection refused" start-of-body= Nov 04 03:06:25.078226 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:06:25.078082 4405 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-d2b8ee-openshift-default-6f9c87f479-qpj8z" podUID="05f6d23b-ab70-4e62-9065-41f075c26ab6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.116:15021/healthz/ready\": dial tcp 10.134.0.116:15021: connect: connection refused" Nov 04 03:06:26.078324 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:06:26.078281 4405 patch_prober.go:29] interesting pod/e2e-shared-b-w1-d2b8ee-openshift-default-6f9c87f479-qpj8z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.116:15021/healthz/ready\": dial tcp 10.134.0.116:15021: connect: connection refused" start-of-body= Nov 04 03:06:26.078873 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:06:26.078350 4405 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-d2b8ee-openshift-default-6f9c87f479-qpj8z" podUID="05f6d23b-ab70-4e62-9065-41f075c26ab6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.116:15021/healthz/ready\": dial tcp 10.134.0.116:15021: connect: connection refused" Nov 04 03:06:27.040704 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:06:27.039212 4405 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-9f4199-84b49988cf-thdfx"] Nov 04 03:06:27.056071 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:06:27.056017 4405 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-9f4199-84b49988cf-thdfx"] Nov 04 03:06:27.056268 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:06:27.056178 4405 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-9f4199-84b49988cf-thdfx" Nov 04 03:06:27.080999 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:06:27.080803 4405 patch_prober.go:29] interesting pod/e2e-shared-b-w1-d2b8ee-openshift-default-6f9c87f479-qpj8z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.116:15021/healthz/ready\": dial tcp 10.134.0.116:15021: connect: connection refused" start-of-body= Nov 04 03:06:27.080999 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:06:27.080879 4405 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-d2b8ee-openshift-default-6f9c87f479-qpj8z" podUID="05f6d23b-ab70-4e62-9065-41f075c26ab6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.116:15021/healthz/ready\": dial tcp 10.134.0.116:15021: connect: connection refused" Nov 04 03:06:27.101184 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:06:27.089611 4405 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a6be9827-0b28-45b8-bcbe-242840761447-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-9f4199-84b49988cf-thdfx\" (UID: \"a6be9827-0b28-45b8-bcbe-242840761447\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-9f4199-84b49988cf-thdfx" Nov 04 03:06:27.101184 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:06:27.089744 4405 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rwmt8\" (UniqueName: \"kubernetes.io/projected/a6be9827-0b28-45b8-bcbe-242840761447-kube-api-access-rwmt8\") pod \"payload-pre-processing-e2e-inf-a-9f4199-84b49988cf-thdfx\" (UID: \"a6be9827-0b28-45b8-bcbe-242840761447\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-9f4199-84b49988cf-thdfx" Nov 04 03:06:27.207172 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:06:27.207132 4405 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-9f4199-55cf88d648-fr4zd"] Nov 04 03:06:27.307841 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:06:27.211700 4405 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-a-9f4199\"" Nov 04 03:06:27.307841 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:06:27.212600 4405 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rwmt8\" (UniqueName: \"kubernetes.io/projected/a6be9827-0b28-45b8-bcbe-242840761447-kube-api-access-rwmt8\") pod \"payload-pre-processing-e2e-inf-a-9f4199-84b49988cf-thdfx\" (UID: \"a6be9827-0b28-45b8-bcbe-242840761447\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-9f4199-84b49988cf-thdfx" Nov 04 03:06:27.307841 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:06:27.306238 4405 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a6be9827-0b28-45b8-bcbe-242840761447-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-9f4199-84b49988cf-thdfx\" (UID: \"a6be9827-0b28-45b8-bcbe-242840761447\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-9f4199-84b49988cf-thdfx" Nov 04 03:06:27.307841 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:06:27.307402 4405 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a6be9827-0b28-45b8-bcbe-242840761447-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-9f4199-84b49988cf-thdfx\" (UID: \"a6be9827-0b28-45b8-bcbe-242840761447\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-9f4199-84b49988cf-thdfx" Nov 04 03:06:27.355157 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:06:27.355087 4405 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rwmt8\" (UniqueName: \"kubernetes.io/projected/a6be9827-0b28-45b8-bcbe-242840761447-kube-api-access-rwmt8\") pod \"payload-pre-processing-e2e-inf-a-9f4199-84b49988cf-thdfx\" (UID: \"a6be9827-0b28-45b8-bcbe-242840761447\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-9f4199-84b49988cf-thdfx" Nov 04 03:06:27.383692 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:06:27.382772 4405 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-9f4199-55cf88d648-fr4zd"] Nov 04 03:06:27.383692 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:06:27.382872 4405 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-9f4199-55cf88d648-fr4zd" Nov 04 03:06:27.438643 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:06:27.438178 4405 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-a-9f4199-dockercfg-r8z4f\"" Nov 04 03:06:27.514155 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:06:27.514116 4405 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/01be67f1-0362-4960-af5b-4ddad895ab4a-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-9f4199-55cf88d648-fr4zd\" (UID: \"01be67f1-0362-4960-af5b-4ddad895ab4a\") " pod="openshift-ingress/payload-processing-e2e-inf-a-9f4199-55cf88d648-fr4zd" Nov 04 03:06:27.524520 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:06:27.515358 4405 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jp6jf\" (UniqueName: \"kubernetes.io/projected/01be67f1-0362-4960-af5b-4ddad895ab4a-kube-api-access-jp6jf\") pod \"payload-processing-e2e-inf-a-9f4199-55cf88d648-fr4zd\" (UID: \"01be67f1-0362-4960-af5b-4ddad895ab4a\") " pod="openshift-ingress/payload-processing-e2e-inf-a-9f4199-55cf88d648-fr4zd" Nov 04 03:06:27.524520 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:06:27.514567 4405 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-9f4199-84b49988cf-thdfx" Nov 04 03:06:27.637431 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:06:27.626756 4405 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/01be67f1-0362-4960-af5b-4ddad895ab4a-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-9f4199-55cf88d648-fr4zd\" (UID: \"01be67f1-0362-4960-af5b-4ddad895ab4a\") " pod="openshift-ingress/payload-processing-e2e-inf-a-9f4199-55cf88d648-fr4zd" Nov 04 03:06:27.637431 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:06:27.626807 4405 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jp6jf\" (UniqueName: \"kubernetes.io/projected/01be67f1-0362-4960-af5b-4ddad895ab4a-kube-api-access-jp6jf\") pod \"payload-processing-e2e-inf-a-9f4199-55cf88d648-fr4zd\" (UID: \"01be67f1-0362-4960-af5b-4ddad895ab4a\") " pod="openshift-ingress/payload-processing-e2e-inf-a-9f4199-55cf88d648-fr4zd" Nov 04 03:06:27.639425 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:06:27.639394 4405 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/01be67f1-0362-4960-af5b-4ddad895ab4a-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-9f4199-55cf88d648-fr4zd\" (UID: \"01be67f1-0362-4960-af5b-4ddad895ab4a\") " pod="openshift-ingress/payload-processing-e2e-inf-a-9f4199-55cf88d648-fr4zd" Nov 04 03:06:27.703691 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:06:27.699588 4405 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jp6jf\" (UniqueName: \"kubernetes.io/projected/01be67f1-0362-4960-af5b-4ddad895ab4a-kube-api-access-jp6jf\") pod \"payload-processing-e2e-inf-a-9f4199-55cf88d648-fr4zd\" (UID: \"01be67f1-0362-4960-af5b-4ddad895ab4a\") " pod="openshift-ingress/payload-processing-e2e-inf-a-9f4199-55cf88d648-fr4zd" Nov 04 03:06:27.754304 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:06:27.754270 4405 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-9f4199-55cf88d648-fr4zd" Nov 04 03:06:27.917342 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:06:27.917277 4405 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-9f4199-84b49988cf-thdfx"] Nov 04 03:06:27.917723 ip-10-0-134-125 kubenswrapper[4405]: W1104 03:06:27.917651 4405 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda6be9827_0b28_45b8_bcbe_242840761447.slice/crio-298c2f3243e75aaf64f997ed4d793a4bb019cab48b68f09cbb0347e24123f523 WatchSource:0}: Error finding container 298c2f3243e75aaf64f997ed4d793a4bb019cab48b68f09cbb0347e24123f523: Status 404 returned error can't find the container with id 298c2f3243e75aaf64f997ed4d793a4bb019cab48b68f09cbb0347e24123f523 Nov 04 03:06:28.084112 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:06:28.084080 4405 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w1-d2b8ee-openshift-default-6f9c87f479-qpj8z" Nov 04 03:06:28.084536 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:06:28.084489 4405 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w1-d2b8ee-openshift-default-6f9c87f479-qpj8z" Nov 04 03:06:28.085504 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:06:28.085484 4405 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w1-d2b8ee-openshift-default-6f9c87f479-qpj8z" Nov 04 03:06:28.142481 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:06:28.142451 4405 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-9f4199-55cf88d648-fr4zd"] Nov 04 03:06:28.143304 ip-10-0-134-125 kubenswrapper[4405]: W1104 03:06:28.143280 4405 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod01be67f1_0362_4960_af5b_4ddad895ab4a.slice/crio-6812c9b13d39516b32375f7fb675cf7d4a0b218ef08b56016abb6224dcb7973a WatchSource:0}: Error finding container 6812c9b13d39516b32375f7fb675cf7d4a0b218ef08b56016abb6224dcb7973a: Status 404 returned error can't find the container with id 6812c9b13d39516b32375f7fb675cf7d4a0b218ef08b56016abb6224dcb7973a Nov 04 03:06:28.867038 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:06:28.867001 4405 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-9f4199-84b49988cf-thdfx" event={"ID":"a6be9827-0b28-45b8-bcbe-242840761447","Type":"ContainerStarted","Data":"417395d9f1353766a912bfa1a7a3b69228e3c2a877036ad4a75b71f943a00d19"} Nov 04 03:06:28.867252 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:06:28.867062 4405 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-9f4199-84b49988cf-thdfx" event={"ID":"a6be9827-0b28-45b8-bcbe-242840761447","Type":"ContainerStarted","Data":"298c2f3243e75aaf64f997ed4d793a4bb019cab48b68f09cbb0347e24123f523"} Nov 04 03:06:28.867252 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:06:28.867090 4405 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-9f4199-84b49988cf-thdfx" Nov 04 03:06:28.868969 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:06:28.868945 4405 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-9f4199-55cf88d648-fr4zd" event={"ID":"01be67f1-0362-4960-af5b-4ddad895ab4a","Type":"ContainerStarted","Data":"db40ec687e8d94fb92b9732be62d42f0f8ebc51cd768858cf8b918d8f519cb40"} Nov 04 03:06:28.869102 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:06:28.868981 4405 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-9f4199-55cf88d648-fr4zd" event={"ID":"01be67f1-0362-4960-af5b-4ddad895ab4a","Type":"ContainerStarted","Data":"6812c9b13d39516b32375f7fb675cf7d4a0b218ef08b56016abb6224dcb7973a"} Nov 04 03:06:28.983605 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:06:28.983547 4405 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-9f4199-84b49988cf-thdfx" podStartSLOduration=2.983533808 podStartE2EDuration="2.983533808s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 03:06:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 03:06:28.982451086 +0000 UTC m=+2035.413151209" watchObservedRunningTime="2026-11-04 03:06:28.983533808 +0000 UTC m=+2035.414233928" Nov 04 03:06:29.107708 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:06:29.104753 4405 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-a-9f4199-55cf88d648-fr4zd" podStartSLOduration=2.104730959 podStartE2EDuration="2.104730959s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 03:06:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 03:06:29.098156558 +0000 UTC m=+2035.528856670" watchObservedRunningTime="2026-11-04 03:06:29.104730959 +0000 UTC m=+2035.535431081" Nov 04 03:06:29.876387 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:06:29.876351 4405 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-a-9f4199-55cf88d648-fr4zd" Nov 04 03:06:36.894867 ip-10-0-134-125 kubenswrapper[4405]: E1104 03:06:36.894826 4405 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5f2vb" containerName="extractor" Nov 04 03:06:39.190158 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:06:39.190124 4405 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-cdcf67-openshift-default-6ff8f5cbc7-975ss"] Nov 04 03:06:39.190741 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:06:39.190354 4405 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-cdcf67-openshift-default-6ff8f5cbc7-975ss" podUID="e5ccd15d-11fd-4f67-9bd5-7a4cfa35d602" containerName="istio-proxy" containerID="cri-o://795acf882ba76a67847bc16da20bf2fbd18afd9b410a73878a9e0ea63211deb3" gracePeriod=30 Nov 04 03:06:39.880381 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:06:39.880340 4405 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-9f4199-84b49988cf-thdfx" Nov 04 03:06:40.658865 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:06:40.658823 4405 patch_prober.go:29] interesting pod/e2e-worker-w3-cdcf67-openshift-default-6ff8f5cbc7-975ss container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 04 03:06:40.659356 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:06:40.658900 4405 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-cdcf67-openshift-default-6ff8f5cbc7-975ss" podUID="e5ccd15d-11fd-4f67-9bd5-7a4cfa35d602" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 04 03:06:40.885486 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:06:40.885449 4405 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-3b7476-openshift-default-59975457bf-gph88"] Nov 04 03:06:40.885712 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:06:40.885656 4405 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w5-3b7476-openshift-default-59975457bf-gph88" podUID="43a58ae9-e100-4e85-bbe0-fdc36af6f192" containerName="istio-proxy" containerID="cri-o://c64cebb6a88c524d1824f445d30518a0d02ce7ef085793001775e7011068f093" gracePeriod=30 Nov 04 03:06:40.888081 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:06:40.888042 4405 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-a-9f4199-55cf88d648-fr4zd" Nov 04 03:06:40.935370 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:06:40.935290 4405 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-3b7476-openshift-default-59975457bf-vxwkm"] Nov 04 03:06:40.935547 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:06:40.935487 4405 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w5-3b7476-openshift-default-59975457bf-vxwkm" podUID="f2c5af93-085e-462b-8377-a6e8139e66c8" containerName="istio-proxy" containerID="cri-o://fd6959d2c3e3f6141a6342ce834069df4084a2f029a4812d29904c9c0ac88013" gracePeriod=30 Nov 04 03:06:43.281368 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:06:43.281303 4405 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-2g6x2"] Nov 04 03:06:43.281930 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:06:43.281543 4405 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-2g6x2" podUID="23294ab6-3eaf-41fd-8b5b-c7a0214bd532" containerName="discovery" containerID="cri-o://95a1b806d2ddc75ad0d3d6e784dc9ae4e493c362f6a751d7fe17176a00340424" gracePeriod=30 Nov 04 03:06:43.574646 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:06:43.574615 4405 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-2g6x2" Nov 04 03:06:43.637720 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:06:43.637651 4405 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/23294ab6-3eaf-41fd-8b5b-c7a0214bd532-istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/23294ab6-3eaf-41fd-8b5b-c7a0214bd532-istio-csr-ca-configmap\") pod \"23294ab6-3eaf-41fd-8b5b-c7a0214bd532\" (UID: \"23294ab6-3eaf-41fd-8b5b-c7a0214bd532\") " Nov 04 03:06:43.637903 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:06:43.637730 4405 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/23294ab6-3eaf-41fd-8b5b-c7a0214bd532-istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/23294ab6-3eaf-41fd-8b5b-c7a0214bd532-istio-csr-dns-cert\") pod \"23294ab6-3eaf-41fd-8b5b-c7a0214bd532\" (UID: \"23294ab6-3eaf-41fd-8b5b-c7a0214bd532\") " Nov 04 03:06:43.637903 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:06:43.637811 4405 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/23294ab6-3eaf-41fd-8b5b-c7a0214bd532-local-certs\" (UniqueName: \"kubernetes.io/empty-dir/23294ab6-3eaf-41fd-8b5b-c7a0214bd532-local-certs\") pod \"23294ab6-3eaf-41fd-8b5b-c7a0214bd532\" (UID: \"23294ab6-3eaf-41fd-8b5b-c7a0214bd532\") " Nov 04 03:06:43.637903 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:06:43.637859 4405 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/23294ab6-3eaf-41fd-8b5b-c7a0214bd532-istio-token\" (UniqueName: \"kubernetes.io/projected/23294ab6-3eaf-41fd-8b5b-c7a0214bd532-istio-token\") pod \"23294ab6-3eaf-41fd-8b5b-c7a0214bd532\" (UID: \"23294ab6-3eaf-41fd-8b5b-c7a0214bd532\") " Nov 04 03:06:43.637903 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:06:43.637891 4405 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/23294ab6-3eaf-41fd-8b5b-c7a0214bd532-cacerts\" (UniqueName: \"kubernetes.io/secret/23294ab6-3eaf-41fd-8b5b-c7a0214bd532-cacerts\") pod \"23294ab6-3eaf-41fd-8b5b-c7a0214bd532\" (UID: \"23294ab6-3eaf-41fd-8b5b-c7a0214bd532\") " Nov 04 03:06:43.638143 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:06:43.637940 4405 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/23294ab6-3eaf-41fd-8b5b-c7a0214bd532-kube-api-access-sksxl\" (UniqueName: \"kubernetes.io/projected/23294ab6-3eaf-41fd-8b5b-c7a0214bd532-kube-api-access-sksxl\") pod \"23294ab6-3eaf-41fd-8b5b-c7a0214bd532\" (UID: \"23294ab6-3eaf-41fd-8b5b-c7a0214bd532\") " Nov 04 03:06:43.638143 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:06:43.638002 4405 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/23294ab6-3eaf-41fd-8b5b-c7a0214bd532-istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/23294ab6-3eaf-41fd-8b5b-c7a0214bd532-istio-kubeconfig\") pod \"23294ab6-3eaf-41fd-8b5b-c7a0214bd532\" (UID: \"23294ab6-3eaf-41fd-8b5b-c7a0214bd532\") " Nov 04 03:06:43.638143 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:06:43.638096 4405 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/23294ab6-3eaf-41fd-8b5b-c7a0214bd532-istio-csr-ca-configmap" pod "23294ab6-3eaf-41fd-8b5b-c7a0214bd532" (UID: "23294ab6-3eaf-41fd-8b5b-c7a0214bd532"). InnerVolumeSpecName "istio-csr-ca-configmap". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 03:06:43.638856 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:06:43.638820 4405 reconciler_common.go:299] "Volume detached for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/23294ab6-3eaf-41fd-8b5b-c7a0214bd532-istio-csr-ca-configmap\") on node \"ip-10-0-134-125.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:06:43.640254 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:06:43.640229 4405 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/23294ab6-3eaf-41fd-8b5b-c7a0214bd532-cacerts" pod "23294ab6-3eaf-41fd-8b5b-c7a0214bd532" (UID: "23294ab6-3eaf-41fd-8b5b-c7a0214bd532"). InnerVolumeSpecName "cacerts". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 04 03:06:43.640418 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:06:43.640392 4405 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/23294ab6-3eaf-41fd-8b5b-c7a0214bd532-istio-kubeconfig" pod "23294ab6-3eaf-41fd-8b5b-c7a0214bd532" (UID: "23294ab6-3eaf-41fd-8b5b-c7a0214bd532"). InnerVolumeSpecName "istio-kubeconfig". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 04 03:06:43.640753 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:06:43.640726 4405 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/23294ab6-3eaf-41fd-8b5b-c7a0214bd532-istio-token" pod "23294ab6-3eaf-41fd-8b5b-c7a0214bd532" (UID: "23294ab6-3eaf-41fd-8b5b-c7a0214bd532"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 03:06:43.640842 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:06:43.640755 4405 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/23294ab6-3eaf-41fd-8b5b-c7a0214bd532-local-certs" pod "23294ab6-3eaf-41fd-8b5b-c7a0214bd532" (UID: "23294ab6-3eaf-41fd-8b5b-c7a0214bd532"). InnerVolumeSpecName "local-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 03:06:43.640842 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:06:43.640799 4405 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/23294ab6-3eaf-41fd-8b5b-c7a0214bd532-istio-csr-dns-cert" pod "23294ab6-3eaf-41fd-8b5b-c7a0214bd532" (UID: "23294ab6-3eaf-41fd-8b5b-c7a0214bd532"). InnerVolumeSpecName "istio-csr-dns-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 04 03:06:43.642775 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:06:43.642749 4405 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/23294ab6-3eaf-41fd-8b5b-c7a0214bd532-kube-api-access-sksxl" pod "23294ab6-3eaf-41fd-8b5b-c7a0214bd532" (UID: "23294ab6-3eaf-41fd-8b5b-c7a0214bd532"). InnerVolumeSpecName "kube-api-access-sksxl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 03:06:43.740576 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:06:43.740539 4405 reconciler_common.go:299] "Volume detached for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/23294ab6-3eaf-41fd-8b5b-c7a0214bd532-istio-csr-dns-cert\") on node \"ip-10-0-134-125.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:06:43.740576 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:06:43.740575 4405 reconciler_common.go:299] "Volume detached for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/23294ab6-3eaf-41fd-8b5b-c7a0214bd532-local-certs\") on node \"ip-10-0-134-125.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:06:43.740847 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:06:43.740590 4405 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/23294ab6-3eaf-41fd-8b5b-c7a0214bd532-istio-token\") on node \"ip-10-0-134-125.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:06:43.740847 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:06:43.740600 4405 reconciler_common.go:299] "Volume detached for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/23294ab6-3eaf-41fd-8b5b-c7a0214bd532-cacerts\") on node \"ip-10-0-134-125.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:06:43.740847 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:06:43.740611 4405 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sksxl\" (UniqueName: \"kubernetes.io/projected/23294ab6-3eaf-41fd-8b5b-c7a0214bd532-kube-api-access-sksxl\") on node \"ip-10-0-134-125.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:06:43.740847 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:06:43.740620 4405 reconciler_common.go:299] "Volume detached for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/23294ab6-3eaf-41fd-8b5b-c7a0214bd532-istio-kubeconfig\") on node \"ip-10-0-134-125.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:06:44.040632 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:06:44.040598 4405 generic.go:356] "Generic (PLEG): container finished" podID="23294ab6-3eaf-41fd-8b5b-c7a0214bd532" containerID="95a1b806d2ddc75ad0d3d6e784dc9ae4e493c362f6a751d7fe17176a00340424" exitCode=0 Nov 04 03:06:44.040984 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:06:44.040646 4405 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-2g6x2" Nov 04 03:06:44.040984 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:06:44.040693 4405 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-2g6x2" event={"ID":"23294ab6-3eaf-41fd-8b5b-c7a0214bd532","Type":"ContainerDied","Data":"95a1b806d2ddc75ad0d3d6e784dc9ae4e493c362f6a751d7fe17176a00340424"} Nov 04 03:06:44.040984 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:06:44.040748 4405 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-2g6x2" event={"ID":"23294ab6-3eaf-41fd-8b5b-c7a0214bd532","Type":"ContainerDied","Data":"65ca1d127ca6ff52863c43c205c195534763b10b0c552c9f0beaafcd8376976a"} Nov 04 03:06:44.040984 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:06:44.040774 4405 scope.go:122] "RemoveContainer" containerID="95a1b806d2ddc75ad0d3d6e784dc9ae4e493c362f6a751d7fe17176a00340424" Nov 04 03:06:44.051261 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:06:44.051239 4405 scope.go:122] "RemoveContainer" containerID="95a1b806d2ddc75ad0d3d6e784dc9ae4e493c362f6a751d7fe17176a00340424" Nov 04 03:06:44.051567 ip-10-0-134-125 kubenswrapper[4405]: E1104 03:06:44.051543 4405 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"95a1b806d2ddc75ad0d3d6e784dc9ae4e493c362f6a751d7fe17176a00340424\": container with ID starting with 95a1b806d2ddc75ad0d3d6e784dc9ae4e493c362f6a751d7fe17176a00340424 not found: ID does not exist" containerID="95a1b806d2ddc75ad0d3d6e784dc9ae4e493c362f6a751d7fe17176a00340424" Nov 04 03:06:44.051659 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:06:44.051580 4405 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"95a1b806d2ddc75ad0d3d6e784dc9ae4e493c362f6a751d7fe17176a00340424"} err="failed to get container status \"95a1b806d2ddc75ad0d3d6e784dc9ae4e493c362f6a751d7fe17176a00340424\": rpc error: code = NotFound desc = could not find container \"95a1b806d2ddc75ad0d3d6e784dc9ae4e493c362f6a751d7fe17176a00340424\": container with ID starting with 95a1b806d2ddc75ad0d3d6e784dc9ae4e493c362f6a751d7fe17176a00340424 not found: ID does not exist" Nov 04 03:06:44.207768 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:06:44.207724 4405 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-2g6x2"] Nov 04 03:06:44.212024 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:06:44.211996 4405 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-2g6x2"] Nov 04 03:06:44.476890 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:06:44.476863 4405 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-cdcf67-openshift-default-6ff8f5cbc7-975ss" Nov 04 03:06:44.552140 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:06:44.552110 4405 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e5ccd15d-11fd-4f67-9bd5-7a4cfa35d602-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e5ccd15d-11fd-4f67-9bd5-7a4cfa35d602-credential-socket\") pod \"e5ccd15d-11fd-4f67-9bd5-7a4cfa35d602\" (UID: \"e5ccd15d-11fd-4f67-9bd5-7a4cfa35d602\") " Nov 04 03:06:44.552314 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:06:44.552186 4405 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e5ccd15d-11fd-4f67-9bd5-7a4cfa35d602-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e5ccd15d-11fd-4f67-9bd5-7a4cfa35d602-workload-certs\") pod \"e5ccd15d-11fd-4f67-9bd5-7a4cfa35d602\" (UID: \"e5ccd15d-11fd-4f67-9bd5-7a4cfa35d602\") " Nov 04 03:06:44.552314 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:06:44.552261 4405 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e5ccd15d-11fd-4f67-9bd5-7a4cfa35d602-kube-api-access-msfwr\" (UniqueName: \"kubernetes.io/projected/e5ccd15d-11fd-4f67-9bd5-7a4cfa35d602-kube-api-access-msfwr\") pod \"e5ccd15d-11fd-4f67-9bd5-7a4cfa35d602\" (UID: \"e5ccd15d-11fd-4f67-9bd5-7a4cfa35d602\") " Nov 04 03:06:44.552438 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:06:44.552322 4405 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e5ccd15d-11fd-4f67-9bd5-7a4cfa35d602-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e5ccd15d-11fd-4f67-9bd5-7a4cfa35d602-istio-data\") pod \"e5ccd15d-11fd-4f67-9bd5-7a4cfa35d602\" (UID: \"e5ccd15d-11fd-4f67-9bd5-7a4cfa35d602\") " Nov 04 03:06:44.552438 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:06:44.552413 4405 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e5ccd15d-11fd-4f67-9bd5-7a4cfa35d602-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e5ccd15d-11fd-4f67-9bd5-7a4cfa35d602-istio-envoy\") pod \"e5ccd15d-11fd-4f67-9bd5-7a4cfa35d602\" (UID: \"e5ccd15d-11fd-4f67-9bd5-7a4cfa35d602\") " Nov 04 03:06:44.552557 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:06:44.552430 4405 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e5ccd15d-11fd-4f67-9bd5-7a4cfa35d602-credential-socket" pod "e5ccd15d-11fd-4f67-9bd5-7a4cfa35d602" (UID: "e5ccd15d-11fd-4f67-9bd5-7a4cfa35d602"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 03:06:44.552557 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:06:44.552493 4405 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e5ccd15d-11fd-4f67-9bd5-7a4cfa35d602-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e5ccd15d-11fd-4f67-9bd5-7a4cfa35d602-workload-socket\") pod \"e5ccd15d-11fd-4f67-9bd5-7a4cfa35d602\" (UID: \"e5ccd15d-11fd-4f67-9bd5-7a4cfa35d602\") " Nov 04 03:06:44.552557 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:06:44.552538 4405 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/e5ccd15d-11fd-4f67-9bd5-7a4cfa35d602-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e5ccd15d-11fd-4f67-9bd5-7a4cfa35d602-istio-podinfo\") pod \"e5ccd15d-11fd-4f67-9bd5-7a4cfa35d602\" (UID: \"e5ccd15d-11fd-4f67-9bd5-7a4cfa35d602\") " Nov 04 03:06:44.552749 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:06:44.552541 4405 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e5ccd15d-11fd-4f67-9bd5-7a4cfa35d602-workload-certs" pod "e5ccd15d-11fd-4f67-9bd5-7a4cfa35d602" (UID: "e5ccd15d-11fd-4f67-9bd5-7a4cfa35d602"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 03:06:44.552749 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:06:44.552620 4405 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e5ccd15d-11fd-4f67-9bd5-7a4cfa35d602-istio-data" pod "e5ccd15d-11fd-4f67-9bd5-7a4cfa35d602" (UID: "e5ccd15d-11fd-4f67-9bd5-7a4cfa35d602"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 03:06:44.552749 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:06:44.552626 4405 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e5ccd15d-11fd-4f67-9bd5-7a4cfa35d602-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e5ccd15d-11fd-4f67-9bd5-7a4cfa35d602-istiod-ca-cert\") pod \"e5ccd15d-11fd-4f67-9bd5-7a4cfa35d602\" (UID: \"e5ccd15d-11fd-4f67-9bd5-7a4cfa35d602\") " Nov 04 03:06:44.552930 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:06:44.552760 4405 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e5ccd15d-11fd-4f67-9bd5-7a4cfa35d602-workload-socket" pod "e5ccd15d-11fd-4f67-9bd5-7a4cfa35d602" (UID: "e5ccd15d-11fd-4f67-9bd5-7a4cfa35d602"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 03:06:44.552930 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:06:44.552770 4405 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e5ccd15d-11fd-4f67-9bd5-7a4cfa35d602-istio-token\" (UniqueName: \"kubernetes.io/projected/e5ccd15d-11fd-4f67-9bd5-7a4cfa35d602-istio-token\") pod \"e5ccd15d-11fd-4f67-9bd5-7a4cfa35d602\" (UID: \"e5ccd15d-11fd-4f67-9bd5-7a4cfa35d602\") " Nov 04 03:06:44.553040 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:06:44.552933 4405 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e5ccd15d-11fd-4f67-9bd5-7a4cfa35d602-istiod-ca-cert" pod "e5ccd15d-11fd-4f67-9bd5-7a4cfa35d602" (UID: "e5ccd15d-11fd-4f67-9bd5-7a4cfa35d602"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 03:06:44.553539 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:06:44.553516 4405 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e5ccd15d-11fd-4f67-9bd5-7a4cfa35d602-workload-socket\") on node \"ip-10-0-134-125.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:06:44.553659 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:06:44.553544 4405 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e5ccd15d-11fd-4f67-9bd5-7a4cfa35d602-istiod-ca-cert\") on node \"ip-10-0-134-125.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:06:44.553659 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:06:44.553559 4405 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e5ccd15d-11fd-4f67-9bd5-7a4cfa35d602-credential-socket\") on node \"ip-10-0-134-125.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:06:44.553659 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:06:44.553572 4405 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e5ccd15d-11fd-4f67-9bd5-7a4cfa35d602-workload-certs\") on node \"ip-10-0-134-125.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:06:44.553659 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:06:44.553587 4405 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e5ccd15d-11fd-4f67-9bd5-7a4cfa35d602-istio-data\") on node \"ip-10-0-134-125.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:06:44.555009 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:06:44.554983 4405 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e5ccd15d-11fd-4f67-9bd5-7a4cfa35d602-kube-api-access-msfwr" pod "e5ccd15d-11fd-4f67-9bd5-7a4cfa35d602" (UID: "e5ccd15d-11fd-4f67-9bd5-7a4cfa35d602"). InnerVolumeSpecName "kube-api-access-msfwr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 03:06:44.555150 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:06:44.555127 4405 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e5ccd15d-11fd-4f67-9bd5-7a4cfa35d602-istio-token" pod "e5ccd15d-11fd-4f67-9bd5-7a4cfa35d602" (UID: "e5ccd15d-11fd-4f67-9bd5-7a4cfa35d602"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 03:06:44.555229 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:06:44.555206 4405 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e5ccd15d-11fd-4f67-9bd5-7a4cfa35d602-istio-envoy" pod "e5ccd15d-11fd-4f67-9bd5-7a4cfa35d602" (UID: "e5ccd15d-11fd-4f67-9bd5-7a4cfa35d602"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 03:06:44.555282 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:06:44.555215 4405 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e5ccd15d-11fd-4f67-9bd5-7a4cfa35d602-istio-podinfo" pod "e5ccd15d-11fd-4f67-9bd5-7a4cfa35d602" (UID: "e5ccd15d-11fd-4f67-9bd5-7a4cfa35d602"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 04 03:06:44.656055 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:06:44.656010 4405 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-msfwr\" (UniqueName: \"kubernetes.io/projected/e5ccd15d-11fd-4f67-9bd5-7a4cfa35d602-kube-api-access-msfwr\") on node \"ip-10-0-134-125.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:06:44.656055 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:06:44.656054 4405 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e5ccd15d-11fd-4f67-9bd5-7a4cfa35d602-istio-envoy\") on node \"ip-10-0-134-125.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:06:44.656313 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:06:44.656070 4405 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e5ccd15d-11fd-4f67-9bd5-7a4cfa35d602-istio-podinfo\") on node \"ip-10-0-134-125.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:06:44.656313 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:06:44.656083 4405 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e5ccd15d-11fd-4f67-9bd5-7a4cfa35d602-istio-token\") on node \"ip-10-0-134-125.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:06:45.053762 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:06:45.053715 4405 generic.go:356] "Generic (PLEG): container finished" podID="e5ccd15d-11fd-4f67-9bd5-7a4cfa35d602" containerID="795acf882ba76a67847bc16da20bf2fbd18afd9b410a73878a9e0ea63211deb3" exitCode=0 Nov 04 03:06:45.053971 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:06:45.053779 4405 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-cdcf67-openshift-default-6ff8f5cbc7-975ss" Nov 04 03:06:45.053971 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:06:45.053811 4405 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-cdcf67-openshift-default-6ff8f5cbc7-975ss" event={"ID":"e5ccd15d-11fd-4f67-9bd5-7a4cfa35d602","Type":"ContainerDied","Data":"795acf882ba76a67847bc16da20bf2fbd18afd9b410a73878a9e0ea63211deb3"} Nov 04 03:06:45.053971 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:06:45.053894 4405 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-cdcf67-openshift-default-6ff8f5cbc7-975ss" event={"ID":"e5ccd15d-11fd-4f67-9bd5-7a4cfa35d602","Type":"ContainerDied","Data":"4bc10598f608376d1ab4f226dce8a2263f36f4ac6e35a53faeb4630d720c9a37"} Nov 04 03:06:45.053971 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:06:45.053942 4405 scope.go:122] "RemoveContainer" containerID="795acf882ba76a67847bc16da20bf2fbd18afd9b410a73878a9e0ea63211deb3" Nov 04 03:06:45.075357 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:06:45.075322 4405 scope.go:122] "RemoveContainer" containerID="795acf882ba76a67847bc16da20bf2fbd18afd9b410a73878a9e0ea63211deb3" Nov 04 03:06:45.075818 ip-10-0-134-125 kubenswrapper[4405]: E1104 03:06:45.075786 4405 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"795acf882ba76a67847bc16da20bf2fbd18afd9b410a73878a9e0ea63211deb3\": container with ID starting with 795acf882ba76a67847bc16da20bf2fbd18afd9b410a73878a9e0ea63211deb3 not found: ID does not exist" containerID="795acf882ba76a67847bc16da20bf2fbd18afd9b410a73878a9e0ea63211deb3" Nov 04 03:06:45.075930 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:06:45.075830 4405 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"795acf882ba76a67847bc16da20bf2fbd18afd9b410a73878a9e0ea63211deb3"} err="failed to get container status \"795acf882ba76a67847bc16da20bf2fbd18afd9b410a73878a9e0ea63211deb3\": rpc error: code = NotFound desc = could not find container \"795acf882ba76a67847bc16da20bf2fbd18afd9b410a73878a9e0ea63211deb3\": container with ID starting with 795acf882ba76a67847bc16da20bf2fbd18afd9b410a73878a9e0ea63211deb3 not found: ID does not exist" Nov 04 03:06:45.220047 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:06:45.220004 4405 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-cdcf67-openshift-default-6ff8f5cbc7-975ss"] Nov 04 03:06:45.269683 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:06:45.269641 4405 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-cdcf67-openshift-default-6ff8f5cbc7-975ss"] Nov 04 03:06:45.703274 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:06:45.703244 4405 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="23294ab6-3eaf-41fd-8b5b-c7a0214bd532" path="/var/lib/kubelet/pods/23294ab6-3eaf-41fd-8b5b-c7a0214bd532/volumes" Nov 04 03:06:45.703782 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:06:45.703767 4405 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e5ccd15d-11fd-4f67-9bd5-7a4cfa35d602" path="/var/lib/kubelet/pods/e5ccd15d-11fd-4f67-9bd5-7a4cfa35d602/volumes" Nov 04 03:06:46.073907 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:06:46.073871 4405 generic.go:356] "Generic (PLEG): container finished" podID="f2c5af93-085e-462b-8377-a6e8139e66c8" containerID="fd6959d2c3e3f6141a6342ce834069df4084a2f029a4812d29904c9c0ac88013" exitCode=0 Nov 04 03:06:46.074214 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:06:46.074185 4405 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-3b7476-openshift-default-59975457bf-vxwkm" event={"ID":"f2c5af93-085e-462b-8377-a6e8139e66c8","Type":"ContainerDied","Data":"fd6959d2c3e3f6141a6342ce834069df4084a2f029a4812d29904c9c0ac88013"} Nov 04 03:06:46.077868 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:06:46.077838 4405 generic.go:356] "Generic (PLEG): container finished" podID="43a58ae9-e100-4e85-bbe0-fdc36af6f192" containerID="c64cebb6a88c524d1824f445d30518a0d02ce7ef085793001775e7011068f093" exitCode=0 Nov 04 03:06:46.077975 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:06:46.077910 4405 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-3b7476-openshift-default-59975457bf-gph88" event={"ID":"43a58ae9-e100-4e85-bbe0-fdc36af6f192","Type":"ContainerDied","Data":"c64cebb6a88c524d1824f445d30518a0d02ce7ef085793001775e7011068f093"} Nov 04 03:06:46.256110 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:06:46.256026 4405 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-3b7476-openshift-default-59975457bf-gph88" Nov 04 03:06:46.296562 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:06:46.296534 4405 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-3b7476-openshift-default-59975457bf-vxwkm" Nov 04 03:06:46.380804 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:06:46.380738 4405 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f2c5af93-085e-462b-8377-a6e8139e66c8-kube-api-access-284m6\" (UniqueName: \"kubernetes.io/projected/f2c5af93-085e-462b-8377-a6e8139e66c8-kube-api-access-284m6\") pod \"f2c5af93-085e-462b-8377-a6e8139e66c8\" (UID: \"f2c5af93-085e-462b-8377-a6e8139e66c8\") " Nov 04 03:06:46.381058 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:06:46.380824 4405 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/f2c5af93-085e-462b-8377-a6e8139e66c8-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f2c5af93-085e-462b-8377-a6e8139e66c8-istio-podinfo\") pod \"f2c5af93-085e-462b-8377-a6e8139e66c8\" (UID: \"f2c5af93-085e-462b-8377-a6e8139e66c8\") " Nov 04 03:06:46.381058 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:06:46.380869 4405 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f2c5af93-085e-462b-8377-a6e8139e66c8-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f2c5af93-085e-462b-8377-a6e8139e66c8-workload-certs\") pod \"f2c5af93-085e-462b-8377-a6e8139e66c8\" (UID: \"f2c5af93-085e-462b-8377-a6e8139e66c8\") " Nov 04 03:06:46.381058 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:06:46.380899 4405 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/43a58ae9-e100-4e85-bbe0-fdc36af6f192-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/43a58ae9-e100-4e85-bbe0-fdc36af6f192-istiod-ca-cert\") pod \"43a58ae9-e100-4e85-bbe0-fdc36af6f192\" (UID: \"43a58ae9-e100-4e85-bbe0-fdc36af6f192\") " Nov 04 03:06:46.381058 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:06:46.380968 4405 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f2c5af93-085e-462b-8377-a6e8139e66c8-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f2c5af93-085e-462b-8377-a6e8139e66c8-credential-socket\") pod \"f2c5af93-085e-462b-8377-a6e8139e66c8\" (UID: \"f2c5af93-085e-462b-8377-a6e8139e66c8\") " Nov 04 03:06:46.381058 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:06:46.381007 4405 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/43a58ae9-e100-4e85-bbe0-fdc36af6f192-istio-token\" (UniqueName: \"kubernetes.io/projected/43a58ae9-e100-4e85-bbe0-fdc36af6f192-istio-token\") pod \"43a58ae9-e100-4e85-bbe0-fdc36af6f192\" (UID: \"43a58ae9-e100-4e85-bbe0-fdc36af6f192\") " Nov 04 03:06:46.381058 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:06:46.381045 4405 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/43a58ae9-e100-4e85-bbe0-fdc36af6f192-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/43a58ae9-e100-4e85-bbe0-fdc36af6f192-istio-data\") pod \"43a58ae9-e100-4e85-bbe0-fdc36af6f192\" (UID: \"43a58ae9-e100-4e85-bbe0-fdc36af6f192\") " Nov 04 03:06:46.381459 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:06:46.381107 4405 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/43a58ae9-e100-4e85-bbe0-fdc36af6f192-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/43a58ae9-e100-4e85-bbe0-fdc36af6f192-istio-podinfo\") pod \"43a58ae9-e100-4e85-bbe0-fdc36af6f192\" (UID: \"43a58ae9-e100-4e85-bbe0-fdc36af6f192\") " Nov 04 03:06:46.381459 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:06:46.381192 4405 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f2c5af93-085e-462b-8377-a6e8139e66c8-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f2c5af93-085e-462b-8377-a6e8139e66c8-istio-envoy\") pod \"f2c5af93-085e-462b-8377-a6e8139e66c8\" (UID: \"f2c5af93-085e-462b-8377-a6e8139e66c8\") " Nov 04 03:06:46.381459 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:06:46.381269 4405 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f2c5af93-085e-462b-8377-a6e8139e66c8-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f2c5af93-085e-462b-8377-a6e8139e66c8-workload-socket\") pod \"f2c5af93-085e-462b-8377-a6e8139e66c8\" (UID: \"f2c5af93-085e-462b-8377-a6e8139e66c8\") " Nov 04 03:06:46.381459 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:06:46.381375 4405 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f2c5af93-085e-462b-8377-a6e8139e66c8-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f2c5af93-085e-462b-8377-a6e8139e66c8-istiod-ca-cert\") pod \"f2c5af93-085e-462b-8377-a6e8139e66c8\" (UID: \"f2c5af93-085e-462b-8377-a6e8139e66c8\") " Nov 04 03:06:46.381459 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:06:46.381423 4405 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/43a58ae9-e100-4e85-bbe0-fdc36af6f192-kube-api-access-9ddrz\" (UniqueName: \"kubernetes.io/projected/43a58ae9-e100-4e85-bbe0-fdc36af6f192-kube-api-access-9ddrz\") pod \"43a58ae9-e100-4e85-bbe0-fdc36af6f192\" (UID: \"43a58ae9-e100-4e85-bbe0-fdc36af6f192\") " Nov 04 03:06:46.381823 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:06:46.381525 4405 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/43a58ae9-e100-4e85-bbe0-fdc36af6f192-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/43a58ae9-e100-4e85-bbe0-fdc36af6f192-workload-socket\") pod \"43a58ae9-e100-4e85-bbe0-fdc36af6f192\" (UID: \"43a58ae9-e100-4e85-bbe0-fdc36af6f192\") " Nov 04 03:06:46.382000 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:06:46.381966 4405 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/43a58ae9-e100-4e85-bbe0-fdc36af6f192-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/43a58ae9-e100-4e85-bbe0-fdc36af6f192-istio-envoy\") pod \"43a58ae9-e100-4e85-bbe0-fdc36af6f192\" (UID: \"43a58ae9-e100-4e85-bbe0-fdc36af6f192\") " Nov 04 03:06:46.382087 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:06:46.382068 4405 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/43a58ae9-e100-4e85-bbe0-fdc36af6f192-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/43a58ae9-e100-4e85-bbe0-fdc36af6f192-credential-socket\") pod \"43a58ae9-e100-4e85-bbe0-fdc36af6f192\" (UID: \"43a58ae9-e100-4e85-bbe0-fdc36af6f192\") " Nov 04 03:06:46.382157 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:06:46.382105 4405 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f2c5af93-085e-462b-8377-a6e8139e66c8-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f2c5af93-085e-462b-8377-a6e8139e66c8-istio-data\") pod \"f2c5af93-085e-462b-8377-a6e8139e66c8\" (UID: \"f2c5af93-085e-462b-8377-a6e8139e66c8\") " Nov 04 03:06:46.382157 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:06:46.382141 4405 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/43a58ae9-e100-4e85-bbe0-fdc36af6f192-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/43a58ae9-e100-4e85-bbe0-fdc36af6f192-workload-certs\") pod \"43a58ae9-e100-4e85-bbe0-fdc36af6f192\" (UID: \"43a58ae9-e100-4e85-bbe0-fdc36af6f192\") " Nov 04 03:06:46.389716 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:06:46.382310 4405 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f2c5af93-085e-462b-8377-a6e8139e66c8-istio-token\" (UniqueName: \"kubernetes.io/projected/f2c5af93-085e-462b-8377-a6e8139e66c8-istio-token\") pod \"f2c5af93-085e-462b-8377-a6e8139e66c8\" (UID: \"f2c5af93-085e-462b-8377-a6e8139e66c8\") " Nov 04 03:06:46.389716 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:06:46.382939 4405 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f2c5af93-085e-462b-8377-a6e8139e66c8-istio-data" pod "f2c5af93-085e-462b-8377-a6e8139e66c8" (UID: "f2c5af93-085e-462b-8377-a6e8139e66c8"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 03:06:46.389716 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:06:46.383051 4405 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/43a58ae9-e100-4e85-bbe0-fdc36af6f192-istiod-ca-cert" pod "43a58ae9-e100-4e85-bbe0-fdc36af6f192" (UID: "43a58ae9-e100-4e85-bbe0-fdc36af6f192"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 03:06:46.389716 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:06:46.383200 4405 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f2c5af93-085e-462b-8377-a6e8139e66c8-istiod-ca-cert" pod "f2c5af93-085e-462b-8377-a6e8139e66c8" (UID: "f2c5af93-085e-462b-8377-a6e8139e66c8"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 03:06:46.389716 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:06:46.383119 4405 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f2c5af93-085e-462b-8377-a6e8139e66c8-workload-certs" pod "f2c5af93-085e-462b-8377-a6e8139e66c8" (UID: "f2c5af93-085e-462b-8377-a6e8139e66c8"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 03:06:46.389716 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:06:46.383438 4405 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/43a58ae9-e100-4e85-bbe0-fdc36af6f192-workload-socket" pod "43a58ae9-e100-4e85-bbe0-fdc36af6f192" (UID: "43a58ae9-e100-4e85-bbe0-fdc36af6f192"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 03:06:46.389716 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:06:46.383538 4405 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/43a58ae9-e100-4e85-bbe0-fdc36af6f192-workload-certs" pod "43a58ae9-e100-4e85-bbe0-fdc36af6f192" (UID: "43a58ae9-e100-4e85-bbe0-fdc36af6f192"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 03:06:46.389716 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:06:46.383583 4405 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/43a58ae9-e100-4e85-bbe0-fdc36af6f192-credential-socket" pod "43a58ae9-e100-4e85-bbe0-fdc36af6f192" (UID: "43a58ae9-e100-4e85-bbe0-fdc36af6f192"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 03:06:46.389716 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:06:46.383710 4405 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f2c5af93-085e-462b-8377-a6e8139e66c8-workload-socket" pod "f2c5af93-085e-462b-8377-a6e8139e66c8" (UID: "f2c5af93-085e-462b-8377-a6e8139e66c8"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 03:06:46.389716 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:06:46.383843 4405 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f2c5af93-085e-462b-8377-a6e8139e66c8-credential-socket" pod "f2c5af93-085e-462b-8377-a6e8139e66c8" (UID: "f2c5af93-085e-462b-8377-a6e8139e66c8"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 03:06:46.389716 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:06:46.384353 4405 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/43a58ae9-e100-4e85-bbe0-fdc36af6f192-credential-socket\") on node \"ip-10-0-134-125.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:06:46.389716 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:06:46.384378 4405 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f2c5af93-085e-462b-8377-a6e8139e66c8-istio-data\") on node \"ip-10-0-134-125.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:06:46.389716 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:06:46.384393 4405 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/43a58ae9-e100-4e85-bbe0-fdc36af6f192-workload-certs\") on node \"ip-10-0-134-125.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:06:46.389716 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:06:46.384406 4405 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f2c5af93-085e-462b-8377-a6e8139e66c8-workload-certs\") on node \"ip-10-0-134-125.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:06:46.389716 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:06:46.384420 4405 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/43a58ae9-e100-4e85-bbe0-fdc36af6f192-istiod-ca-cert\") on node \"ip-10-0-134-125.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:06:46.389716 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:06:46.384434 4405 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f2c5af93-085e-462b-8377-a6e8139e66c8-credential-socket\") on node \"ip-10-0-134-125.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:06:46.389716 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:06:46.384447 4405 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f2c5af93-085e-462b-8377-a6e8139e66c8-workload-socket\") on node \"ip-10-0-134-125.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:06:46.389716 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:06:46.384462 4405 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f2c5af93-085e-462b-8377-a6e8139e66c8-istiod-ca-cert\") on node \"ip-10-0-134-125.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:06:46.389716 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:06:46.384476 4405 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/43a58ae9-e100-4e85-bbe0-fdc36af6f192-workload-socket\") on node \"ip-10-0-134-125.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:06:46.389716 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:06:46.384762 4405 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/43a58ae9-e100-4e85-bbe0-fdc36af6f192-istio-data" pod "43a58ae9-e100-4e85-bbe0-fdc36af6f192" (UID: "43a58ae9-e100-4e85-bbe0-fdc36af6f192"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 03:06:46.389716 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:06:46.389455 4405 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/43a58ae9-e100-4e85-bbe0-fdc36af6f192-istio-podinfo" pod "43a58ae9-e100-4e85-bbe0-fdc36af6f192" (UID: "43a58ae9-e100-4e85-bbe0-fdc36af6f192"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 04 03:06:46.389716 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:06:46.389457 4405 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f2c5af93-085e-462b-8377-a6e8139e66c8-istio-token" pod "f2c5af93-085e-462b-8377-a6e8139e66c8" (UID: "f2c5af93-085e-462b-8377-a6e8139e66c8"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 03:06:46.389716 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:06:46.389482 4405 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/43a58ae9-e100-4e85-bbe0-fdc36af6f192-istio-token" pod "43a58ae9-e100-4e85-bbe0-fdc36af6f192" (UID: "43a58ae9-e100-4e85-bbe0-fdc36af6f192"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 03:06:46.389716 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:06:46.389496 4405 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f2c5af93-085e-462b-8377-a6e8139e66c8-kube-api-access-284m6" pod "f2c5af93-085e-462b-8377-a6e8139e66c8" (UID: "f2c5af93-085e-462b-8377-a6e8139e66c8"). InnerVolumeSpecName "kube-api-access-284m6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 03:06:46.389716 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:06:46.389530 4405 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f2c5af93-085e-462b-8377-a6e8139e66c8-istio-envoy" pod "f2c5af93-085e-462b-8377-a6e8139e66c8" (UID: "f2c5af93-085e-462b-8377-a6e8139e66c8"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 03:06:46.391269 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:06:46.389838 4405 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/43a58ae9-e100-4e85-bbe0-fdc36af6f192-istio-envoy" pod "43a58ae9-e100-4e85-bbe0-fdc36af6f192" (UID: "43a58ae9-e100-4e85-bbe0-fdc36af6f192"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 03:06:46.391437 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:06:46.391410 4405 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f2c5af93-085e-462b-8377-a6e8139e66c8-istio-podinfo" pod "f2c5af93-085e-462b-8377-a6e8139e66c8" (UID: "f2c5af93-085e-462b-8377-a6e8139e66c8"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 04 03:06:46.391514 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:06:46.391467 4405 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/43a58ae9-e100-4e85-bbe0-fdc36af6f192-kube-api-access-9ddrz" pod "43a58ae9-e100-4e85-bbe0-fdc36af6f192" (UID: "43a58ae9-e100-4e85-bbe0-fdc36af6f192"). InnerVolumeSpecName "kube-api-access-9ddrz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 03:06:46.490505 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:06:46.489297 4405 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f2c5af93-085e-462b-8377-a6e8139e66c8-istio-token\") on node \"ip-10-0-134-125.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:06:46.490505 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:06:46.489338 4405 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-284m6\" (UniqueName: \"kubernetes.io/projected/f2c5af93-085e-462b-8377-a6e8139e66c8-kube-api-access-284m6\") on node \"ip-10-0-134-125.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:06:46.490505 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:06:46.489356 4405 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f2c5af93-085e-462b-8377-a6e8139e66c8-istio-podinfo\") on node \"ip-10-0-134-125.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:06:46.490505 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:06:46.489370 4405 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/43a58ae9-e100-4e85-bbe0-fdc36af6f192-istio-token\") on node \"ip-10-0-134-125.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:06:46.490505 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:06:46.489385 4405 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/43a58ae9-e100-4e85-bbe0-fdc36af6f192-istio-data\") on node \"ip-10-0-134-125.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:06:46.490505 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:06:46.489398 4405 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/43a58ae9-e100-4e85-bbe0-fdc36af6f192-istio-podinfo\") on node \"ip-10-0-134-125.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:06:46.490505 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:06:46.489412 4405 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f2c5af93-085e-462b-8377-a6e8139e66c8-istio-envoy\") on node \"ip-10-0-134-125.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:06:46.490505 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:06:46.489481 4405 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9ddrz\" (UniqueName: \"kubernetes.io/projected/43a58ae9-e100-4e85-bbe0-fdc36af6f192-kube-api-access-9ddrz\") on node \"ip-10-0-134-125.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:06:46.490505 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:06:46.489523 4405 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/43a58ae9-e100-4e85-bbe0-fdc36af6f192-istio-envoy\") on node \"ip-10-0-134-125.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:06:47.088355 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:06:47.088321 4405 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-3b7476-openshift-default-59975457bf-vxwkm" Nov 04 03:06:47.088355 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:06:47.088337 4405 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-3b7476-openshift-default-59975457bf-vxwkm" event={"ID":"f2c5af93-085e-462b-8377-a6e8139e66c8","Type":"ContainerDied","Data":"05f95d319febc023166e3161b4d00c1f4193b61800628c9176f51e493753377e"} Nov 04 03:06:47.088969 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:06:47.088407 4405 scope.go:122] "RemoveContainer" containerID="fd6959d2c3e3f6141a6342ce834069df4084a2f029a4812d29904c9c0ac88013" Nov 04 03:06:47.090838 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:06:47.090811 4405 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-3b7476-openshift-default-59975457bf-gph88" event={"ID":"43a58ae9-e100-4e85-bbe0-fdc36af6f192","Type":"ContainerDied","Data":"1bb94f627433b1d2903194bb1ecd293f3762ced4bb07cf9c29a70a438528dc9c"} Nov 04 03:06:47.090997 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:06:47.090910 4405 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-3b7476-openshift-default-59975457bf-gph88" Nov 04 03:06:47.102562 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:06:47.102542 4405 scope.go:122] "RemoveContainer" containerID="c64cebb6a88c524d1824f445d30518a0d02ce7ef085793001775e7011068f093" Nov 04 03:06:47.255850 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:06:47.255768 4405 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-3b7476-openshift-default-59975457bf-gph88"] Nov 04 03:06:47.307584 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:06:47.307547 4405 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w5-3b7476-openshift-default-59975457bf-gph88"] Nov 04 03:06:47.403351 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:06:47.403317 4405 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-3b7476-openshift-default-59975457bf-vxwkm"] Nov 04 03:06:47.435820 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:06:47.435787 4405 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w5-3b7476-openshift-default-59975457bf-vxwkm"] Nov 04 03:06:47.704226 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:06:47.704193 4405 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="43a58ae9-e100-4e85-bbe0-fdc36af6f192" path="/var/lib/kubelet/pods/43a58ae9-e100-4e85-bbe0-fdc36af6f192/volumes" Nov 04 03:06:47.704799 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:06:47.704774 4405 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f2c5af93-085e-462b-8377-a6e8139e66c8" path="/var/lib/kubelet/pods/f2c5af93-085e-462b-8377-a6e8139e66c8/volumes" Nov 04 03:06:49.203877 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:06:49.203839 4405 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-cdb11d-87b64cbd4-bvfpd"] Nov 04 03:06:49.318053 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:06:49.318019 4405 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-cdb11d-585c6fc6dc-dtbnf"] Nov 04 03:06:52.621218 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:06:52.621184 4405 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-9d2b96-7d4bf9f94-959mx"] Nov 04 03:06:52.621915 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:06:52.621889 4405 cpu_manager.go:414] "containerMap: removing container" podUID="23294ab6-3eaf-41fd-8b5b-c7a0214bd532" containerName="discovery" Nov 04 03:06:52.621915 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:06:52.621908 4405 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="23294ab6-3eaf-41fd-8b5b-c7a0214bd532" containerName="discovery" Nov 04 03:06:52.621915 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:06:52.621922 4405 cpu_manager.go:414] "containerMap: removing container" podUID="43a58ae9-e100-4e85-bbe0-fdc36af6f192" containerName="istio-proxy" Nov 04 03:06:52.622170 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:06:52.621931 4405 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="43a58ae9-e100-4e85-bbe0-fdc36af6f192" containerName="istio-proxy" Nov 04 03:06:52.622170 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:06:52.621946 4405 cpu_manager.go:414] "containerMap: removing container" podUID="e5ccd15d-11fd-4f67-9bd5-7a4cfa35d602" containerName="istio-proxy" Nov 04 03:06:52.622170 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:06:52.621952 4405 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e5ccd15d-11fd-4f67-9bd5-7a4cfa35d602" containerName="istio-proxy" Nov 04 03:06:52.622170 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:06:52.621963 4405 cpu_manager.go:414] "containerMap: removing container" podUID="f2c5af93-085e-462b-8377-a6e8139e66c8" containerName="istio-proxy" Nov 04 03:06:52.622170 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:06:52.621968 4405 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f2c5af93-085e-462b-8377-a6e8139e66c8" containerName="istio-proxy" Nov 04 03:06:52.622170 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:06:52.622092 4405 memory_manager.go:365] "RemoveStaleState removing state" podUID="23294ab6-3eaf-41fd-8b5b-c7a0214bd532" containerName="discovery" Nov 04 03:06:52.622170 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:06:52.622100 4405 memory_manager.go:365] "RemoveStaleState removing state" podUID="43a58ae9-e100-4e85-bbe0-fdc36af6f192" containerName="istio-proxy" Nov 04 03:06:52.622170 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:06:52.622106 4405 memory_manager.go:365] "RemoveStaleState removing state" podUID="e5ccd15d-11fd-4f67-9bd5-7a4cfa35d602" containerName="istio-proxy" Nov 04 03:06:52.622170 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:06:52.622114 4405 memory_manager.go:365] "RemoveStaleState removing state" podUID="f2c5af93-085e-462b-8377-a6e8139e66c8" containerName="istio-proxy" Nov 04 03:06:52.627086 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:06:52.627068 4405 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-9d2b96-7d4bf9f94-959mx" Nov 04 03:06:52.725079 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:06:52.725040 4405 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-22mx6\" (UniqueName: \"kubernetes.io/projected/43f2feb2-2005-4eca-a400-0d66bd1a4aac-kube-api-access-22mx6\") pod \"maas-api-e2e-api-b-9d2b96-7d4bf9f94-959mx\" (UID: \"43f2feb2-2005-4eca-a400-0d66bd1a4aac\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-9d2b96-7d4bf9f94-959mx" Nov 04 03:06:52.725255 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:06:52.725121 4405 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/43f2feb2-2005-4eca-a400-0d66bd1a4aac-maas-api-tls\") pod \"maas-api-e2e-api-b-9d2b96-7d4bf9f94-959mx\" (UID: \"43f2feb2-2005-4eca-a400-0d66bd1a4aac\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-9d2b96-7d4bf9f94-959mx" Nov 04 03:06:52.725312 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:06:52.725284 4405 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/43f2feb2-2005-4eca-a400-0d66bd1a4aac-openshift-service-ca\") pod \"maas-api-e2e-api-b-9d2b96-7d4bf9f94-959mx\" (UID: \"43f2feb2-2005-4eca-a400-0d66bd1a4aac\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-9d2b96-7d4bf9f94-959mx" Nov 04 03:06:52.725463 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:06:52.725444 4405 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/43f2feb2-2005-4eca-a400-0d66bd1a4aac-metrics-certs\") pod \"maas-api-e2e-api-b-9d2b96-7d4bf9f94-959mx\" (UID: \"43f2feb2-2005-4eca-a400-0d66bd1a4aac\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-9d2b96-7d4bf9f94-959mx" Nov 04 03:06:52.728967 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:06:52.728942 4405 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-b-9d2b96-serving-cert\"" Nov 04 03:06:52.733024 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:06:52.733001 4405 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-9d2b96-7d4bf9f94-959mx"] Nov 04 03:06:52.826698 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:06:52.826632 4405 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-22mx6\" (UniqueName: \"kubernetes.io/projected/43f2feb2-2005-4eca-a400-0d66bd1a4aac-kube-api-access-22mx6\") pod \"maas-api-e2e-api-b-9d2b96-7d4bf9f94-959mx\" (UID: \"43f2feb2-2005-4eca-a400-0d66bd1a4aac\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-9d2b96-7d4bf9f94-959mx" Nov 04 03:06:52.826882 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:06:52.826733 4405 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/43f2feb2-2005-4eca-a400-0d66bd1a4aac-maas-api-tls\") pod \"maas-api-e2e-api-b-9d2b96-7d4bf9f94-959mx\" (UID: \"43f2feb2-2005-4eca-a400-0d66bd1a4aac\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-9d2b96-7d4bf9f94-959mx" Nov 04 03:06:52.826882 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:06:52.826854 4405 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/43f2feb2-2005-4eca-a400-0d66bd1a4aac-openshift-service-ca\") pod \"maas-api-e2e-api-b-9d2b96-7d4bf9f94-959mx\" (UID: \"43f2feb2-2005-4eca-a400-0d66bd1a4aac\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-9d2b96-7d4bf9f94-959mx" Nov 04 03:06:52.827024 ip-10-0-134-125 kubenswrapper[4405]: E1104 03:06:52.826872 4405 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-api-b-9d2b96-serving-cert: secret "maas-api-e2e-api-b-9d2b96-serving-cert" not found Nov 04 03:06:52.827024 ip-10-0-134-125 kubenswrapper[4405]: E1104 03:06:52.826942 4405 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/43f2feb2-2005-4eca-a400-0d66bd1a4aac-maas-api-tls podName:43f2feb2-2005-4eca-a400-0d66bd1a4aac nodeName:}" failed. No retries permitted until 2026-11-04 03:06:53.32692191 +0000 UTC m=+2059.757622012 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/43f2feb2-2005-4eca-a400-0d66bd1a4aac-maas-api-tls") pod "maas-api-e2e-api-b-9d2b96-7d4bf9f94-959mx" (UID: "43f2feb2-2005-4eca-a400-0d66bd1a4aac") : secret "maas-api-e2e-api-b-9d2b96-serving-cert" not found Nov 04 03:06:52.827139 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:06:52.827060 4405 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/43f2feb2-2005-4eca-a400-0d66bd1a4aac-metrics-certs\") pod \"maas-api-e2e-api-b-9d2b96-7d4bf9f94-959mx\" (UID: \"43f2feb2-2005-4eca-a400-0d66bd1a4aac\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-9d2b96-7d4bf9f94-959mx" Nov 04 03:06:52.827875 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:06:52.827853 4405 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/43f2feb2-2005-4eca-a400-0d66bd1a4aac-openshift-service-ca\") pod \"maas-api-e2e-api-b-9d2b96-7d4bf9f94-959mx\" (UID: \"43f2feb2-2005-4eca-a400-0d66bd1a4aac\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-9d2b96-7d4bf9f94-959mx" Nov 04 03:06:52.829917 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:06:52.829895 4405 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/43f2feb2-2005-4eca-a400-0d66bd1a4aac-metrics-certs\") pod \"maas-api-e2e-api-b-9d2b96-7d4bf9f94-959mx\" (UID: \"43f2feb2-2005-4eca-a400-0d66bd1a4aac\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-9d2b96-7d4bf9f94-959mx" Nov 04 03:06:52.902745 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:06:52.901967 4405 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-22mx6\" (UniqueName: \"kubernetes.io/projected/43f2feb2-2005-4eca-a400-0d66bd1a4aac-kube-api-access-22mx6\") pod \"maas-api-e2e-api-b-9d2b96-7d4bf9f94-959mx\" (UID: \"43f2feb2-2005-4eca-a400-0d66bd1a4aac\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-9d2b96-7d4bf9f94-959mx" Nov 04 03:06:53.336122 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:06:53.336080 4405 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/43f2feb2-2005-4eca-a400-0d66bd1a4aac-maas-api-tls\") pod \"maas-api-e2e-api-b-9d2b96-7d4bf9f94-959mx\" (UID: \"43f2feb2-2005-4eca-a400-0d66bd1a4aac\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-9d2b96-7d4bf9f94-959mx" Nov 04 03:06:53.338734 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:06:53.338711 4405 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/43f2feb2-2005-4eca-a400-0d66bd1a4aac-maas-api-tls\") pod \"maas-api-e2e-api-b-9d2b96-7d4bf9f94-959mx\" (UID: \"43f2feb2-2005-4eca-a400-0d66bd1a4aac\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-9d2b96-7d4bf9f94-959mx" Nov 04 03:06:53.541729 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:06:53.541655 4405 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-9d2b96-7d4bf9f94-959mx" Nov 04 03:06:53.923330 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:06:53.923298 4405 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-9d2b96-7d4bf9f94-959mx"] Nov 04 03:06:53.924157 ip-10-0-134-125 kubenswrapper[4405]: W1104 03:06:53.924129 4405 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod43f2feb2_2005_4eca_a400_0d66bd1a4aac.slice/crio-721d215bc3c00f8300a925455688b77b0b8b8145f25fa4402c56052e1cc41f32 WatchSource:0}: Error finding container 721d215bc3c00f8300a925455688b77b0b8b8145f25fa4402c56052e1cc41f32: Status 404 returned error can't find the container with id 721d215bc3c00f8300a925455688b77b0b8b8145f25fa4402c56052e1cc41f32 Nov 04 03:06:54.146523 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:06:54.146487 4405 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-9d2b96-7d4bf9f94-959mx" event={"ID":"43f2feb2-2005-4eca-a400-0d66bd1a4aac","Type":"ContainerStarted","Data":"721d215bc3c00f8300a925455688b77b0b8b8145f25fa4402c56052e1cc41f32"} Nov 04 03:06:54.194439 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:06:54.194354 4405 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-fe4e0c-openshift-default-c7f4fc84c-mmngc"] Nov 04 03:06:54.194587 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:06:54.194550 4405 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-fe4e0c-openshift-default-c7f4fc84c-mmngc" podUID="8d37c0a6-e58b-4f0b-a043-6184b537a719" containerName="istio-proxy" containerID="cri-o://9d1aa8164135250664e9340506c6a3ecc0d38105e5bb63b6ebb9d6e8de2d0a91" gracePeriod=30 Nov 04 03:06:54.205076 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:06:54.205042 4405 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-a-cdb11d-87b64cbd4-bvfpd" podUID="7acf0cd1-800b-4796-9b10-a2041f971104" containerName="payload-processing" containerID="cri-o://8bb720159944d524388822d75ceeefccdf1a72e4c5a6e2e792b273bbe538c932" gracePeriod=25 Nov 04 03:06:54.319146 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:06:54.319107 4405 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-a-cdb11d-585c6fc6dc-dtbnf" podUID="b7f09efb-5b70-457a-93c0-9125cae4c2dc" containerName="payload-pre-processing" containerID="cri-o://2e551a9e5311105bb05d1c69ee90c56d56381ee1193ec9e88484c06c96164099" gracePeriod=25 Nov 04 03:06:54.510933 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:06:54.510882 4405 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-cdb11d-87b64cbd4-bvfpd" Nov 04 03:06:54.556829 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:06:54.556794 4405 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7acf0cd1-800b-4796-9b10-a2041f971104-kube-api-access-gmqnc\" (UniqueName: \"kubernetes.io/projected/7acf0cd1-800b-4796-9b10-a2041f971104-kube-api-access-gmqnc\") pod \"7acf0cd1-800b-4796-9b10-a2041f971104\" (UID: \"7acf0cd1-800b-4796-9b10-a2041f971104\") " Nov 04 03:06:54.557090 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:06:54.557038 4405 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7acf0cd1-800b-4796-9b10-a2041f971104-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7acf0cd1-800b-4796-9b10-a2041f971104-plugins-config-volume\") pod \"7acf0cd1-800b-4796-9b10-a2041f971104\" (UID: \"7acf0cd1-800b-4796-9b10-a2041f971104\") " Nov 04 03:06:54.558043 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:06:54.557993 4405 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7acf0cd1-800b-4796-9b10-a2041f971104-plugins-config-volume" pod "7acf0cd1-800b-4796-9b10-a2041f971104" (UID: "7acf0cd1-800b-4796-9b10-a2041f971104"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 03:06:54.560580 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:06:54.560542 4405 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7acf0cd1-800b-4796-9b10-a2041f971104-kube-api-access-gmqnc" pod "7acf0cd1-800b-4796-9b10-a2041f971104" (UID: "7acf0cd1-800b-4796-9b10-a2041f971104"). InnerVolumeSpecName "kube-api-access-gmqnc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 03:06:54.609453 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:06:54.609429 4405 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-cdb11d-585c6fc6dc-dtbnf" Nov 04 03:06:54.659090 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:06:54.659048 4405 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b7f09efb-5b70-457a-93c0-9125cae4c2dc-kube-api-access-glhr5\" (UniqueName: \"kubernetes.io/projected/b7f09efb-5b70-457a-93c0-9125cae4c2dc-kube-api-access-glhr5\") pod \"b7f09efb-5b70-457a-93c0-9125cae4c2dc\" (UID: \"b7f09efb-5b70-457a-93c0-9125cae4c2dc\") " Nov 04 03:06:54.659311 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:06:54.659187 4405 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b7f09efb-5b70-457a-93c0-9125cae4c2dc-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b7f09efb-5b70-457a-93c0-9125cae4c2dc-plugins-config-volume\") pod \"b7f09efb-5b70-457a-93c0-9125cae4c2dc\" (UID: \"b7f09efb-5b70-457a-93c0-9125cae4c2dc\") " Nov 04 03:06:54.659726 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:06:54.659641 4405 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b7f09efb-5b70-457a-93c0-9125cae4c2dc-plugins-config-volume" pod "b7f09efb-5b70-457a-93c0-9125cae4c2dc" (UID: "b7f09efb-5b70-457a-93c0-9125cae4c2dc"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 03:06:54.659867 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:06:54.659752 4405 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7acf0cd1-800b-4796-9b10-a2041f971104-plugins-config-volume\") on node \"ip-10-0-134-125.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:06:54.659867 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:06:54.659774 4405 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gmqnc\" (UniqueName: \"kubernetes.io/projected/7acf0cd1-800b-4796-9b10-a2041f971104-kube-api-access-gmqnc\") on node \"ip-10-0-134-125.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:06:54.661715 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:06:54.661649 4405 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b7f09efb-5b70-457a-93c0-9125cae4c2dc-kube-api-access-glhr5" pod "b7f09efb-5b70-457a-93c0-9125cae4c2dc" (UID: "b7f09efb-5b70-457a-93c0-9125cae4c2dc"). InnerVolumeSpecName "kube-api-access-glhr5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 03:06:54.761368 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:06:54.761276 4405 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-glhr5\" (UniqueName: \"kubernetes.io/projected/b7f09efb-5b70-457a-93c0-9125cae4c2dc-kube-api-access-glhr5\") on node \"ip-10-0-134-125.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:06:54.761368 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:06:54.761305 4405 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b7f09efb-5b70-457a-93c0-9125cae4c2dc-plugins-config-volume\") on node \"ip-10-0-134-125.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:06:55.155867 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:06:55.155826 4405 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-9d2b96-7d4bf9f94-959mx" event={"ID":"43f2feb2-2005-4eca-a400-0d66bd1a4aac","Type":"ContainerStarted","Data":"f42f027d5f382c7bc439859a5e1264080e1a2829513ebed784760e7830fef49f"} Nov 04 03:06:55.156344 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:06:55.155930 4405 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-9d2b96-7d4bf9f94-959mx" Nov 04 03:06:55.157681 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:06:55.157644 4405 generic.go:356] "Generic (PLEG): container finished" podID="b7f09efb-5b70-457a-93c0-9125cae4c2dc" containerID="2e551a9e5311105bb05d1c69ee90c56d56381ee1193ec9e88484c06c96164099" exitCode=0 Nov 04 03:06:55.157823 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:06:55.157714 4405 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-cdb11d-585c6fc6dc-dtbnf" Nov 04 03:06:55.157823 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:06:55.157729 4405 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-cdb11d-585c6fc6dc-dtbnf" event={"ID":"b7f09efb-5b70-457a-93c0-9125cae4c2dc","Type":"ContainerDied","Data":"2e551a9e5311105bb05d1c69ee90c56d56381ee1193ec9e88484c06c96164099"} Nov 04 03:06:55.157823 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:06:55.157775 4405 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-cdb11d-585c6fc6dc-dtbnf" event={"ID":"b7f09efb-5b70-457a-93c0-9125cae4c2dc","Type":"ContainerDied","Data":"7fcf14dd15fea81fe8a3cc7735eadcec747ffc8baa65df0bc0d502a9b5bb26f2"} Nov 04 03:06:55.157823 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:06:55.157803 4405 scope.go:122] "RemoveContainer" containerID="2e551a9e5311105bb05d1c69ee90c56d56381ee1193ec9e88484c06c96164099" Nov 04 03:06:55.159406 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:06:55.159384 4405 generic.go:356] "Generic (PLEG): container finished" podID="7acf0cd1-800b-4796-9b10-a2041f971104" containerID="8bb720159944d524388822d75ceeefccdf1a72e4c5a6e2e792b273bbe538c932" exitCode=0 Nov 04 03:06:55.159505 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:06:55.159435 4405 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-cdb11d-87b64cbd4-bvfpd" event={"ID":"7acf0cd1-800b-4796-9b10-a2041f971104","Type":"ContainerDied","Data":"8bb720159944d524388822d75ceeefccdf1a72e4c5a6e2e792b273bbe538c932"} Nov 04 03:06:55.159505 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:06:55.159469 4405 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-cdb11d-87b64cbd4-bvfpd" event={"ID":"7acf0cd1-800b-4796-9b10-a2041f971104","Type":"ContainerDied","Data":"86099ed10dce718f0e598ca98b5eb9aa2d625911c6a40428a511771e16a2022c"} Nov 04 03:06:55.159591 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:06:55.159526 4405 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-cdb11d-87b64cbd4-bvfpd" Nov 04 03:06:55.173452 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:06:55.172897 4405 scope.go:122] "RemoveContainer" containerID="2e551a9e5311105bb05d1c69ee90c56d56381ee1193ec9e88484c06c96164099" Nov 04 03:06:55.175128 ip-10-0-134-125 kubenswrapper[4405]: E1104 03:06:55.175103 4405 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2e551a9e5311105bb05d1c69ee90c56d56381ee1193ec9e88484c06c96164099\": container with ID starting with 2e551a9e5311105bb05d1c69ee90c56d56381ee1193ec9e88484c06c96164099 not found: ID does not exist" containerID="2e551a9e5311105bb05d1c69ee90c56d56381ee1193ec9e88484c06c96164099" Nov 04 03:06:55.175259 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:06:55.175135 4405 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2e551a9e5311105bb05d1c69ee90c56d56381ee1193ec9e88484c06c96164099"} err="failed to get container status \"2e551a9e5311105bb05d1c69ee90c56d56381ee1193ec9e88484c06c96164099\": rpc error: code = NotFound desc = could not find container \"2e551a9e5311105bb05d1c69ee90c56d56381ee1193ec9e88484c06c96164099\": container with ID starting with 2e551a9e5311105bb05d1c69ee90c56d56381ee1193ec9e88484c06c96164099 not found: ID does not exist" Nov 04 03:06:55.175259 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:06:55.175154 4405 scope.go:122] "RemoveContainer" containerID="8bb720159944d524388822d75ceeefccdf1a72e4c5a6e2e792b273bbe538c932" Nov 04 03:06:55.187749 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:06:55.187729 4405 scope.go:122] "RemoveContainer" containerID="8bb720159944d524388822d75ceeefccdf1a72e4c5a6e2e792b273bbe538c932" Nov 04 03:06:55.188062 ip-10-0-134-125 kubenswrapper[4405]: E1104 03:06:55.188041 4405 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8bb720159944d524388822d75ceeefccdf1a72e4c5a6e2e792b273bbe538c932\": container with ID starting with 8bb720159944d524388822d75ceeefccdf1a72e4c5a6e2e792b273bbe538c932 not found: ID does not exist" containerID="8bb720159944d524388822d75ceeefccdf1a72e4c5a6e2e792b273bbe538c932" Nov 04 03:06:55.188132 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:06:55.188073 4405 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8bb720159944d524388822d75ceeefccdf1a72e4c5a6e2e792b273bbe538c932"} err="failed to get container status \"8bb720159944d524388822d75ceeefccdf1a72e4c5a6e2e792b273bbe538c932\": rpc error: code = NotFound desc = could not find container \"8bb720159944d524388822d75ceeefccdf1a72e4c5a6e2e792b273bbe538c932\": container with ID starting with 8bb720159944d524388822d75ceeefccdf1a72e4c5a6e2e792b273bbe538c932 not found: ID does not exist" Nov 04 03:06:55.277456 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:06:55.277394 4405 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-9d2b96-7d4bf9f94-959mx" podStartSLOduration=2.364559751 podStartE2EDuration="3.277372165s" totalImagesPullingTime="912.812414ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 03:06:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 03:06:55.27306895 +0000 UTC m=+2061.703769070" watchObservedRunningTime="2026-11-04 03:06:55.277372165 +0000 UTC m=+2061.708072287" Nov 04 03:06:55.386122 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:06:55.386086 4405 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-cdb11d-87b64cbd4-bvfpd"] Nov 04 03:06:55.390920 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:06:55.390884 4405 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-cdb11d-87b64cbd4-bvfpd"] Nov 04 03:06:55.499637 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:06:55.499544 4405 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-cdb11d-585c6fc6dc-dtbnf"] Nov 04 03:06:55.504890 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:06:55.504859 4405 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-cdb11d-585c6fc6dc-dtbnf"] Nov 04 03:06:55.748694 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:06:55.741078 4405 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7acf0cd1-800b-4796-9b10-a2041f971104" path="/var/lib/kubelet/pods/7acf0cd1-800b-4796-9b10-a2041f971104/volumes" Nov 04 03:06:55.748694 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:06:55.741856 4405 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b7f09efb-5b70-457a-93c0-9125cae4c2dc" path="/var/lib/kubelet/pods/b7f09efb-5b70-457a-93c0-9125cae4c2dc/volumes" Nov 04 03:06:56.724192 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:06:56.724158 4405 patch_prober.go:29] interesting pod/e2e-worker-w0-fe4e0c-openshift-default-c7f4fc84c-mmngc container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 04 03:06:56.724662 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:06:56.724221 4405 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-fe4e0c-openshift-default-c7f4fc84c-mmngc" podUID="8d37c0a6-e58b-4f0b-a043-6184b537a719" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 04 03:06:59.478979 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:06:59.478906 4405 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-fe4e0c-openshift-default-c7f4fc84c-mmngc" Nov 04 03:06:59.538734 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:06:59.538703 4405 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8d37c0a6-e58b-4f0b-a043-6184b537a719-istio-token\" (UniqueName: \"kubernetes.io/projected/8d37c0a6-e58b-4f0b-a043-6184b537a719-istio-token\") pod \"8d37c0a6-e58b-4f0b-a043-6184b537a719\" (UID: \"8d37c0a6-e58b-4f0b-a043-6184b537a719\") " Nov 04 03:06:59.538944 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:06:59.538742 4405 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8d37c0a6-e58b-4f0b-a043-6184b537a719-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8d37c0a6-e58b-4f0b-a043-6184b537a719-workload-socket\") pod \"8d37c0a6-e58b-4f0b-a043-6184b537a719\" (UID: \"8d37c0a6-e58b-4f0b-a043-6184b537a719\") " Nov 04 03:06:59.538944 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:06:59.538818 4405 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/8d37c0a6-e58b-4f0b-a043-6184b537a719-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8d37c0a6-e58b-4f0b-a043-6184b537a719-istio-podinfo\") pod \"8d37c0a6-e58b-4f0b-a043-6184b537a719\" (UID: \"8d37c0a6-e58b-4f0b-a043-6184b537a719\") " Nov 04 03:06:59.538944 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:06:59.538854 4405 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8d37c0a6-e58b-4f0b-a043-6184b537a719-kube-api-access-7vvw8\" (UniqueName: \"kubernetes.io/projected/8d37c0a6-e58b-4f0b-a043-6184b537a719-kube-api-access-7vvw8\") pod \"8d37c0a6-e58b-4f0b-a043-6184b537a719\" (UID: \"8d37c0a6-e58b-4f0b-a043-6184b537a719\") " Nov 04 03:06:59.538944 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:06:59.538915 4405 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8d37c0a6-e58b-4f0b-a043-6184b537a719-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8d37c0a6-e58b-4f0b-a043-6184b537a719-workload-certs\") pod \"8d37c0a6-e58b-4f0b-a043-6184b537a719\" (UID: \"8d37c0a6-e58b-4f0b-a043-6184b537a719\") " Nov 04 03:06:59.539165 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:06:59.538978 4405 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8d37c0a6-e58b-4f0b-a043-6184b537a719-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8d37c0a6-e58b-4f0b-a043-6184b537a719-istio-data\") pod \"8d37c0a6-e58b-4f0b-a043-6184b537a719\" (UID: \"8d37c0a6-e58b-4f0b-a043-6184b537a719\") " Nov 04 03:06:59.539165 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:06:59.539117 4405 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8d37c0a6-e58b-4f0b-a043-6184b537a719-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8d37c0a6-e58b-4f0b-a043-6184b537a719-istio-envoy\") pod \"8d37c0a6-e58b-4f0b-a043-6184b537a719\" (UID: \"8d37c0a6-e58b-4f0b-a043-6184b537a719\") " Nov 04 03:06:59.539165 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:06:59.539147 4405 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8d37c0a6-e58b-4f0b-a043-6184b537a719-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8d37c0a6-e58b-4f0b-a043-6184b537a719-istiod-ca-cert\") pod \"8d37c0a6-e58b-4f0b-a043-6184b537a719\" (UID: \"8d37c0a6-e58b-4f0b-a043-6184b537a719\") " Nov 04 03:06:59.539165 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:06:59.539149 4405 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8d37c0a6-e58b-4f0b-a043-6184b537a719-workload-socket" pod "8d37c0a6-e58b-4f0b-a043-6184b537a719" (UID: "8d37c0a6-e58b-4f0b-a043-6184b537a719"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 03:06:59.539369 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:06:59.539266 4405 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8d37c0a6-e58b-4f0b-a043-6184b537a719-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8d37c0a6-e58b-4f0b-a043-6184b537a719-credential-socket\") pod \"8d37c0a6-e58b-4f0b-a043-6184b537a719\" (UID: \"8d37c0a6-e58b-4f0b-a043-6184b537a719\") " Nov 04 03:06:59.539369 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:06:59.539345 4405 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8d37c0a6-e58b-4f0b-a043-6184b537a719-workload-certs" pod "8d37c0a6-e58b-4f0b-a043-6184b537a719" (UID: "8d37c0a6-e58b-4f0b-a043-6184b537a719"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 03:06:59.540130 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:06:59.540080 4405 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8d37c0a6-e58b-4f0b-a043-6184b537a719-credential-socket" pod "8d37c0a6-e58b-4f0b-a043-6184b537a719" (UID: "8d37c0a6-e58b-4f0b-a043-6184b537a719"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 03:06:59.540267 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:06:59.540238 4405 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8d37c0a6-e58b-4f0b-a043-6184b537a719-credential-socket\") on node \"ip-10-0-134-125.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:06:59.540267 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:06:59.540259 4405 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8d37c0a6-e58b-4f0b-a043-6184b537a719-workload-socket\") on node \"ip-10-0-134-125.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:06:59.540384 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:06:59.540274 4405 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8d37c0a6-e58b-4f0b-a043-6184b537a719-workload-certs\") on node \"ip-10-0-134-125.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:06:59.540475 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:06:59.540451 4405 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8d37c0a6-e58b-4f0b-a043-6184b537a719-istiod-ca-cert" pod "8d37c0a6-e58b-4f0b-a043-6184b537a719" (UID: "8d37c0a6-e58b-4f0b-a043-6184b537a719"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 03:06:59.540548 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:06:59.540449 4405 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8d37c0a6-e58b-4f0b-a043-6184b537a719-istio-data" pod "8d37c0a6-e58b-4f0b-a043-6184b537a719" (UID: "8d37c0a6-e58b-4f0b-a043-6184b537a719"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 03:06:59.541623 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:06:59.541596 4405 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/8d37c0a6-e58b-4f0b-a043-6184b537a719-istio-podinfo" pod "8d37c0a6-e58b-4f0b-a043-6184b537a719" (UID: "8d37c0a6-e58b-4f0b-a043-6184b537a719"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 04 03:06:59.541900 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:06:59.541878 4405 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8d37c0a6-e58b-4f0b-a043-6184b537a719-istio-token" pod "8d37c0a6-e58b-4f0b-a043-6184b537a719" (UID: "8d37c0a6-e58b-4f0b-a043-6184b537a719"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 03:06:59.542183 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:06:59.542154 4405 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8d37c0a6-e58b-4f0b-a043-6184b537a719-kube-api-access-7vvw8" pod "8d37c0a6-e58b-4f0b-a043-6184b537a719" (UID: "8d37c0a6-e58b-4f0b-a043-6184b537a719"). InnerVolumeSpecName "kube-api-access-7vvw8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 03:06:59.542271 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:06:59.542186 4405 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8d37c0a6-e58b-4f0b-a043-6184b537a719-istio-envoy" pod "8d37c0a6-e58b-4f0b-a043-6184b537a719" (UID: "8d37c0a6-e58b-4f0b-a043-6184b537a719"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 03:06:59.641680 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:06:59.641633 4405 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8d37c0a6-e58b-4f0b-a043-6184b537a719-istio-token\") on node \"ip-10-0-134-125.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:06:59.641680 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:06:59.641681 4405 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8d37c0a6-e58b-4f0b-a043-6184b537a719-istio-podinfo\") on node \"ip-10-0-134-125.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:06:59.641886 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:06:59.641693 4405 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7vvw8\" (UniqueName: \"kubernetes.io/projected/8d37c0a6-e58b-4f0b-a043-6184b537a719-kube-api-access-7vvw8\") on node \"ip-10-0-134-125.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:06:59.641886 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:06:59.641704 4405 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8d37c0a6-e58b-4f0b-a043-6184b537a719-istio-data\") on node \"ip-10-0-134-125.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:06:59.641886 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:06:59.641713 4405 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8d37c0a6-e58b-4f0b-a043-6184b537a719-istio-envoy\") on node \"ip-10-0-134-125.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:06:59.641886 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:06:59.641721 4405 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8d37c0a6-e58b-4f0b-a043-6184b537a719-istiod-ca-cert\") on node \"ip-10-0-134-125.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:07:00.198815 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:07:00.198777 4405 generic.go:356] "Generic (PLEG): container finished" podID="8d37c0a6-e58b-4f0b-a043-6184b537a719" containerID="9d1aa8164135250664e9340506c6a3ecc0d38105e5bb63b6ebb9d6e8de2d0a91" exitCode=0 Nov 04 03:07:00.199034 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:07:00.198830 4405 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-fe4e0c-openshift-default-c7f4fc84c-mmngc" event={"ID":"8d37c0a6-e58b-4f0b-a043-6184b537a719","Type":"ContainerDied","Data":"9d1aa8164135250664e9340506c6a3ecc0d38105e5bb63b6ebb9d6e8de2d0a91"} Nov 04 03:07:00.199034 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:07:00.198865 4405 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-fe4e0c-openshift-default-c7f4fc84c-mmngc" Nov 04 03:07:00.199034 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:07:00.198908 4405 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-fe4e0c-openshift-default-c7f4fc84c-mmngc" event={"ID":"8d37c0a6-e58b-4f0b-a043-6184b537a719","Type":"ContainerDied","Data":"d6a30fdd7902fe148fa82f2dc6f8840e9196daa265d3c66605c491b6ccd87885"} Nov 04 03:07:00.199034 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:07:00.198936 4405 scope.go:122] "RemoveContainer" containerID="9d1aa8164135250664e9340506c6a3ecc0d38105e5bb63b6ebb9d6e8de2d0a91" Nov 04 03:07:00.210509 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:07:00.210475 4405 scope.go:122] "RemoveContainer" containerID="9d1aa8164135250664e9340506c6a3ecc0d38105e5bb63b6ebb9d6e8de2d0a91" Nov 04 03:07:00.210842 ip-10-0-134-125 kubenswrapper[4405]: E1104 03:07:00.210809 4405 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9d1aa8164135250664e9340506c6a3ecc0d38105e5bb63b6ebb9d6e8de2d0a91\": container with ID starting with 9d1aa8164135250664e9340506c6a3ecc0d38105e5bb63b6ebb9d6e8de2d0a91 not found: ID does not exist" containerID="9d1aa8164135250664e9340506c6a3ecc0d38105e5bb63b6ebb9d6e8de2d0a91" Nov 04 03:07:00.210929 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:07:00.210857 4405 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9d1aa8164135250664e9340506c6a3ecc0d38105e5bb63b6ebb9d6e8de2d0a91"} err="failed to get container status \"9d1aa8164135250664e9340506c6a3ecc0d38105e5bb63b6ebb9d6e8de2d0a91\": rpc error: code = NotFound desc = could not find container \"9d1aa8164135250664e9340506c6a3ecc0d38105e5bb63b6ebb9d6e8de2d0a91\": container with ID starting with 9d1aa8164135250664e9340506c6a3ecc0d38105e5bb63b6ebb9d6e8de2d0a91 not found: ID does not exist" Nov 04 03:07:00.417364 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:07:00.417062 4405 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-fe4e0c-openshift-default-c7f4fc84c-mmngc"] Nov 04 03:07:00.465964 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:07:00.465876 4405 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-fe4e0c-openshift-default-c7f4fc84c-mmngc"] Nov 04 03:07:01.175472 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:07:01.175442 4405 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-9d2b96-7d4bf9f94-959mx" Nov 04 03:07:01.704161 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:07:01.704126 4405 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8d37c0a6-e58b-4f0b-a043-6184b537a719" path="/var/lib/kubelet/pods/8d37c0a6-e58b-4f0b-a043-6184b537a719/volumes" Nov 04 03:07:01.738176 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:07:01.738140 4405 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-22e72e-openshift-default-64656d9fb9-26n5c"] Nov 04 03:07:01.739060 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:07:01.739035 4405 cpu_manager.go:414] "containerMap: removing container" podUID="b7f09efb-5b70-457a-93c0-9125cae4c2dc" containerName="payload-pre-processing" Nov 04 03:07:01.739060 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:07:01.739058 4405 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b7f09efb-5b70-457a-93c0-9125cae4c2dc" containerName="payload-pre-processing" Nov 04 03:07:01.739270 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:07:01.739083 4405 cpu_manager.go:414] "containerMap: removing container" podUID="7acf0cd1-800b-4796-9b10-a2041f971104" containerName="payload-processing" Nov 04 03:07:01.739270 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:07:01.739091 4405 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7acf0cd1-800b-4796-9b10-a2041f971104" containerName="payload-processing" Nov 04 03:07:01.739270 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:07:01.739116 4405 cpu_manager.go:414] "containerMap: removing container" podUID="8d37c0a6-e58b-4f0b-a043-6184b537a719" containerName="istio-proxy" Nov 04 03:07:01.739270 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:07:01.739123 4405 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8d37c0a6-e58b-4f0b-a043-6184b537a719" containerName="istio-proxy" Nov 04 03:07:01.739467 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:07:01.739281 4405 memory_manager.go:365] "RemoveStaleState removing state" podUID="8d37c0a6-e58b-4f0b-a043-6184b537a719" containerName="istio-proxy" Nov 04 03:07:01.739467 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:07:01.739310 4405 memory_manager.go:365] "RemoveStaleState removing state" podUID="b7f09efb-5b70-457a-93c0-9125cae4c2dc" containerName="payload-pre-processing" Nov 04 03:07:01.739467 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:07:01.739322 4405 memory_manager.go:365] "RemoveStaleState removing state" podUID="7acf0cd1-800b-4796-9b10-a2041f971104" containerName="payload-processing" Nov 04 03:07:01.744466 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:07:01.744443 4405 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-22e72e-openshift-default-64656d9fb9-26n5c" Nov 04 03:07:01.909612 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:07:01.909580 4405 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-22e72e-openshift-default-dockercfg-fzmfp\"" Nov 04 03:07:01.991760 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:07:01.987033 4405 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fd747189-5360-4360-83cd-07a4e414fda4-istiod-ca-cert\") pod \"e2e-inf-a-22e72e-openshift-default-64656d9fb9-26n5c\" (UID: \"fd747189-5360-4360-83cd-07a4e414fda4\") " pod="openshift-ingress/e2e-inf-a-22e72e-openshift-default-64656d9fb9-26n5c" Nov 04 03:07:01.991760 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:07:01.987157 4405 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fd747189-5360-4360-83cd-07a4e414fda4-istio-token\") pod \"e2e-inf-a-22e72e-openshift-default-64656d9fb9-26n5c\" (UID: \"fd747189-5360-4360-83cd-07a4e414fda4\") " pod="openshift-ingress/e2e-inf-a-22e72e-openshift-default-64656d9fb9-26n5c" Nov 04 03:07:01.991760 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:07:01.987218 4405 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fd747189-5360-4360-83cd-07a4e414fda4-credential-socket\") pod \"e2e-inf-a-22e72e-openshift-default-64656d9fb9-26n5c\" (UID: \"fd747189-5360-4360-83cd-07a4e414fda4\") " pod="openshift-ingress/e2e-inf-a-22e72e-openshift-default-64656d9fb9-26n5c" Nov 04 03:07:01.991760 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:07:01.987329 4405 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fd747189-5360-4360-83cd-07a4e414fda4-workload-socket\") pod \"e2e-inf-a-22e72e-openshift-default-64656d9fb9-26n5c\" (UID: \"fd747189-5360-4360-83cd-07a4e414fda4\") " pod="openshift-ingress/e2e-inf-a-22e72e-openshift-default-64656d9fb9-26n5c" Nov 04 03:07:01.991760 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:07:01.987456 4405 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dwk9j\" (UniqueName: \"kubernetes.io/projected/fd747189-5360-4360-83cd-07a4e414fda4-kube-api-access-dwk9j\") pod \"e2e-inf-a-22e72e-openshift-default-64656d9fb9-26n5c\" (UID: \"fd747189-5360-4360-83cd-07a4e414fda4\") " pod="openshift-ingress/e2e-inf-a-22e72e-openshift-default-64656d9fb9-26n5c" Nov 04 03:07:01.991760 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:07:01.987505 4405 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fd747189-5360-4360-83cd-07a4e414fda4-istio-data\") pod \"e2e-inf-a-22e72e-openshift-default-64656d9fb9-26n5c\" (UID: \"fd747189-5360-4360-83cd-07a4e414fda4\") " pod="openshift-ingress/e2e-inf-a-22e72e-openshift-default-64656d9fb9-26n5c" Nov 04 03:07:01.991760 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:07:01.987542 4405 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fd747189-5360-4360-83cd-07a4e414fda4-workload-certs\") pod \"e2e-inf-a-22e72e-openshift-default-64656d9fb9-26n5c\" (UID: \"fd747189-5360-4360-83cd-07a4e414fda4\") " pod="openshift-ingress/e2e-inf-a-22e72e-openshift-default-64656d9fb9-26n5c" Nov 04 03:07:01.991760 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:07:01.987599 4405 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fd747189-5360-4360-83cd-07a4e414fda4-istio-envoy\") pod \"e2e-inf-a-22e72e-openshift-default-64656d9fb9-26n5c\" (UID: \"fd747189-5360-4360-83cd-07a4e414fda4\") " pod="openshift-ingress/e2e-inf-a-22e72e-openshift-default-64656d9fb9-26n5c" Nov 04 03:07:01.991760 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:07:01.987627 4405 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fd747189-5360-4360-83cd-07a4e414fda4-istio-podinfo\") pod \"e2e-inf-a-22e72e-openshift-default-64656d9fb9-26n5c\" (UID: \"fd747189-5360-4360-83cd-07a4e414fda4\") " pod="openshift-ingress/e2e-inf-a-22e72e-openshift-default-64656d9fb9-26n5c" Nov 04 03:07:01.991760 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:07:01.988293 4405 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-22e72e-openshift-default-64656d9fb9-26n5c"] Nov 04 03:07:02.089464 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:07:02.089426 4405 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fd747189-5360-4360-83cd-07a4e414fda4-istiod-ca-cert\") pod \"e2e-inf-a-22e72e-openshift-default-64656d9fb9-26n5c\" (UID: \"fd747189-5360-4360-83cd-07a4e414fda4\") " pod="openshift-ingress/e2e-inf-a-22e72e-openshift-default-64656d9fb9-26n5c" Nov 04 03:07:02.089696 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:07:02.089510 4405 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fd747189-5360-4360-83cd-07a4e414fda4-istio-token\") pod \"e2e-inf-a-22e72e-openshift-default-64656d9fb9-26n5c\" (UID: \"fd747189-5360-4360-83cd-07a4e414fda4\") " pod="openshift-ingress/e2e-inf-a-22e72e-openshift-default-64656d9fb9-26n5c" Nov 04 03:07:02.089696 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:07:02.089549 4405 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fd747189-5360-4360-83cd-07a4e414fda4-credential-socket\") pod \"e2e-inf-a-22e72e-openshift-default-64656d9fb9-26n5c\" (UID: \"fd747189-5360-4360-83cd-07a4e414fda4\") " pod="openshift-ingress/e2e-inf-a-22e72e-openshift-default-64656d9fb9-26n5c" Nov 04 03:07:02.089696 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:07:02.089628 4405 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fd747189-5360-4360-83cd-07a4e414fda4-workload-socket\") pod \"e2e-inf-a-22e72e-openshift-default-64656d9fb9-26n5c\" (UID: \"fd747189-5360-4360-83cd-07a4e414fda4\") " pod="openshift-ingress/e2e-inf-a-22e72e-openshift-default-64656d9fb9-26n5c" Nov 04 03:07:02.089877 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:07:02.089763 4405 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dwk9j\" (UniqueName: \"kubernetes.io/projected/fd747189-5360-4360-83cd-07a4e414fda4-kube-api-access-dwk9j\") pod \"e2e-inf-a-22e72e-openshift-default-64656d9fb9-26n5c\" (UID: \"fd747189-5360-4360-83cd-07a4e414fda4\") " pod="openshift-ingress/e2e-inf-a-22e72e-openshift-default-64656d9fb9-26n5c" Nov 04 03:07:02.089877 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:07:02.089807 4405 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fd747189-5360-4360-83cd-07a4e414fda4-istio-data\") pod \"e2e-inf-a-22e72e-openshift-default-64656d9fb9-26n5c\" (UID: \"fd747189-5360-4360-83cd-07a4e414fda4\") " pod="openshift-ingress/e2e-inf-a-22e72e-openshift-default-64656d9fb9-26n5c" Nov 04 03:07:02.089877 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:07:02.089847 4405 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fd747189-5360-4360-83cd-07a4e414fda4-workload-certs\") pod \"e2e-inf-a-22e72e-openshift-default-64656d9fb9-26n5c\" (UID: \"fd747189-5360-4360-83cd-07a4e414fda4\") " pod="openshift-ingress/e2e-inf-a-22e72e-openshift-default-64656d9fb9-26n5c" Nov 04 03:07:02.090041 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:07:02.089902 4405 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fd747189-5360-4360-83cd-07a4e414fda4-istio-envoy\") pod \"e2e-inf-a-22e72e-openshift-default-64656d9fb9-26n5c\" (UID: \"fd747189-5360-4360-83cd-07a4e414fda4\") " pod="openshift-ingress/e2e-inf-a-22e72e-openshift-default-64656d9fb9-26n5c" Nov 04 03:07:02.090041 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:07:02.089931 4405 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fd747189-5360-4360-83cd-07a4e414fda4-istio-podinfo\") pod \"e2e-inf-a-22e72e-openshift-default-64656d9fb9-26n5c\" (UID: \"fd747189-5360-4360-83cd-07a4e414fda4\") " pod="openshift-ingress/e2e-inf-a-22e72e-openshift-default-64656d9fb9-26n5c" Nov 04 03:07:02.090041 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:07:02.090024 4405 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fd747189-5360-4360-83cd-07a4e414fda4-credential-socket\") pod \"e2e-inf-a-22e72e-openshift-default-64656d9fb9-26n5c\" (UID: \"fd747189-5360-4360-83cd-07a4e414fda4\") " pod="openshift-ingress/e2e-inf-a-22e72e-openshift-default-64656d9fb9-26n5c" Nov 04 03:07:02.090220 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:07:02.090028 4405 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fd747189-5360-4360-83cd-07a4e414fda4-workload-socket\") pod \"e2e-inf-a-22e72e-openshift-default-64656d9fb9-26n5c\" (UID: \"fd747189-5360-4360-83cd-07a4e414fda4\") " pod="openshift-ingress/e2e-inf-a-22e72e-openshift-default-64656d9fb9-26n5c" Nov 04 03:07:02.090277 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:07:02.090243 4405 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fd747189-5360-4360-83cd-07a4e414fda4-istio-data\") pod \"e2e-inf-a-22e72e-openshift-default-64656d9fb9-26n5c\" (UID: \"fd747189-5360-4360-83cd-07a4e414fda4\") " pod="openshift-ingress/e2e-inf-a-22e72e-openshift-default-64656d9fb9-26n5c" Nov 04 03:07:02.090393 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:07:02.090371 4405 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fd747189-5360-4360-83cd-07a4e414fda4-workload-certs\") pod \"e2e-inf-a-22e72e-openshift-default-64656d9fb9-26n5c\" (UID: \"fd747189-5360-4360-83cd-07a4e414fda4\") " pod="openshift-ingress/e2e-inf-a-22e72e-openshift-default-64656d9fb9-26n5c" Nov 04 03:07:02.091162 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:07:02.091115 4405 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fd747189-5360-4360-83cd-07a4e414fda4-istiod-ca-cert\") pod \"e2e-inf-a-22e72e-openshift-default-64656d9fb9-26n5c\" (UID: \"fd747189-5360-4360-83cd-07a4e414fda4\") " pod="openshift-ingress/e2e-inf-a-22e72e-openshift-default-64656d9fb9-26n5c" Nov 04 03:07:02.094201 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:07:02.093658 4405 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fd747189-5360-4360-83cd-07a4e414fda4-istio-envoy\") pod \"e2e-inf-a-22e72e-openshift-default-64656d9fb9-26n5c\" (UID: \"fd747189-5360-4360-83cd-07a4e414fda4\") " pod="openshift-ingress/e2e-inf-a-22e72e-openshift-default-64656d9fb9-26n5c" Nov 04 03:07:02.094201 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:07:02.094124 4405 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fd747189-5360-4360-83cd-07a4e414fda4-istio-podinfo\") pod \"e2e-inf-a-22e72e-openshift-default-64656d9fb9-26n5c\" (UID: \"fd747189-5360-4360-83cd-07a4e414fda4\") " pod="openshift-ingress/e2e-inf-a-22e72e-openshift-default-64656d9fb9-26n5c" Nov 04 03:07:02.151339 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:07:02.151300 4405 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fd747189-5360-4360-83cd-07a4e414fda4-istio-token\") pod \"e2e-inf-a-22e72e-openshift-default-64656d9fb9-26n5c\" (UID: \"fd747189-5360-4360-83cd-07a4e414fda4\") " pod="openshift-ingress/e2e-inf-a-22e72e-openshift-default-64656d9fb9-26n5c" Nov 04 03:07:02.155333 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:07:02.155302 4405 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dwk9j\" (UniqueName: \"kubernetes.io/projected/fd747189-5360-4360-83cd-07a4e414fda4-kube-api-access-dwk9j\") pod \"e2e-inf-a-22e72e-openshift-default-64656d9fb9-26n5c\" (UID: \"fd747189-5360-4360-83cd-07a4e414fda4\") " pod="openshift-ingress/e2e-inf-a-22e72e-openshift-default-64656d9fb9-26n5c" Nov 04 03:07:02.358546 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:07:02.358508 4405 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-22e72e-openshift-default-64656d9fb9-26n5c" Nov 04 03:07:02.799019 ip-10-0-134-125 kubenswrapper[4405]: W1104 03:07:02.798991 4405 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfd747189_5360_4360_83cd_07a4e414fda4.slice/crio-22f652c5ab1a3f26e4618c6db785aa4bee2684e7a3403afd44cb2947bf7a97ce WatchSource:0}: Error finding container 22f652c5ab1a3f26e4618c6db785aa4bee2684e7a3403afd44cb2947bf7a97ce: Status 404 returned error can't find the container with id 22f652c5ab1a3f26e4618c6db785aa4bee2684e7a3403afd44cb2947bf7a97ce Nov 04 03:07:02.799137 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:07:02.799027 4405 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-22e72e-openshift-default-64656d9fb9-26n5c"] Nov 04 03:07:02.803533 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:07:02.803493 4405 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 04 03:07:02.803609 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:07:02.803583 4405 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 04 03:07:02.803654 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:07:02.803627 4405 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 04 03:07:03.229874 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:07:03.229831 4405 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-22e72e-openshift-default-64656d9fb9-26n5c" event={"ID":"fd747189-5360-4360-83cd-07a4e414fda4","Type":"ContainerStarted","Data":"27b80f527345011ee23f01f21487a67b9800509257b6b45c3ec35831fdfe7633"} Nov 04 03:07:03.230073 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:07:03.229904 4405 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-22e72e-openshift-default-64656d9fb9-26n5c" event={"ID":"fd747189-5360-4360-83cd-07a4e414fda4","Type":"ContainerStarted","Data":"22f652c5ab1a3f26e4618c6db785aa4bee2684e7a3403afd44cb2947bf7a97ce"} Nov 04 03:07:03.352339 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:07:03.352283 4405 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-22e72e-openshift-default-64656d9fb9-26n5c" podStartSLOduration=2.352266434 podStartE2EDuration="2.352266434s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 03:07:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 03:07:03.349309148 +0000 UTC m=+2069.780009271" watchObservedRunningTime="2026-11-04 03:07:03.352266434 +0000 UTC m=+2069.782966553" Nov 04 03:07:03.359471 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:07:03.359445 4405 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-22e72e-openshift-default-64656d9fb9-26n5c" Nov 04 03:07:03.365337 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:07:03.365313 4405 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-22e72e-openshift-default-64656d9fb9-26n5c" Nov 04 03:07:04.239454 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:07:04.239420 4405 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-22e72e-openshift-default-64656d9fb9-26n5c" Nov 04 03:07:04.240599 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:07:04.240576 4405 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-22e72e-openshift-default-64656d9fb9-26n5c" Nov 04 03:07:06.886529 ip-10-0-134-125 kubenswrapper[4405]: E1104 03:07:06.886485 4405 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5f2vb" containerName="extractor" Nov 04 03:07:09.438624 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:07:09.438585 4405 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-57f116-5b5dd4bd44-fzgp7"] Nov 04 03:07:09.463089 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:07:09.463048 4405 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-bl5mg"] Nov 04 03:07:09.552971 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:07:09.552936 4405 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-57f116-599cbf5bbc-2fp4d"] Nov 04 03:07:13.769006 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:07:13.768970 4405 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-22e72e-7ffdf7c747-vbt9t"] Nov 04 03:07:13.774513 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:07:13.774485 4405 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-22e72e-7ffdf7c747-vbt9t" Nov 04 03:07:13.838994 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:07:13.838948 4405 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-22e72e-7ffdf7c747-vbt9t"] Nov 04 03:07:13.880986 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:07:13.880951 4405 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-a-22e72e-dockercfg-84m8n\"" Nov 04 03:07:13.906772 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:07:13.897920 4405 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-a-22e72e\"" Nov 04 03:07:13.984755 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:07:13.969211 4405 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3dcb7be0-d703-40eb-8bc4-9228972e6376-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-22e72e-7ffdf7c747-vbt9t\" (UID: \"3dcb7be0-d703-40eb-8bc4-9228972e6376\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-22e72e-7ffdf7c747-vbt9t" Nov 04 03:07:13.996019 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:07:13.995963 4405 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jjnr9\" (UniqueName: \"kubernetes.io/projected/3dcb7be0-d703-40eb-8bc4-9228972e6376-kube-api-access-jjnr9\") pod \"payload-pre-processing-e2e-inf-a-22e72e-7ffdf7c747-vbt9t\" (UID: \"3dcb7be0-d703-40eb-8bc4-9228972e6376\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-22e72e-7ffdf7c747-vbt9t" Nov 04 03:07:14.008322 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:07:14.008291 4405 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-22e72e-85b6d5f746-nw6tv"] Nov 04 03:07:14.014606 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:07:14.014577 4405 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-22e72e-85b6d5f746-nw6tv" Nov 04 03:07:14.018997 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:07:14.018948 4405 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-22e72e-85b6d5f746-nw6tv"] Nov 04 03:07:14.098068 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:07:14.098035 4405 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jjnr9\" (UniqueName: \"kubernetes.io/projected/3dcb7be0-d703-40eb-8bc4-9228972e6376-kube-api-access-jjnr9\") pod \"payload-pre-processing-e2e-inf-a-22e72e-7ffdf7c747-vbt9t\" (UID: \"3dcb7be0-d703-40eb-8bc4-9228972e6376\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-22e72e-7ffdf7c747-vbt9t" Nov 04 03:07:14.098249 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:07:14.098111 4405 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3dcb7be0-d703-40eb-8bc4-9228972e6376-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-22e72e-7ffdf7c747-vbt9t\" (UID: \"3dcb7be0-d703-40eb-8bc4-9228972e6376\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-22e72e-7ffdf7c747-vbt9t" Nov 04 03:07:14.098249 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:07:14.098149 4405 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e6996ff6-b08c-4c58-916d-b79ae32ade57-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-22e72e-85b6d5f746-nw6tv\" (UID: \"e6996ff6-b08c-4c58-916d-b79ae32ade57\") " pod="openshift-ingress/payload-processing-e2e-inf-a-22e72e-85b6d5f746-nw6tv" Nov 04 03:07:14.098375 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:07:14.098338 4405 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h4f65\" (UniqueName: \"kubernetes.io/projected/e6996ff6-b08c-4c58-916d-b79ae32ade57-kube-api-access-h4f65\") pod \"payload-processing-e2e-inf-a-22e72e-85b6d5f746-nw6tv\" (UID: \"e6996ff6-b08c-4c58-916d-b79ae32ade57\") " pod="openshift-ingress/payload-processing-e2e-inf-a-22e72e-85b6d5f746-nw6tv" Nov 04 03:07:14.098837 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:07:14.098817 4405 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3dcb7be0-d703-40eb-8bc4-9228972e6376-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-22e72e-7ffdf7c747-vbt9t\" (UID: \"3dcb7be0-d703-40eb-8bc4-9228972e6376\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-22e72e-7ffdf7c747-vbt9t" Nov 04 03:07:14.158947 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:07:14.158909 4405 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jjnr9\" (UniqueName: \"kubernetes.io/projected/3dcb7be0-d703-40eb-8bc4-9228972e6376-kube-api-access-jjnr9\") pod \"payload-pre-processing-e2e-inf-a-22e72e-7ffdf7c747-vbt9t\" (UID: \"3dcb7be0-d703-40eb-8bc4-9228972e6376\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-22e72e-7ffdf7c747-vbt9t" Nov 04 03:07:14.199883 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:07:14.199847 4405 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h4f65\" (UniqueName: \"kubernetes.io/projected/e6996ff6-b08c-4c58-916d-b79ae32ade57-kube-api-access-h4f65\") pod \"payload-processing-e2e-inf-a-22e72e-85b6d5f746-nw6tv\" (UID: \"e6996ff6-b08c-4c58-916d-b79ae32ade57\") " pod="openshift-ingress/payload-processing-e2e-inf-a-22e72e-85b6d5f746-nw6tv" Nov 04 03:07:14.200100 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:07:14.200031 4405 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e6996ff6-b08c-4c58-916d-b79ae32ade57-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-22e72e-85b6d5f746-nw6tv\" (UID: \"e6996ff6-b08c-4c58-916d-b79ae32ade57\") " pod="openshift-ingress/payload-processing-e2e-inf-a-22e72e-85b6d5f746-nw6tv" Nov 04 03:07:14.200803 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:07:14.200781 4405 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e6996ff6-b08c-4c58-916d-b79ae32ade57-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-22e72e-85b6d5f746-nw6tv\" (UID: \"e6996ff6-b08c-4c58-916d-b79ae32ade57\") " pod="openshift-ingress/payload-processing-e2e-inf-a-22e72e-85b6d5f746-nw6tv" Nov 04 03:07:14.266903 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:07:14.266873 4405 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h4f65\" (UniqueName: \"kubernetes.io/projected/e6996ff6-b08c-4c58-916d-b79ae32ade57-kube-api-access-h4f65\") pod \"payload-processing-e2e-inf-a-22e72e-85b6d5f746-nw6tv\" (UID: \"e6996ff6-b08c-4c58-916d-b79ae32ade57\") " pod="openshift-ingress/payload-processing-e2e-inf-a-22e72e-85b6d5f746-nw6tv" Nov 04 03:07:14.337135 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:07:14.337104 4405 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-22e72e-85b6d5f746-nw6tv" Nov 04 03:07:14.393633 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:07:14.393597 4405 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-22e72e-7ffdf7c747-vbt9t" Nov 04 03:07:14.439686 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:07:14.439612 4405 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-57f116-5b5dd4bd44-fzgp7" podUID="6ff412ca-a023-4cde-8c75-6f9acbfb5121" containerName="payload-processing" containerID="cri-o://592f36e31a6613b7397f2e567a3e9c6b74f02023978134ed29d82f766ff4120f" gracePeriod=25 Nov 04 03:07:14.553810 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:07:14.553761 4405 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-57f116-599cbf5bbc-2fp4d" podUID="449e5858-ec18-4e0e-ab17-520312115a05" containerName="payload-pre-processing" containerID="cri-o://9889ba24a9f3fba0e0546f8ead70aa4c6f50e01d14799a687618e48419675746" gracePeriod=25 Nov 04 03:07:14.730295 ip-10-0-134-125 kubenswrapper[4405]: W1104 03:07:14.730258 4405 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode6996ff6_b08c_4c58_916d_b79ae32ade57.slice/crio-20cc72c3e791df4ce671a0dbcd24e6874a54634394fa0778496eacffcde0f531 WatchSource:0}: Error finding container 20cc72c3e791df4ce671a0dbcd24e6874a54634394fa0778496eacffcde0f531: Status 404 returned error can't find the container with id 20cc72c3e791df4ce671a0dbcd24e6874a54634394fa0778496eacffcde0f531 Nov 04 03:07:14.733926 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:07:14.733898 4405 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-22e72e-85b6d5f746-nw6tv"] Nov 04 03:07:14.742105 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:07:14.742073 4405 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-57f116-5b5dd4bd44-fzgp7" Nov 04 03:07:14.846625 ip-10-0-134-125 kubenswrapper[4405]: W1104 03:07:14.846553 4405 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3dcb7be0_d703_40eb_8bc4_9228972e6376.slice/crio-1eb8986d7347b5293c0d5c33ea356ba6f5370ce3399ccc750b5317d035cb51cc WatchSource:0}: Error finding container 1eb8986d7347b5293c0d5c33ea356ba6f5370ce3399ccc750b5317d035cb51cc: Status 404 returned error can't find the container with id 1eb8986d7347b5293c0d5c33ea356ba6f5370ce3399ccc750b5317d035cb51cc Nov 04 03:07:14.846625 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:07:14.846571 4405 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-22e72e-7ffdf7c747-vbt9t"] Nov 04 03:07:14.865042 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:07:14.865016 4405 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-57f116-599cbf5bbc-2fp4d" Nov 04 03:07:14.912483 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:07:14.912451 4405 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6ff412ca-a023-4cde-8c75-6f9acbfb5121-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6ff412ca-a023-4cde-8c75-6f9acbfb5121-plugins-config-volume\") pod \"6ff412ca-a023-4cde-8c75-6f9acbfb5121\" (UID: \"6ff412ca-a023-4cde-8c75-6f9acbfb5121\") " Nov 04 03:07:14.912631 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:07:14.912556 4405 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6ff412ca-a023-4cde-8c75-6f9acbfb5121-kube-api-access-ft8vl\" (UniqueName: \"kubernetes.io/projected/6ff412ca-a023-4cde-8c75-6f9acbfb5121-kube-api-access-ft8vl\") pod \"6ff412ca-a023-4cde-8c75-6f9acbfb5121\" (UID: \"6ff412ca-a023-4cde-8c75-6f9acbfb5121\") " Nov 04 03:07:14.913171 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:07:14.913119 4405 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6ff412ca-a023-4cde-8c75-6f9acbfb5121-plugins-config-volume" pod "6ff412ca-a023-4cde-8c75-6f9acbfb5121" (UID: "6ff412ca-a023-4cde-8c75-6f9acbfb5121"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 03:07:14.913317 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:07:14.913296 4405 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6ff412ca-a023-4cde-8c75-6f9acbfb5121-plugins-config-volume\") on node \"ip-10-0-134-125.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:07:14.915044 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:07:14.915022 4405 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6ff412ca-a023-4cde-8c75-6f9acbfb5121-kube-api-access-ft8vl" pod "6ff412ca-a023-4cde-8c75-6f9acbfb5121" (UID: "6ff412ca-a023-4cde-8c75-6f9acbfb5121"). InnerVolumeSpecName "kube-api-access-ft8vl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 03:07:15.013941 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:07:15.013901 4405 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/449e5858-ec18-4e0e-ab17-520312115a05-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/449e5858-ec18-4e0e-ab17-520312115a05-plugins-config-volume\") pod \"449e5858-ec18-4e0e-ab17-520312115a05\" (UID: \"449e5858-ec18-4e0e-ab17-520312115a05\") " Nov 04 03:07:15.014310 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:07:15.014287 4405 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/449e5858-ec18-4e0e-ab17-520312115a05-kube-api-access-2n4l6\" (UniqueName: \"kubernetes.io/projected/449e5858-ec18-4e0e-ab17-520312115a05-kube-api-access-2n4l6\") pod \"449e5858-ec18-4e0e-ab17-520312115a05\" (UID: \"449e5858-ec18-4e0e-ab17-520312115a05\") " Nov 04 03:07:15.014395 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:07:15.014307 4405 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/449e5858-ec18-4e0e-ab17-520312115a05-plugins-config-volume" pod "449e5858-ec18-4e0e-ab17-520312115a05" (UID: "449e5858-ec18-4e0e-ab17-520312115a05"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 03:07:15.014768 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:07:15.014749 4405 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ft8vl\" (UniqueName: \"kubernetes.io/projected/6ff412ca-a023-4cde-8c75-6f9acbfb5121-kube-api-access-ft8vl\") on node \"ip-10-0-134-125.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:07:15.014848 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:07:15.014773 4405 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/449e5858-ec18-4e0e-ab17-520312115a05-plugins-config-volume\") on node \"ip-10-0-134-125.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:07:15.016642 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:07:15.016616 4405 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/449e5858-ec18-4e0e-ab17-520312115a05-kube-api-access-2n4l6" pod "449e5858-ec18-4e0e-ab17-520312115a05" (UID: "449e5858-ec18-4e0e-ab17-520312115a05"). InnerVolumeSpecName "kube-api-access-2n4l6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 03:07:15.116146 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:07:15.116110 4405 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2n4l6\" (UniqueName: \"kubernetes.io/projected/449e5858-ec18-4e0e-ab17-520312115a05-kube-api-access-2n4l6\") on node \"ip-10-0-134-125.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:07:15.335476 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:07:15.335441 4405 generic.go:356] "Generic (PLEG): container finished" podID="449e5858-ec18-4e0e-ab17-520312115a05" containerID="9889ba24a9f3fba0e0546f8ead70aa4c6f50e01d14799a687618e48419675746" exitCode=0 Nov 04 03:07:15.335727 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:07:15.335537 4405 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-57f116-599cbf5bbc-2fp4d" event={"ID":"449e5858-ec18-4e0e-ab17-520312115a05","Type":"ContainerDied","Data":"9889ba24a9f3fba0e0546f8ead70aa4c6f50e01d14799a687618e48419675746"} Nov 04 03:07:15.335727 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:07:15.335539 4405 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-57f116-599cbf5bbc-2fp4d" Nov 04 03:07:15.335727 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:07:15.335596 4405 scope.go:122] "RemoveContainer" containerID="9889ba24a9f3fba0e0546f8ead70aa4c6f50e01d14799a687618e48419675746" Nov 04 03:07:15.335727 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:07:15.335584 4405 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-57f116-599cbf5bbc-2fp4d" event={"ID":"449e5858-ec18-4e0e-ab17-520312115a05","Type":"ContainerDied","Data":"4ddf520445cffebb967548194bb4b68d6abd53c0d70c61c6178cf4b21c2b0eec"} Nov 04 03:07:15.337794 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:07:15.337764 4405 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-22e72e-7ffdf7c747-vbt9t" event={"ID":"3dcb7be0-d703-40eb-8bc4-9228972e6376","Type":"ContainerStarted","Data":"f0bee4ce4f85cd635f14d3de269c65283086c9c46e53e9d981264dbb778f4b4d"} Nov 04 03:07:15.337949 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:07:15.337825 4405 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-22e72e-7ffdf7c747-vbt9t" Nov 04 03:07:15.337949 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:07:15.337854 4405 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-22e72e-7ffdf7c747-vbt9t" event={"ID":"3dcb7be0-d703-40eb-8bc4-9228972e6376","Type":"ContainerStarted","Data":"1eb8986d7347b5293c0d5c33ea356ba6f5370ce3399ccc750b5317d035cb51cc"} Nov 04 03:07:15.339466 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:07:15.339439 4405 generic.go:356] "Generic (PLEG): container finished" podID="6ff412ca-a023-4cde-8c75-6f9acbfb5121" containerID="592f36e31a6613b7397f2e567a3e9c6b74f02023978134ed29d82f766ff4120f" exitCode=0 Nov 04 03:07:15.339588 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:07:15.339477 4405 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-57f116-5b5dd4bd44-fzgp7" Nov 04 03:07:15.339588 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:07:15.339524 4405 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-57f116-5b5dd4bd44-fzgp7" event={"ID":"6ff412ca-a023-4cde-8c75-6f9acbfb5121","Type":"ContainerDied","Data":"592f36e31a6613b7397f2e567a3e9c6b74f02023978134ed29d82f766ff4120f"} Nov 04 03:07:15.339588 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:07:15.339570 4405 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-57f116-5b5dd4bd44-fzgp7" event={"ID":"6ff412ca-a023-4cde-8c75-6f9acbfb5121","Type":"ContainerDied","Data":"fc88e64e72fa679d6d555a244472de5e267213bc0c556004e9d5a4dd1711b3db"} Nov 04 03:07:15.341623 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:07:15.341603 4405 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-22e72e-85b6d5f746-nw6tv" event={"ID":"e6996ff6-b08c-4c58-916d-b79ae32ade57","Type":"ContainerStarted","Data":"0756f8b9248fc773c1c0c7399ff1099b004d30c313df24cc1c5bb041feb84ebb"} Nov 04 03:07:15.341755 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:07:15.341652 4405 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-a-22e72e-85b6d5f746-nw6tv" Nov 04 03:07:15.341755 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:07:15.341694 4405 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-22e72e-85b6d5f746-nw6tv" event={"ID":"e6996ff6-b08c-4c58-916d-b79ae32ade57","Type":"ContainerStarted","Data":"20cc72c3e791df4ce671a0dbcd24e6874a54634394fa0778496eacffcde0f531"} Nov 04 03:07:15.349468 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:07:15.349293 4405 scope.go:122] "RemoveContainer" containerID="9889ba24a9f3fba0e0546f8ead70aa4c6f50e01d14799a687618e48419675746" Nov 04 03:07:15.349603 ip-10-0-134-125 kubenswrapper[4405]: E1104 03:07:15.349585 4405 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9889ba24a9f3fba0e0546f8ead70aa4c6f50e01d14799a687618e48419675746\": container with ID starting with 9889ba24a9f3fba0e0546f8ead70aa4c6f50e01d14799a687618e48419675746 not found: ID does not exist" containerID="9889ba24a9f3fba0e0546f8ead70aa4c6f50e01d14799a687618e48419675746" Nov 04 03:07:15.349652 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:07:15.349611 4405 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9889ba24a9f3fba0e0546f8ead70aa4c6f50e01d14799a687618e48419675746"} err="failed to get container status \"9889ba24a9f3fba0e0546f8ead70aa4c6f50e01d14799a687618e48419675746\": rpc error: code = NotFound desc = could not find container \"9889ba24a9f3fba0e0546f8ead70aa4c6f50e01d14799a687618e48419675746\": container with ID starting with 9889ba24a9f3fba0e0546f8ead70aa4c6f50e01d14799a687618e48419675746 not found: ID does not exist" Nov 04 03:07:15.349652 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:07:15.349626 4405 scope.go:122] "RemoveContainer" containerID="592f36e31a6613b7397f2e567a3e9c6b74f02023978134ed29d82f766ff4120f" Nov 04 03:07:15.359382 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:07:15.359358 4405 scope.go:122] "RemoveContainer" containerID="592f36e31a6613b7397f2e567a3e9c6b74f02023978134ed29d82f766ff4120f" Nov 04 03:07:15.359692 ip-10-0-134-125 kubenswrapper[4405]: E1104 03:07:15.359656 4405 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"592f36e31a6613b7397f2e567a3e9c6b74f02023978134ed29d82f766ff4120f\": container with ID starting with 592f36e31a6613b7397f2e567a3e9c6b74f02023978134ed29d82f766ff4120f not found: ID does not exist" containerID="592f36e31a6613b7397f2e567a3e9c6b74f02023978134ed29d82f766ff4120f" Nov 04 03:07:15.359752 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:07:15.359706 4405 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"592f36e31a6613b7397f2e567a3e9c6b74f02023978134ed29d82f766ff4120f"} err="failed to get container status \"592f36e31a6613b7397f2e567a3e9c6b74f02023978134ed29d82f766ff4120f\": rpc error: code = NotFound desc = could not find container \"592f36e31a6613b7397f2e567a3e9c6b74f02023978134ed29d82f766ff4120f\": container with ID starting with 592f36e31a6613b7397f2e567a3e9c6b74f02023978134ed29d82f766ff4120f not found: ID does not exist" Nov 04 03:07:15.508006 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:07:15.507898 4405 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-a-22e72e-85b6d5f746-nw6tv" podStartSLOduration=2.507883929 podStartE2EDuration="2.507883929s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 03:07:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 03:07:15.504345267 +0000 UTC m=+2081.935045388" watchObservedRunningTime="2026-11-04 03:07:15.507883929 +0000 UTC m=+2081.938584049" Nov 04 03:07:15.618237 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:07:15.618201 4405 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-57f116-599cbf5bbc-2fp4d"] Nov 04 03:07:15.632327 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:07:15.628192 4405 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-57f116-599cbf5bbc-2fp4d"] Nov 04 03:07:15.707075 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:07:15.707044 4405 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="449e5858-ec18-4e0e-ab17-520312115a05" path="/var/lib/kubelet/pods/449e5858-ec18-4e0e-ab17-520312115a05/volumes" Nov 04 03:07:15.737026 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:07:15.736972 4405 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-22e72e-7ffdf7c747-vbt9t" podStartSLOduration=2.736958214 podStartE2EDuration="2.736958214s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 03:07:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 03:07:15.733618499 +0000 UTC m=+2082.164318622" watchObservedRunningTime="2026-11-04 03:07:15.736958214 +0000 UTC m=+2082.167658334" Nov 04 03:07:15.852889 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:07:15.852812 4405 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-57f116-5b5dd4bd44-fzgp7"] Nov 04 03:07:15.857588 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:07:15.857533 4405 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-57f116-5b5dd4bd44-fzgp7"] Nov 04 03:07:17.705555 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:07:17.705518 4405 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6ff412ca-a023-4cde-8c75-6f9acbfb5121" path="/var/lib/kubelet/pods/6ff412ca-a023-4cde-8c75-6f9acbfb5121/volumes" Nov 04 03:07:20.623922 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:07:20.623881 4405 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w1-551e2b-79ddfccbf8-6rkbp"] Nov 04 03:07:25.625072 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:07:25.625012 4405 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-551e2b-79ddfccbf8-6rkbp" podUID="85f4e570-eb96-4408-b9ed-12e96698c3fb" containerName="payload-processing" containerID="cri-o://9c3795342fd02f046adf5c19a2eff0e2539afcb640e9a67f31b2a0ebe0fa5dfc" gracePeriod=25 Nov 04 03:07:25.930153 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:07:25.930121 4405 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-551e2b-79ddfccbf8-6rkbp" Nov 04 03:07:25.994402 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:07:25.994359 4405 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/85f4e570-eb96-4408-b9ed-12e96698c3fb-kube-api-access-55jkz\" (UniqueName: \"kubernetes.io/projected/85f4e570-eb96-4408-b9ed-12e96698c3fb-kube-api-access-55jkz\") pod \"85f4e570-eb96-4408-b9ed-12e96698c3fb\" (UID: \"85f4e570-eb96-4408-b9ed-12e96698c3fb\") " Nov 04 03:07:25.994614 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:07:25.994441 4405 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/85f4e570-eb96-4408-b9ed-12e96698c3fb-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/85f4e570-eb96-4408-b9ed-12e96698c3fb-plugins-config-volume\") pod \"85f4e570-eb96-4408-b9ed-12e96698c3fb\" (UID: \"85f4e570-eb96-4408-b9ed-12e96698c3fb\") " Nov 04 03:07:25.995076 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:07:25.995041 4405 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/85f4e570-eb96-4408-b9ed-12e96698c3fb-plugins-config-volume" pod "85f4e570-eb96-4408-b9ed-12e96698c3fb" (UID: "85f4e570-eb96-4408-b9ed-12e96698c3fb"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 03:07:25.996741 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:07:25.996712 4405 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/85f4e570-eb96-4408-b9ed-12e96698c3fb-kube-api-access-55jkz" pod "85f4e570-eb96-4408-b9ed-12e96698c3fb" (UID: "85f4e570-eb96-4408-b9ed-12e96698c3fb"). InnerVolumeSpecName "kube-api-access-55jkz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 03:07:26.096381 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:07:26.096344 4405 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-55jkz\" (UniqueName: \"kubernetes.io/projected/85f4e570-eb96-4408-b9ed-12e96698c3fb-kube-api-access-55jkz\") on node \"ip-10-0-134-125.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:07:26.096381 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:07:26.096379 4405 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/85f4e570-eb96-4408-b9ed-12e96698c3fb-plugins-config-volume\") on node \"ip-10-0-134-125.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:07:26.355132 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:07:26.355099 4405 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-22e72e-7ffdf7c747-vbt9t" Nov 04 03:07:26.355304 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:07:26.355170 4405 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-a-22e72e-85b6d5f746-nw6tv" Nov 04 03:07:26.443587 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:07:26.443549 4405 generic.go:356] "Generic (PLEG): container finished" podID="85f4e570-eb96-4408-b9ed-12e96698c3fb" containerID="9c3795342fd02f046adf5c19a2eff0e2539afcb640e9a67f31b2a0ebe0fa5dfc" exitCode=0 Nov 04 03:07:26.443783 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:07:26.443610 4405 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-551e2b-79ddfccbf8-6rkbp" Nov 04 03:07:26.443783 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:07:26.443637 4405 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-551e2b-79ddfccbf8-6rkbp" event={"ID":"85f4e570-eb96-4408-b9ed-12e96698c3fb","Type":"ContainerDied","Data":"9c3795342fd02f046adf5c19a2eff0e2539afcb640e9a67f31b2a0ebe0fa5dfc"} Nov 04 03:07:26.443783 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:07:26.443706 4405 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-551e2b-79ddfccbf8-6rkbp" event={"ID":"85f4e570-eb96-4408-b9ed-12e96698c3fb","Type":"ContainerDied","Data":"696ecf2d574b10dd9f4e3cdf18b50deb0d445f676af0a11faf457f0fb719477d"} Nov 04 03:07:26.443783 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:07:26.443734 4405 scope.go:122] "RemoveContainer" containerID="9c3795342fd02f046adf5c19a2eff0e2539afcb640e9a67f31b2a0ebe0fa5dfc" Nov 04 03:07:26.455087 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:07:26.455062 4405 scope.go:122] "RemoveContainer" containerID="9c3795342fd02f046adf5c19a2eff0e2539afcb640e9a67f31b2a0ebe0fa5dfc" Nov 04 03:07:26.455382 ip-10-0-134-125 kubenswrapper[4405]: E1104 03:07:26.455362 4405 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9c3795342fd02f046adf5c19a2eff0e2539afcb640e9a67f31b2a0ebe0fa5dfc\": container with ID starting with 9c3795342fd02f046adf5c19a2eff0e2539afcb640e9a67f31b2a0ebe0fa5dfc not found: ID does not exist" containerID="9c3795342fd02f046adf5c19a2eff0e2539afcb640e9a67f31b2a0ebe0fa5dfc" Nov 04 03:07:26.455450 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:07:26.455391 4405 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9c3795342fd02f046adf5c19a2eff0e2539afcb640e9a67f31b2a0ebe0fa5dfc"} err="failed to get container status \"9c3795342fd02f046adf5c19a2eff0e2539afcb640e9a67f31b2a0ebe0fa5dfc\": rpc error: code = NotFound desc = could not find container \"9c3795342fd02f046adf5c19a2eff0e2539afcb640e9a67f31b2a0ebe0fa5dfc\": container with ID starting with 9c3795342fd02f046adf5c19a2eff0e2539afcb640e9a67f31b2a0ebe0fa5dfc not found: ID does not exist" Nov 04 03:07:26.711038 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:07:26.710953 4405 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w1-551e2b-79ddfccbf8-6rkbp"] Nov 04 03:07:26.714530 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:07:26.714508 4405 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w1-551e2b-79ddfccbf8-6rkbp"] Nov 04 03:07:27.702804 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:07:27.702763 4405 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="85f4e570-eb96-4408-b9ed-12e96698c3fb" path="/var/lib/kubelet/pods/85f4e570-eb96-4408-b9ed-12e96698c3fb/volumes" Nov 04 03:07:33.974975 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:07:33.974949 4405 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-vwcx8_ffbb2d83-5a93-4a5b-bce3-e40afb8f58ea/ovn-acl-logging/0.log" Nov 04 03:07:33.977863 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:07:33.977837 4405 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-vwcx8_ffbb2d83-5a93-4a5b-bce3-e40afb8f58ea/ovn-acl-logging/0.log" Nov 04 03:07:36.886794 ip-10-0-134-125 kubenswrapper[4405]: E1104 03:07:36.886756 4405 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5f2vb" containerName="extractor" Nov 04 03:07:56.023912 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:07:56.023824 4405 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-9f4199-85bd8d95f6-6k95j"] Nov 04 03:07:56.024579 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:07:56.024558 4405 cpu_manager.go:414] "containerMap: removing container" podUID="6ff412ca-a023-4cde-8c75-6f9acbfb5121" containerName="payload-processing" Nov 04 03:07:56.024690 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:07:56.024581 4405 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6ff412ca-a023-4cde-8c75-6f9acbfb5121" containerName="payload-processing" Nov 04 03:07:56.024690 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:07:56.024620 4405 cpu_manager.go:414] "containerMap: removing container" podUID="85f4e570-eb96-4408-b9ed-12e96698c3fb" containerName="payload-processing" Nov 04 03:07:56.024690 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:07:56.024625 4405 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="85f4e570-eb96-4408-b9ed-12e96698c3fb" containerName="payload-processing" Nov 04 03:07:56.024690 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:07:56.024641 4405 cpu_manager.go:414] "containerMap: removing container" podUID="449e5858-ec18-4e0e-ab17-520312115a05" containerName="payload-pre-processing" Nov 04 03:07:56.024690 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:07:56.024648 4405 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="449e5858-ec18-4e0e-ab17-520312115a05" containerName="payload-pre-processing" Nov 04 03:07:56.024959 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:07:56.024817 4405 memory_manager.go:365] "RemoveStaleState removing state" podUID="85f4e570-eb96-4408-b9ed-12e96698c3fb" containerName="payload-processing" Nov 04 03:07:56.024959 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:07:56.024832 4405 memory_manager.go:365] "RemoveStaleState removing state" podUID="449e5858-ec18-4e0e-ab17-520312115a05" containerName="payload-pre-processing" Nov 04 03:07:56.024959 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:07:56.024845 4405 memory_manager.go:365] "RemoveStaleState removing state" podUID="6ff412ca-a023-4cde-8c75-6f9acbfb5121" containerName="payload-processing" Nov 04 03:07:56.030417 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:07:56.030395 4405 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-9f4199-85bd8d95f6-6k95j" Nov 04 03:07:56.083849 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:07:56.083816 4405 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-a-9f4199-serving-cert\"" Nov 04 03:07:56.140005 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:07:56.139971 4405 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-9f4199-85bd8d95f6-6k95j"] Nov 04 03:07:56.154113 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:07:56.154078 4405 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/97c488a6-ee10-46bf-bc00-ef8578040f88-metrics-certs\") pod \"maas-api-e2e-inf-a-9f4199-85bd8d95f6-6k95j\" (UID: \"97c488a6-ee10-46bf-bc00-ef8578040f88\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-9f4199-85bd8d95f6-6k95j" Nov 04 03:07:56.154354 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:07:56.154335 4405 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dkmvj\" (UniqueName: \"kubernetes.io/projected/97c488a6-ee10-46bf-bc00-ef8578040f88-kube-api-access-dkmvj\") pod \"maas-api-e2e-inf-a-9f4199-85bd8d95f6-6k95j\" (UID: \"97c488a6-ee10-46bf-bc00-ef8578040f88\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-9f4199-85bd8d95f6-6k95j" Nov 04 03:07:56.154437 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:07:56.154380 4405 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/97c488a6-ee10-46bf-bc00-ef8578040f88-openshift-service-ca\") pod \"maas-api-e2e-inf-a-9f4199-85bd8d95f6-6k95j\" (UID: \"97c488a6-ee10-46bf-bc00-ef8578040f88\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-9f4199-85bd8d95f6-6k95j" Nov 04 03:07:56.154544 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:07:56.154526 4405 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/97c488a6-ee10-46bf-bc00-ef8578040f88-maas-api-tls\") pod \"maas-api-e2e-inf-a-9f4199-85bd8d95f6-6k95j\" (UID: \"97c488a6-ee10-46bf-bc00-ef8578040f88\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-9f4199-85bd8d95f6-6k95j" Nov 04 03:07:56.255841 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:07:56.255806 4405 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/97c488a6-ee10-46bf-bc00-ef8578040f88-openshift-service-ca\") pod \"maas-api-e2e-inf-a-9f4199-85bd8d95f6-6k95j\" (UID: \"97c488a6-ee10-46bf-bc00-ef8578040f88\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-9f4199-85bd8d95f6-6k95j" Nov 04 03:07:56.256022 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:07:56.255895 4405 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/97c488a6-ee10-46bf-bc00-ef8578040f88-maas-api-tls\") pod \"maas-api-e2e-inf-a-9f4199-85bd8d95f6-6k95j\" (UID: \"97c488a6-ee10-46bf-bc00-ef8578040f88\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-9f4199-85bd8d95f6-6k95j" Nov 04 03:07:56.256022 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:07:56.255938 4405 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/97c488a6-ee10-46bf-bc00-ef8578040f88-metrics-certs\") pod \"maas-api-e2e-inf-a-9f4199-85bd8d95f6-6k95j\" (UID: \"97c488a6-ee10-46bf-bc00-ef8578040f88\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-9f4199-85bd8d95f6-6k95j" Nov 04 03:07:56.256104 ip-10-0-134-125 kubenswrapper[4405]: E1104 03:07:56.256055 4405 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-inf-a-9f4199-serving-cert: secret "maas-api-e2e-inf-a-9f4199-serving-cert" not found Nov 04 03:07:56.256104 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:07:56.256084 4405 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dkmvj\" (UniqueName: \"kubernetes.io/projected/97c488a6-ee10-46bf-bc00-ef8578040f88-kube-api-access-dkmvj\") pod \"maas-api-e2e-inf-a-9f4199-85bd8d95f6-6k95j\" (UID: \"97c488a6-ee10-46bf-bc00-ef8578040f88\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-9f4199-85bd8d95f6-6k95j" Nov 04 03:07:56.256183 ip-10-0-134-125 kubenswrapper[4405]: E1104 03:07:56.256146 4405 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/97c488a6-ee10-46bf-bc00-ef8578040f88-maas-api-tls podName:97c488a6-ee10-46bf-bc00-ef8578040f88 nodeName:}" failed. No retries permitted until 2026-11-04 03:07:56.756123053 +0000 UTC m=+2123.186823169 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/97c488a6-ee10-46bf-bc00-ef8578040f88-maas-api-tls") pod "maas-api-e2e-inf-a-9f4199-85bd8d95f6-6k95j" (UID: "97c488a6-ee10-46bf-bc00-ef8578040f88") : secret "maas-api-e2e-inf-a-9f4199-serving-cert" not found Nov 04 03:07:56.256506 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:07:56.256476 4405 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/97c488a6-ee10-46bf-bc00-ef8578040f88-openshift-service-ca\") pod \"maas-api-e2e-inf-a-9f4199-85bd8d95f6-6k95j\" (UID: \"97c488a6-ee10-46bf-bc00-ef8578040f88\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-9f4199-85bd8d95f6-6k95j" Nov 04 03:07:56.258555 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:07:56.258530 4405 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/97c488a6-ee10-46bf-bc00-ef8578040f88-metrics-certs\") pod \"maas-api-e2e-inf-a-9f4199-85bd8d95f6-6k95j\" (UID: \"97c488a6-ee10-46bf-bc00-ef8578040f88\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-9f4199-85bd8d95f6-6k95j" Nov 04 03:07:56.330165 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:07:56.330074 4405 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dkmvj\" (UniqueName: \"kubernetes.io/projected/97c488a6-ee10-46bf-bc00-ef8578040f88-kube-api-access-dkmvj\") pod \"maas-api-e2e-inf-a-9f4199-85bd8d95f6-6k95j\" (UID: \"97c488a6-ee10-46bf-bc00-ef8578040f88\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-9f4199-85bd8d95f6-6k95j" Nov 04 03:07:56.763330 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:07:56.763297 4405 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/97c488a6-ee10-46bf-bc00-ef8578040f88-maas-api-tls\") pod \"maas-api-e2e-inf-a-9f4199-85bd8d95f6-6k95j\" (UID: \"97c488a6-ee10-46bf-bc00-ef8578040f88\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-9f4199-85bd8d95f6-6k95j" Nov 04 03:07:56.766333 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:07:56.766305 4405 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/97c488a6-ee10-46bf-bc00-ef8578040f88-maas-api-tls\") pod \"maas-api-e2e-inf-a-9f4199-85bd8d95f6-6k95j\" (UID: \"97c488a6-ee10-46bf-bc00-ef8578040f88\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-9f4199-85bd8d95f6-6k95j" Nov 04 03:07:56.945997 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:07:56.945960 4405 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-9f4199-85bd8d95f6-6k95j" Nov 04 03:07:57.138223 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:07:57.138189 4405 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-bl5mg"] Nov 04 03:07:57.334601 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:07:57.334572 4405 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-9f4199-85bd8d95f6-6k95j"] Nov 04 03:07:57.336269 ip-10-0-134-125 kubenswrapper[4405]: W1104 03:07:57.336238 4405 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod97c488a6_ee10_46bf_bc00_ef8578040f88.slice/crio-1d6eae8cb911e0e1ff1c3ba64c47be1e99e048cea70937abe27e8a96680ce7de WatchSource:0}: Error finding container 1d6eae8cb911e0e1ff1c3ba64c47be1e99e048cea70937abe27e8a96680ce7de: Status 404 returned error can't find the container with id 1d6eae8cb911e0e1ff1c3ba64c47be1e99e048cea70937abe27e8a96680ce7de Nov 04 03:07:57.338315 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:07:57.338291 4405 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 04 03:07:57.680969 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:07:57.680881 4405 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-9f4199-85bd8d95f6-6k95j" event={"ID":"97c488a6-ee10-46bf-bc00-ef8578040f88","Type":"ContainerStarted","Data":"1d6eae8cb911e0e1ff1c3ba64c47be1e99e048cea70937abe27e8a96680ce7de"} Nov 04 03:07:58.689151 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:07:58.689110 4405 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-9f4199-85bd8d95f6-6k95j" event={"ID":"97c488a6-ee10-46bf-bc00-ef8578040f88","Type":"ContainerStarted","Data":"f71dd36502a0c52e983365f478eca1a00b26224c4ced97a4b3cb302f0660d9ac"} Nov 04 03:07:58.689703 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:07:58.689338 4405 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-9f4199-85bd8d95f6-6k95j" Nov 04 03:07:58.807798 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:07:58.807741 4405 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-9f4199-85bd8d95f6-6k95j" podStartSLOduration=2.932708131 podStartE2EDuration="3.807724389s" totalImagesPullingTime="875.016258ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 03:07:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 03:07:58.805225004 +0000 UTC m=+2125.235925126" watchObservedRunningTime="2026-11-04 03:07:58.807724389 +0000 UTC m=+2125.238424509" Nov 04 03:08:04.703566 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:08:04.703531 4405 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-9f4199-85bd8d95f6-6k95j" Nov 04 03:08:06.885102 ip-10-0-134-125 kubenswrapper[4405]: E1104 03:08:06.885057 4405 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5f2vb" containerName="extractor" Nov 04 03:08:21.901498 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:08:21.901458 4405 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-297806-75b4854d47-mplp7"] Nov 04 03:08:21.901960 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:08:21.901690 4405 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-297806-75b4854d47-mplp7" podUID="ef8afda8-47fe-4cab-85a0-01397c9ab56b" containerName="maas-api" containerID="cri-o://b6b196b6ce9d9840cc4bdd9f34ef1e2c3ab3aef96d0467354500d11dcd08ea8a" gracePeriod=30 Nov 04 03:08:22.058916 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:08:22.058872 4405 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-297806-75b4854d47-mplp7" podUID="ef8afda8-47fe-4cab-85a0-01397c9ab56b" containerName="maas-api" probeResult="failure" output="Get \"https://10.134.0.95:8443/health\": dial tcp 10.134.0.95:8443: connect: connection refused" Nov 04 03:08:22.233711 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:08:22.233624 4405 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-297806-75b4854d47-mplp7" Nov 04 03:08:22.340690 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:08:22.339719 4405 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ef8afda8-47fe-4cab-85a0-01397c9ab56b-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ef8afda8-47fe-4cab-85a0-01397c9ab56b-maas-api-tls\") pod \"ef8afda8-47fe-4cab-85a0-01397c9ab56b\" (UID: \"ef8afda8-47fe-4cab-85a0-01397c9ab56b\") " Nov 04 03:08:22.340690 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:08:22.339819 4405 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ef8afda8-47fe-4cab-85a0-01397c9ab56b-metrics-certs\" (UniqueName: \"kubernetes.io/secret/ef8afda8-47fe-4cab-85a0-01397c9ab56b-metrics-certs\") pod \"ef8afda8-47fe-4cab-85a0-01397c9ab56b\" (UID: \"ef8afda8-47fe-4cab-85a0-01397c9ab56b\") " Nov 04 03:08:22.340690 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:08:22.339861 4405 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ef8afda8-47fe-4cab-85a0-01397c9ab56b-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ef8afda8-47fe-4cab-85a0-01397c9ab56b-openshift-service-ca\") pod \"ef8afda8-47fe-4cab-85a0-01397c9ab56b\" (UID: \"ef8afda8-47fe-4cab-85a0-01397c9ab56b\") " Nov 04 03:08:22.340690 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:08:22.339886 4405 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ef8afda8-47fe-4cab-85a0-01397c9ab56b-kube-api-access-nkq6b\" (UniqueName: \"kubernetes.io/projected/ef8afda8-47fe-4cab-85a0-01397c9ab56b-kube-api-access-nkq6b\") pod \"ef8afda8-47fe-4cab-85a0-01397c9ab56b\" (UID: \"ef8afda8-47fe-4cab-85a0-01397c9ab56b\") " Nov 04 03:08:22.348170 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:08:22.341881 4405 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ef8afda8-47fe-4cab-85a0-01397c9ab56b-openshift-service-ca" pod "ef8afda8-47fe-4cab-85a0-01397c9ab56b" (UID: "ef8afda8-47fe-4cab-85a0-01397c9ab56b"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 03:08:22.348170 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:08:22.344744 4405 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ef8afda8-47fe-4cab-85a0-01397c9ab56b-maas-api-tls" pod "ef8afda8-47fe-4cab-85a0-01397c9ab56b" (UID: "ef8afda8-47fe-4cab-85a0-01397c9ab56b"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 04 03:08:22.358185 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:08:22.350905 4405 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ef8afda8-47fe-4cab-85a0-01397c9ab56b-kube-api-access-nkq6b" pod "ef8afda8-47fe-4cab-85a0-01397c9ab56b" (UID: "ef8afda8-47fe-4cab-85a0-01397c9ab56b"). InnerVolumeSpecName "kube-api-access-nkq6b". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 03:08:22.358185 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:08:22.352497 4405 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ef8afda8-47fe-4cab-85a0-01397c9ab56b-metrics-certs" pod "ef8afda8-47fe-4cab-85a0-01397c9ab56b" (UID: "ef8afda8-47fe-4cab-85a0-01397c9ab56b"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 04 03:08:22.441727 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:08:22.441643 4405 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ef8afda8-47fe-4cab-85a0-01397c9ab56b-metrics-certs\") on node \"ip-10-0-134-125.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:08:22.441727 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:08:22.441724 4405 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ef8afda8-47fe-4cab-85a0-01397c9ab56b-openshift-service-ca\") on node \"ip-10-0-134-125.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:08:22.441727 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:08:22.441737 4405 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nkq6b\" (UniqueName: \"kubernetes.io/projected/ef8afda8-47fe-4cab-85a0-01397c9ab56b-kube-api-access-nkq6b\") on node \"ip-10-0-134-125.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:08:22.441999 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:08:22.441747 4405 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ef8afda8-47fe-4cab-85a0-01397c9ab56b-maas-api-tls\") on node \"ip-10-0-134-125.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:08:22.949264 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:08:22.949219 4405 generic.go:356] "Generic (PLEG): container finished" podID="ef8afda8-47fe-4cab-85a0-01397c9ab56b" containerID="b6b196b6ce9d9840cc4bdd9f34ef1e2c3ab3aef96d0467354500d11dcd08ea8a" exitCode=0 Nov 04 03:08:22.949819 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:08:22.949342 4405 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-297806-75b4854d47-mplp7" event={"ID":"ef8afda8-47fe-4cab-85a0-01397c9ab56b","Type":"ContainerDied","Data":"b6b196b6ce9d9840cc4bdd9f34ef1e2c3ab3aef96d0467354500d11dcd08ea8a"} Nov 04 03:08:22.949819 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:08:22.949429 4405 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-297806-75b4854d47-mplp7" event={"ID":"ef8afda8-47fe-4cab-85a0-01397c9ab56b","Type":"ContainerDied","Data":"263bc5d988b1174a8029467959f4aa6107635580c0df85dbd8b271f79e3a30ed"} Nov 04 03:08:22.949819 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:08:22.949460 4405 scope.go:122] "RemoveContainer" containerID="b6b196b6ce9d9840cc4bdd9f34ef1e2c3ab3aef96d0467354500d11dcd08ea8a" Nov 04 03:08:22.949819 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:08:22.949641 4405 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-297806-75b4854d47-mplp7" Nov 04 03:08:22.974808 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:08:22.974780 4405 scope.go:122] "RemoveContainer" containerID="b6b196b6ce9d9840cc4bdd9f34ef1e2c3ab3aef96d0467354500d11dcd08ea8a" Nov 04 03:08:22.975162 ip-10-0-134-125 kubenswrapper[4405]: E1104 03:08:22.975135 4405 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b6b196b6ce9d9840cc4bdd9f34ef1e2c3ab3aef96d0467354500d11dcd08ea8a\": container with ID starting with b6b196b6ce9d9840cc4bdd9f34ef1e2c3ab3aef96d0467354500d11dcd08ea8a not found: ID does not exist" containerID="b6b196b6ce9d9840cc4bdd9f34ef1e2c3ab3aef96d0467354500d11dcd08ea8a" Nov 04 03:08:22.975254 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:08:22.975175 4405 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b6b196b6ce9d9840cc4bdd9f34ef1e2c3ab3aef96d0467354500d11dcd08ea8a"} err="failed to get container status \"b6b196b6ce9d9840cc4bdd9f34ef1e2c3ab3aef96d0467354500d11dcd08ea8a\": rpc error: code = NotFound desc = could not find container \"b6b196b6ce9d9840cc4bdd9f34ef1e2c3ab3aef96d0467354500d11dcd08ea8a\": container with ID starting with b6b196b6ce9d9840cc4bdd9f34ef1e2c3ab3aef96d0467354500d11dcd08ea8a not found: ID does not exist" Nov 04 03:08:23.119171 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:08:23.119134 4405 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-297806-75b4854d47-mplp7"] Nov 04 03:08:23.169262 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:08:23.169215 4405 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-297806-75b4854d47-mplp7"] Nov 04 03:08:23.703271 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:08:23.703239 4405 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ef8afda8-47fe-4cab-85a0-01397c9ab56b" path="/var/lib/kubelet/pods/ef8afda8-47fe-4cab-85a0-01397c9ab56b/volumes" Nov 04 03:08:36.891208 ip-10-0-134-125 kubenswrapper[4405]: E1104 03:08:36.891159 4405 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5f2vb" containerName="extractor" Nov 04 03:08:45.002249 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:08:45.002215 4405 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-bl5mg"] Nov 04 03:08:49.422699 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:08:49.422643 4405 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w1-d2b8ee-56b894f96d-dbtft"] Nov 04 03:08:54.130846 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:08:54.130812 4405 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-d2b8ee-6ffd864944-9fgxg"] Nov 04 03:08:54.131368 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:08:54.131084 4405 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-d2b8ee-6ffd864944-9fgxg" podUID="541d536e-5ac3-4f98-9f6b-a827a5ef340a" containerName="maas-api" containerID="cri-o://e80ef986f4158c0dcab4283cfdfcc6f2dc2314c4c99e1619955564af2eda604b" gracePeriod=30 Nov 04 03:08:54.342746 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:08:54.342414 4405 generic.go:356] "Generic (PLEG): container finished" podID="541d536e-5ac3-4f98-9f6b-a827a5ef340a" containerID="e80ef986f4158c0dcab4283cfdfcc6f2dc2314c4c99e1619955564af2eda604b" exitCode=0 Nov 04 03:08:54.342746 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:08:54.342611 4405 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-d2b8ee-6ffd864944-9fgxg" event={"ID":"541d536e-5ac3-4f98-9f6b-a827a5ef340a","Type":"ContainerDied","Data":"e80ef986f4158c0dcab4283cfdfcc6f2dc2314c4c99e1619955564af2eda604b"} Nov 04 03:08:54.423572 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:08:54.423528 4405 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-d2b8ee-56b894f96d-dbtft" podUID="595ef150-1fae-478f-8974-0b0efc667be8" containerName="payload-processing" containerID="cri-o://52c79ac172eee50bf33e12a0951aba93690dab4022df88924db189888deef78c" gracePeriod=25 Nov 04 03:08:54.439982 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:08:54.439923 4405 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-d2b8ee-6ffd864944-9fgxg" Nov 04 03:08:54.504719 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:08:54.504685 4405 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/541d536e-5ac3-4f98-9f6b-a827a5ef340a-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/541d536e-5ac3-4f98-9f6b-a827a5ef340a-openshift-service-ca\") pod \"541d536e-5ac3-4f98-9f6b-a827a5ef340a\" (UID: \"541d536e-5ac3-4f98-9f6b-a827a5ef340a\") " Nov 04 03:08:54.504911 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:08:54.504860 4405 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/541d536e-5ac3-4f98-9f6b-a827a5ef340a-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/541d536e-5ac3-4f98-9f6b-a827a5ef340a-maas-api-tls\") pod \"541d536e-5ac3-4f98-9f6b-a827a5ef340a\" (UID: \"541d536e-5ac3-4f98-9f6b-a827a5ef340a\") " Nov 04 03:08:54.505027 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:08:54.505010 4405 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/541d536e-5ac3-4f98-9f6b-a827a5ef340a-metrics-certs\" (UniqueName: \"kubernetes.io/secret/541d536e-5ac3-4f98-9f6b-a827a5ef340a-metrics-certs\") pod \"541d536e-5ac3-4f98-9f6b-a827a5ef340a\" (UID: \"541d536e-5ac3-4f98-9f6b-a827a5ef340a\") " Nov 04 03:08:54.505136 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:08:54.505097 4405 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/541d536e-5ac3-4f98-9f6b-a827a5ef340a-openshift-service-ca" pod "541d536e-5ac3-4f98-9f6b-a827a5ef340a" (UID: "541d536e-5ac3-4f98-9f6b-a827a5ef340a"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 03:08:54.505206 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:08:54.505130 4405 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/541d536e-5ac3-4f98-9f6b-a827a5ef340a-kube-api-access-gphgn\" (UniqueName: \"kubernetes.io/projected/541d536e-5ac3-4f98-9f6b-a827a5ef340a-kube-api-access-gphgn\") pod \"541d536e-5ac3-4f98-9f6b-a827a5ef340a\" (UID: \"541d536e-5ac3-4f98-9f6b-a827a5ef340a\") " Nov 04 03:08:54.505988 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:08:54.505960 4405 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/541d536e-5ac3-4f98-9f6b-a827a5ef340a-openshift-service-ca\") on node \"ip-10-0-134-125.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:08:54.507172 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:08:54.507150 4405 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/541d536e-5ac3-4f98-9f6b-a827a5ef340a-maas-api-tls" pod "541d536e-5ac3-4f98-9f6b-a827a5ef340a" (UID: "541d536e-5ac3-4f98-9f6b-a827a5ef340a"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 04 03:08:54.507281 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:08:54.507234 4405 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/541d536e-5ac3-4f98-9f6b-a827a5ef340a-metrics-certs" pod "541d536e-5ac3-4f98-9f6b-a827a5ef340a" (UID: "541d536e-5ac3-4f98-9f6b-a827a5ef340a"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 04 03:08:54.507324 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:08:54.507291 4405 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/541d536e-5ac3-4f98-9f6b-a827a5ef340a-kube-api-access-gphgn" pod "541d536e-5ac3-4f98-9f6b-a827a5ef340a" (UID: "541d536e-5ac3-4f98-9f6b-a827a5ef340a"). InnerVolumeSpecName "kube-api-access-gphgn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 03:08:54.610956 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:08:54.610918 4405 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gphgn\" (UniqueName: \"kubernetes.io/projected/541d536e-5ac3-4f98-9f6b-a827a5ef340a-kube-api-access-gphgn\") on node \"ip-10-0-134-125.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:08:54.611191 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:08:54.611171 4405 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/541d536e-5ac3-4f98-9f6b-a827a5ef340a-maas-api-tls\") on node \"ip-10-0-134-125.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:08:54.611309 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:08:54.611296 4405 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/541d536e-5ac3-4f98-9f6b-a827a5ef340a-metrics-certs\") on node \"ip-10-0-134-125.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:08:54.725320 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:08:54.725282 4405 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-d2b8ee-56b894f96d-dbtft" Nov 04 03:08:54.825707 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:08:54.815069 4405 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/595ef150-1fae-478f-8974-0b0efc667be8-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/595ef150-1fae-478f-8974-0b0efc667be8-plugins-config-volume\") pod \"595ef150-1fae-478f-8974-0b0efc667be8\" (UID: \"595ef150-1fae-478f-8974-0b0efc667be8\") " Nov 04 03:08:54.825707 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:08:54.815131 4405 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/595ef150-1fae-478f-8974-0b0efc667be8-kube-api-access-qdg2g\" (UniqueName: \"kubernetes.io/projected/595ef150-1fae-478f-8974-0b0efc667be8-kube-api-access-qdg2g\") pod \"595ef150-1fae-478f-8974-0b0efc667be8\" (UID: \"595ef150-1fae-478f-8974-0b0efc667be8\") " Nov 04 03:08:54.825707 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:08:54.815513 4405 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/595ef150-1fae-478f-8974-0b0efc667be8-plugins-config-volume" pod "595ef150-1fae-478f-8974-0b0efc667be8" (UID: "595ef150-1fae-478f-8974-0b0efc667be8"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 03:08:54.825707 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:08:54.815975 4405 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/595ef150-1fae-478f-8974-0b0efc667be8-plugins-config-volume\") on node \"ip-10-0-134-125.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:08:54.851477 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:08:54.851442 4405 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/595ef150-1fae-478f-8974-0b0efc667be8-kube-api-access-qdg2g" pod "595ef150-1fae-478f-8974-0b0efc667be8" (UID: "595ef150-1fae-478f-8974-0b0efc667be8"). InnerVolumeSpecName "kube-api-access-qdg2g". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 03:08:54.917955 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:08:54.917916 4405 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qdg2g\" (UniqueName: \"kubernetes.io/projected/595ef150-1fae-478f-8974-0b0efc667be8-kube-api-access-qdg2g\") on node \"ip-10-0-134-125.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:08:55.369655 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:08:55.369627 4405 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-d2b8ee-6ffd864944-9fgxg" Nov 04 03:08:55.370307 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:08:55.369640 4405 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-d2b8ee-6ffd864944-9fgxg" event={"ID":"541d536e-5ac3-4f98-9f6b-a827a5ef340a","Type":"ContainerDied","Data":"8aff5795d4a2cc8c7e8a7570cd20c3d122e5f6f260f8145e042a434fa4aa6c8a"} Nov 04 03:08:55.370429 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:08:55.370380 4405 scope.go:122] "RemoveContainer" containerID="e80ef986f4158c0dcab4283cfdfcc6f2dc2314c4c99e1619955564af2eda604b" Nov 04 03:08:55.372474 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:08:55.372022 4405 generic.go:356] "Generic (PLEG): container finished" podID="595ef150-1fae-478f-8974-0b0efc667be8" containerID="52c79ac172eee50bf33e12a0951aba93690dab4022df88924db189888deef78c" exitCode=0 Nov 04 03:08:55.372474 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:08:55.372115 4405 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-d2b8ee-56b894f96d-dbtft" event={"ID":"595ef150-1fae-478f-8974-0b0efc667be8","Type":"ContainerDied","Data":"52c79ac172eee50bf33e12a0951aba93690dab4022df88924db189888deef78c"} Nov 04 03:08:55.372474 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:08:55.372153 4405 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-d2b8ee-56b894f96d-dbtft" Nov 04 03:08:55.372684 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:08:55.372159 4405 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-d2b8ee-56b894f96d-dbtft" event={"ID":"595ef150-1fae-478f-8974-0b0efc667be8","Type":"ContainerDied","Data":"962418f2bc79ca6c614af94cf20064d0a02500034635449af92de3a7b78e990c"} Nov 04 03:08:55.385206 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:08:55.385188 4405 scope.go:122] "RemoveContainer" containerID="52c79ac172eee50bf33e12a0951aba93690dab4022df88924db189888deef78c" Nov 04 03:08:55.396373 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:08:55.396346 4405 scope.go:122] "RemoveContainer" containerID="52c79ac172eee50bf33e12a0951aba93690dab4022df88924db189888deef78c" Nov 04 03:08:55.396807 ip-10-0-134-125 kubenswrapper[4405]: E1104 03:08:55.396761 4405 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"52c79ac172eee50bf33e12a0951aba93690dab4022df88924db189888deef78c\": container with ID starting with 52c79ac172eee50bf33e12a0951aba93690dab4022df88924db189888deef78c not found: ID does not exist" containerID="52c79ac172eee50bf33e12a0951aba93690dab4022df88924db189888deef78c" Nov 04 03:08:55.396807 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:08:55.396796 4405 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"52c79ac172eee50bf33e12a0951aba93690dab4022df88924db189888deef78c"} err="failed to get container status \"52c79ac172eee50bf33e12a0951aba93690dab4022df88924db189888deef78c\": rpc error: code = NotFound desc = could not find container \"52c79ac172eee50bf33e12a0951aba93690dab4022df88924db189888deef78c\": container with ID starting with 52c79ac172eee50bf33e12a0951aba93690dab4022df88924db189888deef78c not found: ID does not exist" Nov 04 03:08:55.537419 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:08:55.537332 4405 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w1-d2b8ee-56b894f96d-dbtft"] Nov 04 03:08:55.585439 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:08:55.585405 4405 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w1-d2b8ee-56b894f96d-dbtft"] Nov 04 03:08:55.646643 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:08:55.646611 4405 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-d2b8ee-6ffd864944-9fgxg"] Nov 04 03:08:55.650521 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:08:55.650496 4405 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-d2b8ee-6ffd864944-9fgxg"] Nov 04 03:08:55.708026 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:08:55.707986 4405 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="541d536e-5ac3-4f98-9f6b-a827a5ef340a" path="/var/lib/kubelet/pods/541d536e-5ac3-4f98-9f6b-a827a5ef340a/volumes" Nov 04 03:08:55.708692 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:08:55.708654 4405 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="595ef150-1fae-478f-8974-0b0efc667be8" path="/var/lib/kubelet/pods/595ef150-1fae-478f-8974-0b0efc667be8/volumes" Nov 04 03:09:03.099781 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:09:03.099744 4405 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-d2b8ee-openshift-default-6f9c87f479-qpj8z"] Nov 04 03:09:03.100356 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:09:03.100002 4405 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w1-d2b8ee-openshift-default-6f9c87f479-qpj8z" podUID="05f6d23b-ab70-4e62-9065-41f075c26ab6" containerName="istio-proxy" containerID="cri-o://53d64ae561768b772001dc3e78a94d2c43c3af8d3eb2c22d4f465a47376c8d4b" gracePeriod=30 Nov 04 03:09:04.571344 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:09:04.563142 4405 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w5-4b2bf4-openshift-default-59785c9d99-b52gb"] Nov 04 03:09:04.571344 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:09:04.563501 4405 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w5-4b2bf4-openshift-default-59785c9d99-b52gb" podUID="e21df403-3af7-4a07-8793-452782d57a3b" containerName="istio-proxy" containerID="cri-o://d34e8eb65f5fcad3a97427105b2cda652ec0581ac447867457d6f0460d4c7e6b" gracePeriod=30 Nov 04 03:09:06.890773 ip-10-0-134-125 kubenswrapper[4405]: E1104 03:09:06.890735 4405 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5f2vb" containerName="extractor" Nov 04 03:09:08.394749 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:09:08.394727 4405 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w1-d2b8ee-openshift-default-6f9c87f479-qpj8z" Nov 04 03:09:08.470546 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:09:08.470509 4405 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/05f6d23b-ab70-4e62-9065-41f075c26ab6-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/05f6d23b-ab70-4e62-9065-41f075c26ab6-istio-podinfo\") pod \"05f6d23b-ab70-4e62-9065-41f075c26ab6\" (UID: \"05f6d23b-ab70-4e62-9065-41f075c26ab6\") " Nov 04 03:09:08.470814 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:09:08.470601 4405 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/05f6d23b-ab70-4e62-9065-41f075c26ab6-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/05f6d23b-ab70-4e62-9065-41f075c26ab6-credential-socket\") pod \"05f6d23b-ab70-4e62-9065-41f075c26ab6\" (UID: \"05f6d23b-ab70-4e62-9065-41f075c26ab6\") " Nov 04 03:09:08.470814 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:09:08.470627 4405 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/05f6d23b-ab70-4e62-9065-41f075c26ab6-istio-token\" (UniqueName: \"kubernetes.io/projected/05f6d23b-ab70-4e62-9065-41f075c26ab6-istio-token\") pod \"05f6d23b-ab70-4e62-9065-41f075c26ab6\" (UID: \"05f6d23b-ab70-4e62-9065-41f075c26ab6\") " Nov 04 03:09:08.470814 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:09:08.470753 4405 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/05f6d23b-ab70-4e62-9065-41f075c26ab6-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/05f6d23b-ab70-4e62-9065-41f075c26ab6-istiod-ca-cert\") pod \"05f6d23b-ab70-4e62-9065-41f075c26ab6\" (UID: \"05f6d23b-ab70-4e62-9065-41f075c26ab6\") " Nov 04 03:09:08.471005 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:09:08.470845 4405 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/05f6d23b-ab70-4e62-9065-41f075c26ab6-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/05f6d23b-ab70-4e62-9065-41f075c26ab6-workload-certs\") pod \"05f6d23b-ab70-4e62-9065-41f075c26ab6\" (UID: \"05f6d23b-ab70-4e62-9065-41f075c26ab6\") " Nov 04 03:09:08.471005 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:09:08.470921 4405 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/05f6d23b-ab70-4e62-9065-41f075c26ab6-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/05f6d23b-ab70-4e62-9065-41f075c26ab6-workload-socket\") pod \"05f6d23b-ab70-4e62-9065-41f075c26ab6\" (UID: \"05f6d23b-ab70-4e62-9065-41f075c26ab6\") " Nov 04 03:09:08.471005 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:09:08.470971 4405 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/05f6d23b-ab70-4e62-9065-41f075c26ab6-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/05f6d23b-ab70-4e62-9065-41f075c26ab6-istio-data\") pod \"05f6d23b-ab70-4e62-9065-41f075c26ab6\" (UID: \"05f6d23b-ab70-4e62-9065-41f075c26ab6\") " Nov 04 03:09:08.471005 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:09:08.470965 4405 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/05f6d23b-ab70-4e62-9065-41f075c26ab6-credential-socket" pod "05f6d23b-ab70-4e62-9065-41f075c26ab6" (UID: "05f6d23b-ab70-4e62-9065-41f075c26ab6"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 03:09:08.471220 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:09:08.471051 4405 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/05f6d23b-ab70-4e62-9065-41f075c26ab6-kube-api-access-q6qv9\" (UniqueName: \"kubernetes.io/projected/05f6d23b-ab70-4e62-9065-41f075c26ab6-kube-api-access-q6qv9\") pod \"05f6d23b-ab70-4e62-9065-41f075c26ab6\" (UID: \"05f6d23b-ab70-4e62-9065-41f075c26ab6\") " Nov 04 03:09:08.471220 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:09:08.471091 4405 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/05f6d23b-ab70-4e62-9065-41f075c26ab6-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/05f6d23b-ab70-4e62-9065-41f075c26ab6-istio-envoy\") pod \"05f6d23b-ab70-4e62-9065-41f075c26ab6\" (UID: \"05f6d23b-ab70-4e62-9065-41f075c26ab6\") " Nov 04 03:09:08.471220 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:09:08.471144 4405 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/05f6d23b-ab70-4e62-9065-41f075c26ab6-istiod-ca-cert" pod "05f6d23b-ab70-4e62-9065-41f075c26ab6" (UID: "05f6d23b-ab70-4e62-9065-41f075c26ab6"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 03:09:08.471220 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:09:08.471147 4405 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/05f6d23b-ab70-4e62-9065-41f075c26ab6-workload-socket" pod "05f6d23b-ab70-4e62-9065-41f075c26ab6" (UID: "05f6d23b-ab70-4e62-9065-41f075c26ab6"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 03:09:08.471607 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:09:08.471275 4405 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/05f6d23b-ab70-4e62-9065-41f075c26ab6-istio-data" pod "05f6d23b-ab70-4e62-9065-41f075c26ab6" (UID: "05f6d23b-ab70-4e62-9065-41f075c26ab6"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 03:09:08.471607 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:09:08.471332 4405 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/05f6d23b-ab70-4e62-9065-41f075c26ab6-workload-certs" pod "05f6d23b-ab70-4e62-9065-41f075c26ab6" (UID: "05f6d23b-ab70-4e62-9065-41f075c26ab6"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 03:09:08.471780 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:09:08.471760 4405 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/05f6d23b-ab70-4e62-9065-41f075c26ab6-istiod-ca-cert\") on node \"ip-10-0-134-125.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:09:08.471780 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:09:08.471779 4405 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/05f6d23b-ab70-4e62-9065-41f075c26ab6-workload-certs\") on node \"ip-10-0-134-125.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:09:08.471911 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:09:08.471789 4405 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/05f6d23b-ab70-4e62-9065-41f075c26ab6-workload-socket\") on node \"ip-10-0-134-125.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:09:08.471911 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:09:08.471798 4405 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/05f6d23b-ab70-4e62-9065-41f075c26ab6-istio-data\") on node \"ip-10-0-134-125.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:09:08.471911 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:09:08.471807 4405 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/05f6d23b-ab70-4e62-9065-41f075c26ab6-credential-socket\") on node \"ip-10-0-134-125.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:09:08.473237 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:09:08.473212 4405 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/05f6d23b-ab70-4e62-9065-41f075c26ab6-istio-token" pod "05f6d23b-ab70-4e62-9065-41f075c26ab6" (UID: "05f6d23b-ab70-4e62-9065-41f075c26ab6"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 03:09:08.473237 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:09:08.473217 4405 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/05f6d23b-ab70-4e62-9065-41f075c26ab6-istio-podinfo" pod "05f6d23b-ab70-4e62-9065-41f075c26ab6" (UID: "05f6d23b-ab70-4e62-9065-41f075c26ab6"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 04 03:09:08.473836 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:09:08.473808 4405 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/05f6d23b-ab70-4e62-9065-41f075c26ab6-kube-api-access-q6qv9" pod "05f6d23b-ab70-4e62-9065-41f075c26ab6" (UID: "05f6d23b-ab70-4e62-9065-41f075c26ab6"). InnerVolumeSpecName "kube-api-access-q6qv9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 03:09:08.473936 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:09:08.473845 4405 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/05f6d23b-ab70-4e62-9065-41f075c26ab6-istio-envoy" pod "05f6d23b-ab70-4e62-9065-41f075c26ab6" (UID: "05f6d23b-ab70-4e62-9065-41f075c26ab6"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 03:09:08.482240 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:09:08.482210 4405 generic.go:356] "Generic (PLEG): container finished" podID="05f6d23b-ab70-4e62-9065-41f075c26ab6" containerID="53d64ae561768b772001dc3e78a94d2c43c3af8d3eb2c22d4f465a47376c8d4b" exitCode=0 Nov 04 03:09:08.482394 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:09:08.482256 4405 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w1-d2b8ee-openshift-default-6f9c87f479-qpj8z" Nov 04 03:09:08.482394 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:09:08.482287 4405 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-d2b8ee-openshift-default-6f9c87f479-qpj8z" event={"ID":"05f6d23b-ab70-4e62-9065-41f075c26ab6","Type":"ContainerDied","Data":"53d64ae561768b772001dc3e78a94d2c43c3af8d3eb2c22d4f465a47376c8d4b"} Nov 04 03:09:08.482394 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:09:08.482349 4405 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-d2b8ee-openshift-default-6f9c87f479-qpj8z" event={"ID":"05f6d23b-ab70-4e62-9065-41f075c26ab6","Type":"ContainerDied","Data":"b3d139c51befb99571e70a5e3f8e93424c03363bc0855a7a6639dcaf39d729c8"} Nov 04 03:09:08.482394 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:09:08.482378 4405 scope.go:122] "RemoveContainer" containerID="53d64ae561768b772001dc3e78a94d2c43c3af8d3eb2c22d4f465a47376c8d4b" Nov 04 03:09:08.504890 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:09:08.504865 4405 scope.go:122] "RemoveContainer" containerID="53d64ae561768b772001dc3e78a94d2c43c3af8d3eb2c22d4f465a47376c8d4b" Nov 04 03:09:08.505252 ip-10-0-134-125 kubenswrapper[4405]: E1104 03:09:08.505230 4405 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"53d64ae561768b772001dc3e78a94d2c43c3af8d3eb2c22d4f465a47376c8d4b\": container with ID starting with 53d64ae561768b772001dc3e78a94d2c43c3af8d3eb2c22d4f465a47376c8d4b not found: ID does not exist" containerID="53d64ae561768b772001dc3e78a94d2c43c3af8d3eb2c22d4f465a47376c8d4b" Nov 04 03:09:08.505320 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:09:08.505262 4405 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"53d64ae561768b772001dc3e78a94d2c43c3af8d3eb2c22d4f465a47376c8d4b"} err="failed to get container status \"53d64ae561768b772001dc3e78a94d2c43c3af8d3eb2c22d4f465a47376c8d4b\": rpc error: code = NotFound desc = could not find container \"53d64ae561768b772001dc3e78a94d2c43c3af8d3eb2c22d4f465a47376c8d4b\": container with ID starting with 53d64ae561768b772001dc3e78a94d2c43c3af8d3eb2c22d4f465a47376c8d4b not found: ID does not exist" Nov 04 03:09:08.573966 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:09:08.573832 4405 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q6qv9\" (UniqueName: \"kubernetes.io/projected/05f6d23b-ab70-4e62-9065-41f075c26ab6-kube-api-access-q6qv9\") on node \"ip-10-0-134-125.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:09:08.573966 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:09:08.573871 4405 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/05f6d23b-ab70-4e62-9065-41f075c26ab6-istio-envoy\") on node \"ip-10-0-134-125.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:09:08.573966 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:09:08.573891 4405 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/05f6d23b-ab70-4e62-9065-41f075c26ab6-istio-podinfo\") on node \"ip-10-0-134-125.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:09:08.573966 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:09:08.573905 4405 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/05f6d23b-ab70-4e62-9065-41f075c26ab6-istio-token\") on node \"ip-10-0-134-125.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:09:08.699983 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:09:08.699901 4405 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-d2b8ee-openshift-default-6f9c87f479-qpj8z"] Nov 04 03:09:08.724619 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:09:08.724586 4405 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-d2b8ee-openshift-default-6f9c87f479-qpj8z"] Nov 04 03:09:09.704741 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:09:09.704699 4405 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="05f6d23b-ab70-4e62-9065-41f075c26ab6" path="/var/lib/kubelet/pods/05f6d23b-ab70-4e62-9065-41f075c26ab6/volumes" Nov 04 03:09:09.851628 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:09:09.851562 4405 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w5-4b2bf4-openshift-default-59785c9d99-b52gb" Nov 04 03:09:09.892966 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:09:09.892934 4405 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e21df403-3af7-4a07-8793-452782d57a3b-kube-api-access-gbcdk\" (UniqueName: \"kubernetes.io/projected/e21df403-3af7-4a07-8793-452782d57a3b-kube-api-access-gbcdk\") pod \"e21df403-3af7-4a07-8793-452782d57a3b\" (UID: \"e21df403-3af7-4a07-8793-452782d57a3b\") " Nov 04 03:09:09.893117 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:09:09.892993 4405 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e21df403-3af7-4a07-8793-452782d57a3b-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e21df403-3af7-4a07-8793-452782d57a3b-istio-data\") pod \"e21df403-3af7-4a07-8793-452782d57a3b\" (UID: \"e21df403-3af7-4a07-8793-452782d57a3b\") " Nov 04 03:09:09.893117 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:09:09.893056 4405 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e21df403-3af7-4a07-8793-452782d57a3b-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e21df403-3af7-4a07-8793-452782d57a3b-istiod-ca-cert\") pod \"e21df403-3af7-4a07-8793-452782d57a3b\" (UID: \"e21df403-3af7-4a07-8793-452782d57a3b\") " Nov 04 03:09:09.893117 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:09:09.893094 4405 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e21df403-3af7-4a07-8793-452782d57a3b-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e21df403-3af7-4a07-8793-452782d57a3b-credential-socket\") pod \"e21df403-3af7-4a07-8793-452782d57a3b\" (UID: \"e21df403-3af7-4a07-8793-452782d57a3b\") " Nov 04 03:09:09.893261 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:09:09.893191 4405 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e21df403-3af7-4a07-8793-452782d57a3b-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e21df403-3af7-4a07-8793-452782d57a3b-workload-certs\") pod \"e21df403-3af7-4a07-8793-452782d57a3b\" (UID: \"e21df403-3af7-4a07-8793-452782d57a3b\") " Nov 04 03:09:09.893261 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:09:09.893234 4405 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e21df403-3af7-4a07-8793-452782d57a3b-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e21df403-3af7-4a07-8793-452782d57a3b-workload-socket\") pod \"e21df403-3af7-4a07-8793-452782d57a3b\" (UID: \"e21df403-3af7-4a07-8793-452782d57a3b\") " Nov 04 03:09:09.893364 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:09:09.893285 4405 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/e21df403-3af7-4a07-8793-452782d57a3b-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e21df403-3af7-4a07-8793-452782d57a3b-istio-podinfo\") pod \"e21df403-3af7-4a07-8793-452782d57a3b\" (UID: \"e21df403-3af7-4a07-8793-452782d57a3b\") " Nov 04 03:09:09.893364 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:09:09.893288 4405 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e21df403-3af7-4a07-8793-452782d57a3b-istio-data" pod "e21df403-3af7-4a07-8793-452782d57a3b" (UID: "e21df403-3af7-4a07-8793-452782d57a3b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 03:09:09.893364 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:09:09.893331 4405 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e21df403-3af7-4a07-8793-452782d57a3b-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e21df403-3af7-4a07-8793-452782d57a3b-istio-envoy\") pod \"e21df403-3af7-4a07-8793-452782d57a3b\" (UID: \"e21df403-3af7-4a07-8793-452782d57a3b\") " Nov 04 03:09:09.893525 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:09:09.893372 4405 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e21df403-3af7-4a07-8793-452782d57a3b-istio-token\" (UniqueName: \"kubernetes.io/projected/e21df403-3af7-4a07-8793-452782d57a3b-istio-token\") pod \"e21df403-3af7-4a07-8793-452782d57a3b\" (UID: \"e21df403-3af7-4a07-8793-452782d57a3b\") " Nov 04 03:09:09.893585 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:09:09.893434 4405 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e21df403-3af7-4a07-8793-452782d57a3b-istiod-ca-cert" pod "e21df403-3af7-4a07-8793-452782d57a3b" (UID: "e21df403-3af7-4a07-8793-452782d57a3b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 03:09:09.893585 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:09:09.893439 4405 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e21df403-3af7-4a07-8793-452782d57a3b-credential-socket" pod "e21df403-3af7-4a07-8793-452782d57a3b" (UID: "e21df403-3af7-4a07-8793-452782d57a3b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 03:09:09.893585 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:09:09.893506 4405 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e21df403-3af7-4a07-8793-452782d57a3b-workload-certs" pod "e21df403-3af7-4a07-8793-452782d57a3b" (UID: "e21df403-3af7-4a07-8793-452782d57a3b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 03:09:09.893585 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:09:09.893567 4405 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e21df403-3af7-4a07-8793-452782d57a3b-workload-socket" pod "e21df403-3af7-4a07-8793-452782d57a3b" (UID: "e21df403-3af7-4a07-8793-452782d57a3b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 03:09:09.893988 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:09:09.893964 4405 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e21df403-3af7-4a07-8793-452782d57a3b-workload-socket\") on node \"ip-10-0-134-125.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:09:09.893988 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:09:09.893980 4405 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e21df403-3af7-4a07-8793-452782d57a3b-istio-data\") on node \"ip-10-0-134-125.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:09:09.894118 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:09:09.893990 4405 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e21df403-3af7-4a07-8793-452782d57a3b-istiod-ca-cert\") on node \"ip-10-0-134-125.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:09:09.894118 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:09:09.894005 4405 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e21df403-3af7-4a07-8793-452782d57a3b-credential-socket\") on node \"ip-10-0-134-125.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:09:09.894118 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:09:09.894015 4405 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e21df403-3af7-4a07-8793-452782d57a3b-workload-certs\") on node \"ip-10-0-134-125.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:09:09.895574 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:09:09.895551 4405 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e21df403-3af7-4a07-8793-452782d57a3b-kube-api-access-gbcdk" pod "e21df403-3af7-4a07-8793-452782d57a3b" (UID: "e21df403-3af7-4a07-8793-452782d57a3b"). InnerVolumeSpecName "kube-api-access-gbcdk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 03:09:09.895703 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:09:09.895688 4405 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e21df403-3af7-4a07-8793-452782d57a3b-istio-envoy" pod "e21df403-3af7-4a07-8793-452782d57a3b" (UID: "e21df403-3af7-4a07-8793-452782d57a3b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 03:09:09.896116 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:09:09.896092 4405 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e21df403-3af7-4a07-8793-452782d57a3b-istio-podinfo" pod "e21df403-3af7-4a07-8793-452782d57a3b" (UID: "e21df403-3af7-4a07-8793-452782d57a3b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 04 03:09:09.896235 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:09:09.896113 4405 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e21df403-3af7-4a07-8793-452782d57a3b-istio-token" pod "e21df403-3af7-4a07-8793-452782d57a3b" (UID: "e21df403-3af7-4a07-8793-452782d57a3b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 03:09:09.995071 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:09:09.995038 4405 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e21df403-3af7-4a07-8793-452782d57a3b-istio-podinfo\") on node \"ip-10-0-134-125.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:09:09.995071 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:09:09.995067 4405 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e21df403-3af7-4a07-8793-452782d57a3b-istio-envoy\") on node \"ip-10-0-134-125.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:09:09.995071 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:09:09.995079 4405 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e21df403-3af7-4a07-8793-452782d57a3b-istio-token\") on node \"ip-10-0-134-125.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:09:09.995319 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:09:09.995088 4405 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gbcdk\" (UniqueName: \"kubernetes.io/projected/e21df403-3af7-4a07-8793-452782d57a3b-kube-api-access-gbcdk\") on node \"ip-10-0-134-125.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:09:10.499047 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:09:10.499012 4405 generic.go:356] "Generic (PLEG): container finished" podID="e21df403-3af7-4a07-8793-452782d57a3b" containerID="d34e8eb65f5fcad3a97427105b2cda652ec0581ac447867457d6f0460d4c7e6b" exitCode=0 Nov 04 03:09:10.499214 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:09:10.499069 4405 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w5-4b2bf4-openshift-default-59785c9d99-b52gb" Nov 04 03:09:10.499214 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:09:10.499110 4405 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w5-4b2bf4-openshift-default-59785c9d99-b52gb" event={"ID":"e21df403-3af7-4a07-8793-452782d57a3b","Type":"ContainerDied","Data":"d34e8eb65f5fcad3a97427105b2cda652ec0581ac447867457d6f0460d4c7e6b"} Nov 04 03:09:10.499214 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:09:10.499170 4405 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w5-4b2bf4-openshift-default-59785c9d99-b52gb" event={"ID":"e21df403-3af7-4a07-8793-452782d57a3b","Type":"ContainerDied","Data":"a6fae144760f3a31dee73b22255cade2589b6bb3ad44f50bbfb0f2f88051abcd"} Nov 04 03:09:10.499214 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:09:10.499194 4405 scope.go:122] "RemoveContainer" containerID="d34e8eb65f5fcad3a97427105b2cda652ec0581ac447867457d6f0460d4c7e6b" Nov 04 03:09:10.510747 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:09:10.510728 4405 scope.go:122] "RemoveContainer" containerID="8e03d4d6805ad573b5dd4a47fc3ba1380e7ed057cac59b1280b3a354a3b3933e" Nov 04 03:09:10.521623 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:09:10.521598 4405 scope.go:122] "RemoveContainer" containerID="d34e8eb65f5fcad3a97427105b2cda652ec0581ac447867457d6f0460d4c7e6b" Nov 04 03:09:10.521913 ip-10-0-134-125 kubenswrapper[4405]: E1104 03:09:10.521890 4405 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d34e8eb65f5fcad3a97427105b2cda652ec0581ac447867457d6f0460d4c7e6b\": container with ID starting with d34e8eb65f5fcad3a97427105b2cda652ec0581ac447867457d6f0460d4c7e6b not found: ID does not exist" containerID="d34e8eb65f5fcad3a97427105b2cda652ec0581ac447867457d6f0460d4c7e6b" Nov 04 03:09:10.521995 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:09:10.521926 4405 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d34e8eb65f5fcad3a97427105b2cda652ec0581ac447867457d6f0460d4c7e6b"} err="failed to get container status \"d34e8eb65f5fcad3a97427105b2cda652ec0581ac447867457d6f0460d4c7e6b\": rpc error: code = NotFound desc = could not find container \"d34e8eb65f5fcad3a97427105b2cda652ec0581ac447867457d6f0460d4c7e6b\": container with ID starting with d34e8eb65f5fcad3a97427105b2cda652ec0581ac447867457d6f0460d4c7e6b not found: ID does not exist" Nov 04 03:09:10.521995 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:09:10.521950 4405 scope.go:122] "RemoveContainer" containerID="8e03d4d6805ad573b5dd4a47fc3ba1380e7ed057cac59b1280b3a354a3b3933e" Nov 04 03:09:10.522223 ip-10-0-134-125 kubenswrapper[4405]: E1104 03:09:10.522205 4405 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8e03d4d6805ad573b5dd4a47fc3ba1380e7ed057cac59b1280b3a354a3b3933e\": container with ID starting with 8e03d4d6805ad573b5dd4a47fc3ba1380e7ed057cac59b1280b3a354a3b3933e not found: ID does not exist" containerID="8e03d4d6805ad573b5dd4a47fc3ba1380e7ed057cac59b1280b3a354a3b3933e" Nov 04 03:09:10.522265 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:09:10.522230 4405 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8e03d4d6805ad573b5dd4a47fc3ba1380e7ed057cac59b1280b3a354a3b3933e"} err="failed to get container status \"8e03d4d6805ad573b5dd4a47fc3ba1380e7ed057cac59b1280b3a354a3b3933e\": rpc error: code = NotFound desc = could not find container \"8e03d4d6805ad573b5dd4a47fc3ba1380e7ed057cac59b1280b3a354a3b3933e\": container with ID starting with 8e03d4d6805ad573b5dd4a47fc3ba1380e7ed057cac59b1280b3a354a3b3933e not found: ID does not exist" Nov 04 03:09:10.665392 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:09:10.665350 4405 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w5-4b2bf4-openshift-default-59785c9d99-b52gb"] Nov 04 03:09:10.714739 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:09:10.714705 4405 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w5-4b2bf4-openshift-default-59785c9d99-b52gb"] Nov 04 03:09:11.704255 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:09:11.704221 4405 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e21df403-3af7-4a07-8793-452782d57a3b" path="/var/lib/kubelet/pods/e21df403-3af7-4a07-8793-452782d57a3b/volumes" Nov 04 03:09:27.186190 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:09:27.186153 4405 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-76597d6855-jjxxp"] Nov 04 03:09:27.186703 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:09:27.186432 4405 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-jjxxp" podUID="121fbac1-908f-4c37-8efa-f17d449ed6b9" containerName="istio-proxy" containerID="cri-o://a2a9f16593a0b563618f94c943923072e9f1c5a1d145cd8c5735c9974dba0cc8" gracePeriod=30 Nov 04 03:09:27.237986 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:09:27.237948 4405 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-76597d6855-dtlt4"] Nov 04 03:09:27.238228 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:09:27.238200 4405 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-dtlt4" podUID="3918cd06-f979-4c62-a75f-916151257784" containerName="istio-proxy" containerID="cri-o://fc53a9a9008d3810a8389307a3b49d36237f3d36cf06b43c2f838d30676feb91" gracePeriod=30 Nov 04 03:09:28.285620 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:09:28.285583 4405 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-9f4199-55cf88d648-fr4zd"] Nov 04 03:09:28.396777 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:09:28.396682 4405 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-9f4199-84b49988cf-thdfx"] Nov 04 03:09:33.301725 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:09:33.296532 4405 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-a-9f4199-55cf88d648-fr4zd" podUID="01be67f1-0362-4960-af5b-4ddad895ab4a" containerName="payload-processing" containerID="cri-o://db40ec687e8d94fb92b9732be62d42f0f8ebc51cd768858cf8b918d8f519cb40" gracePeriod=25 Nov 04 03:09:33.347693 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:09:33.330083 4405 generic.go:356] "Generic (PLEG): container finished" podID="121fbac1-908f-4c37-8efa-f17d449ed6b9" containerID="a2a9f16593a0b563618f94c943923072e9f1c5a1d145cd8c5735c9974dba0cc8" exitCode=0 Nov 04 03:09:33.347693 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:09:33.330381 4405 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-jjxxp" event={"ID":"121fbac1-908f-4c37-8efa-f17d449ed6b9","Type":"ContainerDied","Data":"a2a9f16593a0b563618f94c943923072e9f1c5a1d145cd8c5735c9974dba0cc8"} Nov 04 03:09:33.439729 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:09:33.435834 4405 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-9f4199-84b49988cf-thdfx" podUID="a6be9827-0b28-45b8-bcbe-242840761447" containerName="payload-pre-processing" containerID="cri-o://417395d9f1353766a912bfa1a7a3b69228e3c2a877036ad4a75b71f943a00d19" gracePeriod=25 Nov 04 03:09:33.491761 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:09:33.489076 4405 generic.go:356] "Generic (PLEG): container finished" podID="3918cd06-f979-4c62-a75f-916151257784" containerID="fc53a9a9008d3810a8389307a3b49d36237f3d36cf06b43c2f838d30676feb91" exitCode=0 Nov 04 03:09:33.491761 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:09:33.489246 4405 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-dtlt4" event={"ID":"3918cd06-f979-4c62-a75f-916151257784","Type":"ContainerDied","Data":"fc53a9a9008d3810a8389307a3b49d36237f3d36cf06b43c2f838d30676feb91"} Nov 04 03:09:33.573699 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:09:33.572777 4405 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-ebb7fd/distinct-2-w6-ebb7fd-kserve-6f68c5db86-zrpjs" podUID="b7b786e0-9c94-4126-9949-cd354ebbfb2a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 03:09:33.739694 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:09:33.738786 4405 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76597d6855-dtlt4 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.88:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 04 03:09:33.739694 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:09:33.738878 4405 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-dtlt4" podUID="3918cd06-f979-4c62-a75f-916151257784" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.88:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 04 03:09:33.831321 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:09:33.816075 4405 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-e516ff/distinct-w1-e516ff-kserve-f4474cb9c-glpzw" podUID="8842ad09-22ff-4492-ba5d-d4cacf4eac75" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 04 03:09:33.843932 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:09:33.843896 4405 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-dtlt4" Nov 04 03:09:33.866694 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:09:33.864207 4405 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-a0bcbc/premium-w2-a0bcbc-kserve-78dfd6d78c-lwth2" podUID="33b620c1-389c-48b4-b277-90252018c4b6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 03:09:33.921699 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:09:33.909498 4405 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-cdcf67/distinct-w3-cdcf67-kserve-667cc7f7d7-5m2lq" podUID="52ed22cc-96e4-4ef0-b34d-ed58c89a9183" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 03:09:33.963695 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:09:33.961859 4405 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-ljhhw" podUID="f66b1861-d87b-4d57-ab27-f1a24544a534" containerName="main" probeResult="failure" output="Get \"https://10.134.0.26:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 03:09:33.988700 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:09:33.985640 4405 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76597d6855-jjxxp container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.83:15021/healthz/ready\": dial tcp 10.134.0.83:15021: connect: connection refused" start-of-body= Nov 04 03:09:33.988700 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:09:33.985759 4405 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-jjxxp" podUID="121fbac1-908f-4c37-8efa-f17d449ed6b9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.83:15021/healthz/ready\": dial tcp 10.134.0.83:15021: connect: connection refused" Nov 04 03:09:34.027172 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:09:34.027134 4405 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3918cd06-f979-4c62-a75f-916151257784-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3918cd06-f979-4c62-a75f-916151257784-istiod-ca-cert\") pod \"3918cd06-f979-4c62-a75f-916151257784\" (UID: \"3918cd06-f979-4c62-a75f-916151257784\") " Nov 04 03:09:34.029459 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:09:34.029431 4405 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3918cd06-f979-4c62-a75f-916151257784-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3918cd06-f979-4c62-a75f-916151257784-istio-data\") pod \"3918cd06-f979-4c62-a75f-916151257784\" (UID: \"3918cd06-f979-4c62-a75f-916151257784\") " Nov 04 03:09:34.029641 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:09:34.029625 4405 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3918cd06-f979-4c62-a75f-916151257784-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3918cd06-f979-4c62-a75f-916151257784-workload-certs\") pod \"3918cd06-f979-4c62-a75f-916151257784\" (UID: \"3918cd06-f979-4c62-a75f-916151257784\") " Nov 04 03:09:34.029878 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:09:34.029858 4405 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3918cd06-f979-4c62-a75f-916151257784-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3918cd06-f979-4c62-a75f-916151257784-workload-socket\") pod \"3918cd06-f979-4c62-a75f-916151257784\" (UID: \"3918cd06-f979-4c62-a75f-916151257784\") " Nov 04 03:09:34.030119 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:09:34.030100 4405 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3918cd06-f979-4c62-a75f-916151257784-kube-api-access-vd2mb\" (UniqueName: \"kubernetes.io/projected/3918cd06-f979-4c62-a75f-916151257784-kube-api-access-vd2mb\") pod \"3918cd06-f979-4c62-a75f-916151257784\" (UID: \"3918cd06-f979-4c62-a75f-916151257784\") " Nov 04 03:09:34.030311 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:09:34.030293 4405 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3918cd06-f979-4c62-a75f-916151257784-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3918cd06-f979-4c62-a75f-916151257784-credential-socket\") pod \"3918cd06-f979-4c62-a75f-916151257784\" (UID: \"3918cd06-f979-4c62-a75f-916151257784\") " Nov 04 03:09:34.030509 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:09:34.030492 4405 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3918cd06-f979-4c62-a75f-916151257784-istio-token\" (UniqueName: \"kubernetes.io/projected/3918cd06-f979-4c62-a75f-916151257784-istio-token\") pod \"3918cd06-f979-4c62-a75f-916151257784\" (UID: \"3918cd06-f979-4c62-a75f-916151257784\") " Nov 04 03:09:34.030650 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:09:34.030633 4405 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/3918cd06-f979-4c62-a75f-916151257784-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3918cd06-f979-4c62-a75f-916151257784-istio-podinfo\") pod \"3918cd06-f979-4c62-a75f-916151257784\" (UID: \"3918cd06-f979-4c62-a75f-916151257784\") " Nov 04 03:09:34.039887 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:09:34.039865 4405 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3918cd06-f979-4c62-a75f-916151257784-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3918cd06-f979-4c62-a75f-916151257784-istio-envoy\") pod \"3918cd06-f979-4c62-a75f-916151257784\" (UID: \"3918cd06-f979-4c62-a75f-916151257784\") " Nov 04 03:09:34.087571 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:09:34.028596 4405 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-203c35/simulator-w4-203c35-kserve-f9c4bd695-999ns" podUID="eb0d8bdd-6808-4fd0-b9b5-567dff3e7ca5" containerName="main" probeResult="failure" output="Get \"https://10.134.0.46:8000/ready\": context deadline exceeded" Nov 04 03:09:34.087571 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:09:34.068888 4405 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3918cd06-f979-4c62-a75f-916151257784-istiod-ca-cert" pod "3918cd06-f979-4c62-a75f-916151257784" (UID: "3918cd06-f979-4c62-a75f-916151257784"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 03:09:34.087571 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:09:34.069198 4405 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3918cd06-f979-4c62-a75f-916151257784-credential-socket" pod "3918cd06-f979-4c62-a75f-916151257784" (UID: "3918cd06-f979-4c62-a75f-916151257784"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 03:09:34.087571 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:09:34.069424 4405 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3918cd06-f979-4c62-a75f-916151257784-workload-socket" pod "3918cd06-f979-4c62-a75f-916151257784" (UID: "3918cd06-f979-4c62-a75f-916151257784"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 03:09:34.087571 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:09:34.071043 4405 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3918cd06-f979-4c62-a75f-916151257784-workload-certs" pod "3918cd06-f979-4c62-a75f-916151257784" (UID: "3918cd06-f979-4c62-a75f-916151257784"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 03:09:34.087571 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:09:34.071763 4405 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3918cd06-f979-4c62-a75f-916151257784-istio-data" pod "3918cd06-f979-4c62-a75f-916151257784" (UID: "3918cd06-f979-4c62-a75f-916151257784"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 03:09:34.087571 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:09:34.079957 4405 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/3918cd06-f979-4c62-a75f-916151257784-istio-podinfo" pod "3918cd06-f979-4c62-a75f-916151257784" (UID: "3918cd06-f979-4c62-a75f-916151257784"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 04 03:09:34.091083 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:09:34.091047 4405 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3918cd06-f979-4c62-a75f-916151257784-istio-envoy" pod "3918cd06-f979-4c62-a75f-916151257784" (UID: "3918cd06-f979-4c62-a75f-916151257784"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 03:09:34.091195 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:09:34.091145 4405 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3918cd06-f979-4c62-a75f-916151257784-istio-token" pod "3918cd06-f979-4c62-a75f-916151257784" (UID: "3918cd06-f979-4c62-a75f-916151257784"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 03:09:34.092137 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:09:34.092105 4405 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3918cd06-f979-4c62-a75f-916151257784-kube-api-access-vd2mb" pod "3918cd06-f979-4c62-a75f-916151257784" (UID: "3918cd06-f979-4c62-a75f-916151257784"). InnerVolumeSpecName "kube-api-access-vd2mb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 03:09:34.189785 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:09:34.168646 4405 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3918cd06-f979-4c62-a75f-916151257784-workload-socket\") on node \"ip-10-0-134-125.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:09:34.190052 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:09:34.190030 4405 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3918cd06-f979-4c62-a75f-916151257784-credential-socket\") on node \"ip-10-0-134-125.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:09:34.190166 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:09:34.190153 4405 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3918cd06-f979-4c62-a75f-916151257784-istio-token\") on node \"ip-10-0-134-125.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:09:34.190254 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:09:34.190243 4405 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3918cd06-f979-4c62-a75f-916151257784-istio-podinfo\") on node \"ip-10-0-134-125.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:09:34.190343 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:09:34.190331 4405 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3918cd06-f979-4c62-a75f-916151257784-istio-envoy\") on node \"ip-10-0-134-125.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:09:34.190430 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:09:34.190418 4405 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3918cd06-f979-4c62-a75f-916151257784-istiod-ca-cert\") on node \"ip-10-0-134-125.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:09:34.190521 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:09:34.190510 4405 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3918cd06-f979-4c62-a75f-916151257784-istio-data\") on node \"ip-10-0-134-125.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:09:34.190608 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:09:34.190596 4405 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3918cd06-f979-4c62-a75f-916151257784-workload-certs\") on node \"ip-10-0-134-125.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:09:34.361447 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:09:34.354937 4405 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vd2mb\" (UniqueName: \"kubernetes.io/projected/3918cd06-f979-4c62-a75f-916151257784-kube-api-access-vd2mb\") on node \"ip-10-0-134-125.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:09:34.447378 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:09:34.447347 4405 scope.go:122] "RemoveContainer" containerID="fc53a9a9008d3810a8389307a3b49d36237f3d36cf06b43c2f838d30676feb91" Nov 04 03:09:34.570695 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:09:34.560914 4405 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-dtlt4" event={"ID":"3918cd06-f979-4c62-a75f-916151257784","Type":"ContainerDied","Data":"00baa01c68564641aa37129e2fab5a1bd5e532cfb57e3ceacd79c81b7afb09fe"} Nov 04 03:09:34.570695 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:09:34.561046 4405 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-dtlt4" Nov 04 03:09:34.570695 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:09:34.568906 4405 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-jjxxp" Nov 04 03:09:34.595978 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:09:34.588548 4405 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/121fbac1-908f-4c37-8efa-f17d449ed6b9-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/121fbac1-908f-4c37-8efa-f17d449ed6b9-istio-podinfo\") pod \"121fbac1-908f-4c37-8efa-f17d449ed6b9\" (UID: \"121fbac1-908f-4c37-8efa-f17d449ed6b9\") " Nov 04 03:09:34.595978 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:09:34.588592 4405 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/121fbac1-908f-4c37-8efa-f17d449ed6b9-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/121fbac1-908f-4c37-8efa-f17d449ed6b9-istio-envoy\") pod \"121fbac1-908f-4c37-8efa-f17d449ed6b9\" (UID: \"121fbac1-908f-4c37-8efa-f17d449ed6b9\") " Nov 04 03:09:34.595978 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:09:34.588624 4405 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/121fbac1-908f-4c37-8efa-f17d449ed6b9-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/121fbac1-908f-4c37-8efa-f17d449ed6b9-istiod-ca-cert\") pod \"121fbac1-908f-4c37-8efa-f17d449ed6b9\" (UID: \"121fbac1-908f-4c37-8efa-f17d449ed6b9\") " Nov 04 03:09:34.595978 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:09:34.588714 4405 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/121fbac1-908f-4c37-8efa-f17d449ed6b9-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/121fbac1-908f-4c37-8efa-f17d449ed6b9-workload-socket\") pod \"121fbac1-908f-4c37-8efa-f17d449ed6b9\" (UID: \"121fbac1-908f-4c37-8efa-f17d449ed6b9\") " Nov 04 03:09:34.595978 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:09:34.588807 4405 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/121fbac1-908f-4c37-8efa-f17d449ed6b9-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/121fbac1-908f-4c37-8efa-f17d449ed6b9-istio-data\") pod \"121fbac1-908f-4c37-8efa-f17d449ed6b9\" (UID: \"121fbac1-908f-4c37-8efa-f17d449ed6b9\") " Nov 04 03:09:34.595978 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:09:34.588849 4405 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/121fbac1-908f-4c37-8efa-f17d449ed6b9-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/121fbac1-908f-4c37-8efa-f17d449ed6b9-workload-certs\") pod \"121fbac1-908f-4c37-8efa-f17d449ed6b9\" (UID: \"121fbac1-908f-4c37-8efa-f17d449ed6b9\") " Nov 04 03:09:34.595978 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:09:34.588905 4405 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/121fbac1-908f-4c37-8efa-f17d449ed6b9-kube-api-access-f5zkl\" (UniqueName: \"kubernetes.io/projected/121fbac1-908f-4c37-8efa-f17d449ed6b9-kube-api-access-f5zkl\") pod \"121fbac1-908f-4c37-8efa-f17d449ed6b9\" (UID: \"121fbac1-908f-4c37-8efa-f17d449ed6b9\") " Nov 04 03:09:34.595978 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:09:34.588992 4405 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/121fbac1-908f-4c37-8efa-f17d449ed6b9-istio-token\" (UniqueName: \"kubernetes.io/projected/121fbac1-908f-4c37-8efa-f17d449ed6b9-istio-token\") pod \"121fbac1-908f-4c37-8efa-f17d449ed6b9\" (UID: \"121fbac1-908f-4c37-8efa-f17d449ed6b9\") " Nov 04 03:09:34.595978 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:09:34.589092 4405 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/121fbac1-908f-4c37-8efa-f17d449ed6b9-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/121fbac1-908f-4c37-8efa-f17d449ed6b9-credential-socket\") pod \"121fbac1-908f-4c37-8efa-f17d449ed6b9\" (UID: \"121fbac1-908f-4c37-8efa-f17d449ed6b9\") " Nov 04 03:09:34.595978 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:09:34.590322 4405 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/121fbac1-908f-4c37-8efa-f17d449ed6b9-workload-socket" pod "121fbac1-908f-4c37-8efa-f17d449ed6b9" (UID: "121fbac1-908f-4c37-8efa-f17d449ed6b9"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 03:09:34.595978 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:09:34.592531 4405 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/121fbac1-908f-4c37-8efa-f17d449ed6b9-workload-certs" pod "121fbac1-908f-4c37-8efa-f17d449ed6b9" (UID: "121fbac1-908f-4c37-8efa-f17d449ed6b9"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 03:09:34.595978 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:09:34.592742 4405 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/121fbac1-908f-4c37-8efa-f17d449ed6b9-istio-data" pod "121fbac1-908f-4c37-8efa-f17d449ed6b9" (UID: "121fbac1-908f-4c37-8efa-f17d449ed6b9"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 03:09:34.615752 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:09:34.607730 4405 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/121fbac1-908f-4c37-8efa-f17d449ed6b9-credential-socket" pod "121fbac1-908f-4c37-8efa-f17d449ed6b9" (UID: "121fbac1-908f-4c37-8efa-f17d449ed6b9"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 03:09:34.666355 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:09:34.624864 4405 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/121fbac1-908f-4c37-8efa-f17d449ed6b9-istiod-ca-cert" pod "121fbac1-908f-4c37-8efa-f17d449ed6b9" (UID: "121fbac1-908f-4c37-8efa-f17d449ed6b9"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 03:09:34.666355 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:09:34.627793 4405 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/121fbac1-908f-4c37-8efa-f17d449ed6b9-kube-api-access-f5zkl" pod "121fbac1-908f-4c37-8efa-f17d449ed6b9" (UID: "121fbac1-908f-4c37-8efa-f17d449ed6b9"). InnerVolumeSpecName "kube-api-access-f5zkl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 03:09:34.666355 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:09:34.637052 4405 generic.go:356] "Generic (PLEG): container finished" podID="01be67f1-0362-4960-af5b-4ddad895ab4a" containerID="db40ec687e8d94fb92b9732be62d42f0f8ebc51cd768858cf8b918d8f519cb40" exitCode=0 Nov 04 03:09:34.666355 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:09:34.637194 4405 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-9f4199-55cf88d648-fr4zd" event={"ID":"01be67f1-0362-4960-af5b-4ddad895ab4a","Type":"ContainerDied","Data":"db40ec687e8d94fb92b9732be62d42f0f8ebc51cd768858cf8b918d8f519cb40"} Nov 04 03:09:34.667625 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:09:34.667549 4405 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/121fbac1-908f-4c37-8efa-f17d449ed6b9-istio-podinfo" pod "121fbac1-908f-4c37-8efa-f17d449ed6b9" (UID: "121fbac1-908f-4c37-8efa-f17d449ed6b9"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 04 03:09:34.673312 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:09:34.673279 4405 generic.go:356] "Generic (PLEG): container finished" podID="a6be9827-0b28-45b8-bcbe-242840761447" containerID="417395d9f1353766a912bfa1a7a3b69228e3c2a877036ad4a75b71f943a00d19" exitCode=0 Nov 04 03:09:34.673798 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:09:34.673772 4405 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-9f4199-84b49988cf-thdfx" event={"ID":"a6be9827-0b28-45b8-bcbe-242840761447","Type":"ContainerDied","Data":"417395d9f1353766a912bfa1a7a3b69228e3c2a877036ad4a75b71f943a00d19"} Nov 04 03:09:34.674643 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:09:34.674296 4405 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/121fbac1-908f-4c37-8efa-f17d449ed6b9-istio-token" pod "121fbac1-908f-4c37-8efa-f17d449ed6b9" (UID: "121fbac1-908f-4c37-8efa-f17d449ed6b9"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 03:09:34.676143 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:09:34.676121 4405 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-jjxxp" event={"ID":"121fbac1-908f-4c37-8efa-f17d449ed6b9","Type":"ContainerDied","Data":"54e4270711abffd3cd536a7336bc85a507d4d19ee8cd641a890d3c2ab167d1e3"} Nov 04 03:09:34.676295 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:09:34.676281 4405 scope.go:122] "RemoveContainer" containerID="a2a9f16593a0b563618f94c943923072e9f1c5a1d145cd8c5735c9974dba0cc8" Nov 04 03:09:34.676551 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:09:34.676535 4405 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-jjxxp" Nov 04 03:09:34.682426 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:09:34.680232 4405 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/121fbac1-908f-4c37-8efa-f17d449ed6b9-istio-envoy" pod "121fbac1-908f-4c37-8efa-f17d449ed6b9" (UID: "121fbac1-908f-4c37-8efa-f17d449ed6b9"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 03:09:34.740107 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:09:34.740062 4405 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/121fbac1-908f-4c37-8efa-f17d449ed6b9-istio-token\") on node \"ip-10-0-134-125.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:09:34.740371 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:09:34.740351 4405 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/121fbac1-908f-4c37-8efa-f17d449ed6b9-credential-socket\") on node \"ip-10-0-134-125.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:09:34.740586 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:09:34.740551 4405 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/121fbac1-908f-4c37-8efa-f17d449ed6b9-istio-podinfo\") on node \"ip-10-0-134-125.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:09:34.740763 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:09:34.740746 4405 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/121fbac1-908f-4c37-8efa-f17d449ed6b9-istio-envoy\") on node \"ip-10-0-134-125.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:09:34.740907 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:09:34.740876 4405 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/121fbac1-908f-4c37-8efa-f17d449ed6b9-istiod-ca-cert\") on node \"ip-10-0-134-125.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:09:34.741054 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:09:34.741025 4405 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/121fbac1-908f-4c37-8efa-f17d449ed6b9-workload-socket\") on node \"ip-10-0-134-125.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:09:34.741172 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:09:34.741158 4405 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/121fbac1-908f-4c37-8efa-f17d449ed6b9-istio-data\") on node \"ip-10-0-134-125.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:09:34.741304 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:09:34.741273 4405 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/121fbac1-908f-4c37-8efa-f17d449ed6b9-workload-certs\") on node \"ip-10-0-134-125.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:09:34.741458 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:09:34.741424 4405 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f5zkl\" (UniqueName: \"kubernetes.io/projected/121fbac1-908f-4c37-8efa-f17d449ed6b9-kube-api-access-f5zkl\") on node \"ip-10-0-134-125.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:09:34.758913 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:09:34.756437 4405 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-9f4199-84b49988cf-thdfx" Nov 04 03:09:34.843513 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:09:34.843485 4405 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a6be9827-0b28-45b8-bcbe-242840761447-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a6be9827-0b28-45b8-bcbe-242840761447-plugins-config-volume\") pod \"a6be9827-0b28-45b8-bcbe-242840761447\" (UID: \"a6be9827-0b28-45b8-bcbe-242840761447\") " Nov 04 03:09:34.843661 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:09:34.843539 4405 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a6be9827-0b28-45b8-bcbe-242840761447-kube-api-access-rwmt8\" (UniqueName: \"kubernetes.io/projected/a6be9827-0b28-45b8-bcbe-242840761447-kube-api-access-rwmt8\") pod \"a6be9827-0b28-45b8-bcbe-242840761447\" (UID: \"a6be9827-0b28-45b8-bcbe-242840761447\") " Nov 04 03:09:34.844018 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:09:34.843987 4405 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a6be9827-0b28-45b8-bcbe-242840761447-plugins-config-volume" pod "a6be9827-0b28-45b8-bcbe-242840761447" (UID: "a6be9827-0b28-45b8-bcbe-242840761447"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 03:09:34.844269 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:09:34.844248 4405 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a6be9827-0b28-45b8-bcbe-242840761447-plugins-config-volume\") on node \"ip-10-0-134-125.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:09:34.851830 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:09:34.850963 4405 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a6be9827-0b28-45b8-bcbe-242840761447-kube-api-access-rwmt8" pod "a6be9827-0b28-45b8-bcbe-242840761447" (UID: "a6be9827-0b28-45b8-bcbe-242840761447"). InnerVolumeSpecName "kube-api-access-rwmt8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 03:09:34.856531 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:09:34.856492 4405 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-9f4199-55cf88d648-fr4zd" Nov 04 03:09:34.922892 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:09:34.922787 4405 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-76597d6855-dtlt4"] Nov 04 03:09:34.928006 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:09:34.927968 4405 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-76597d6855-dtlt4"] Nov 04 03:09:34.945716 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:09:34.945695 4405 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/01be67f1-0362-4960-af5b-4ddad895ab4a-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/01be67f1-0362-4960-af5b-4ddad895ab4a-plugins-config-volume\") pod \"01be67f1-0362-4960-af5b-4ddad895ab4a\" (UID: \"01be67f1-0362-4960-af5b-4ddad895ab4a\") " Nov 04 03:09:34.945850 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:09:34.945837 4405 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/01be67f1-0362-4960-af5b-4ddad895ab4a-kube-api-access-jp6jf\" (UniqueName: \"kubernetes.io/projected/01be67f1-0362-4960-af5b-4ddad895ab4a-kube-api-access-jp6jf\") pod \"01be67f1-0362-4960-af5b-4ddad895ab4a\" (UID: \"01be67f1-0362-4960-af5b-4ddad895ab4a\") " Nov 04 03:09:34.946136 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:09:34.946103 4405 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/01be67f1-0362-4960-af5b-4ddad895ab4a-plugins-config-volume" pod "01be67f1-0362-4960-af5b-4ddad895ab4a" (UID: "01be67f1-0362-4960-af5b-4ddad895ab4a"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 03:09:34.946311 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:09:34.946295 4405 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rwmt8\" (UniqueName: \"kubernetes.io/projected/a6be9827-0b28-45b8-bcbe-242840761447-kube-api-access-rwmt8\") on node \"ip-10-0-134-125.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:09:34.946369 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:09:34.946316 4405 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/01be67f1-0362-4960-af5b-4ddad895ab4a-plugins-config-volume\") on node \"ip-10-0-134-125.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:09:34.948052 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:09:34.948031 4405 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/01be67f1-0362-4960-af5b-4ddad895ab4a-kube-api-access-jp6jf" pod "01be67f1-0362-4960-af5b-4ddad895ab4a" (UID: "01be67f1-0362-4960-af5b-4ddad895ab4a"). InnerVolumeSpecName "kube-api-access-jp6jf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 03:09:35.048200 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:09:35.048162 4405 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jp6jf\" (UniqueName: \"kubernetes.io/projected/01be67f1-0362-4960-af5b-4ddad895ab4a-kube-api-access-jp6jf\") on node \"ip-10-0-134-125.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:09:35.260346 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:09:35.260265 4405 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-76597d6855-jjxxp"] Nov 04 03:09:35.266827 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:09:35.266796 4405 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-76597d6855-jjxxp"] Nov 04 03:09:35.685087 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:09:35.685059 4405 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-9f4199-55cf88d648-fr4zd" Nov 04 03:09:35.685586 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:09:35.685086 4405 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-9f4199-55cf88d648-fr4zd" event={"ID":"01be67f1-0362-4960-af5b-4ddad895ab4a","Type":"ContainerDied","Data":"6812c9b13d39516b32375f7fb675cf7d4a0b218ef08b56016abb6224dcb7973a"} Nov 04 03:09:35.685586 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:09:35.685150 4405 scope.go:122] "RemoveContainer" containerID="db40ec687e8d94fb92b9732be62d42f0f8ebc51cd768858cf8b918d8f519cb40" Nov 04 03:09:35.686753 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:09:35.686720 4405 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-9f4199-84b49988cf-thdfx" Nov 04 03:09:35.686753 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:09:35.686735 4405 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-9f4199-84b49988cf-thdfx" event={"ID":"a6be9827-0b28-45b8-bcbe-242840761447","Type":"ContainerDied","Data":"298c2f3243e75aaf64f997ed4d793a4bb019cab48b68f09cbb0347e24123f523"} Nov 04 03:09:35.727566 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:09:35.727544 4405 scope.go:122] "RemoveContainer" containerID="417395d9f1353766a912bfa1a7a3b69228e3c2a877036ad4a75b71f943a00d19" Nov 04 03:09:35.731073 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:09:35.731048 4405 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="121fbac1-908f-4c37-8efa-f17d449ed6b9" path="/var/lib/kubelet/pods/121fbac1-908f-4c37-8efa-f17d449ed6b9/volumes" Nov 04 03:09:35.731569 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:09:35.731555 4405 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3918cd06-f979-4c62-a75f-916151257784" path="/var/lib/kubelet/pods/3918cd06-f979-4c62-a75f-916151257784/volumes" Nov 04 03:09:35.872858 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:09:35.872825 4405 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-9f4199-84b49988cf-thdfx"] Nov 04 03:09:35.876617 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:09:35.876588 4405 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-9f4199-84b49988cf-thdfx"] Nov 04 03:09:35.985914 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:09:35.985835 4405 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-9f4199-55cf88d648-fr4zd"] Nov 04 03:09:36.016740 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:09:36.013838 4405 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-9f4199-55cf88d648-fr4zd"] Nov 04 03:09:36.885281 ip-10-0-134-125 kubenswrapper[4405]: E1104 03:09:36.885231 4405 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5f2vb" containerName="extractor" Nov 04 03:09:37.702763 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:09:37.702731 4405 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="01be67f1-0362-4960-af5b-4ddad895ab4a" path="/var/lib/kubelet/pods/01be67f1-0362-4960-af5b-4ddad895ab4a/volumes" Nov 04 03:09:37.703165 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:09:37.703150 4405 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a6be9827-0b28-45b8-bcbe-242840761447" path="/var/lib/kubelet/pods/a6be9827-0b28-45b8-bcbe-242840761447/volumes" Nov 04 03:09:41.774585 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:09:41.774548 4405 generic.go:356] "Generic (PLEG): container finished" podID="fd747189-5360-4360-83cd-07a4e414fda4" containerID="27b80f527345011ee23f01f21487a67b9800509257b6b45c3ec35831fdfe7633" exitCode=0 Nov 04 03:09:41.775065 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:09:41.774622 4405 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-22e72e-openshift-default-64656d9fb9-26n5c" event={"ID":"fd747189-5360-4360-83cd-07a4e414fda4","Type":"ContainerDied","Data":"27b80f527345011ee23f01f21487a67b9800509257b6b45c3ec35831fdfe7633"} Nov 04 03:09:41.775118 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:09:41.775063 4405 scope.go:122] "RemoveContainer" containerID="27b80f527345011ee23f01f21487a67b9800509257b6b45c3ec35831fdfe7633" Nov 04 03:09:41.776814 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:09:41.776784 4405 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 04 03:09:41.776925 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:09:41.776856 4405 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 04 03:09:41.776925 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:09:41.776889 4405 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 04 03:09:42.640171 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:09:42.640136 4405 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-ebb7fd-openshift-default-544f5768f-mngkl"] Nov 04 03:09:42.640392 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:09:42.640356 4405 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w6-ebb7fd-openshift-default-544f5768f-mngkl" podUID="f55884d8-38e1-416b-853d-9c09082444bf" containerName="istio-proxy" containerID="cri-o://a6f9bfbf61349ba11b6f631437bbf329fa68d34d50afb063d904657629571c89" gracePeriod=30 Nov 04 03:09:42.784417 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:09:42.784377 4405 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-22e72e-openshift-default-64656d9fb9-26n5c" event={"ID":"fd747189-5360-4360-83cd-07a4e414fda4","Type":"ContainerStarted","Data":"9038e37f5c39ccd27ebcaedfcbfff9feb82d7fa234e173e625fe1ebdc0d3f11e"} Nov 04 03:09:43.358912 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:09:43.358876 4405 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-22e72e-openshift-default-64656d9fb9-26n5c" Nov 04 03:09:43.359117 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:09:43.359092 4405 patch_prober.go:29] interesting pod/e2e-inf-a-22e72e-openshift-default-64656d9fb9-26n5c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.120:15021/healthz/ready\": dial tcp 10.134.0.120:15021: connect: connection refused" start-of-body= Nov 04 03:09:43.359165 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:09:43.359143 4405 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-22e72e-openshift-default-64656d9fb9-26n5c" podUID="fd747189-5360-4360-83cd-07a4e414fda4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.120:15021/healthz/ready\": dial tcp 10.134.0.120:15021: connect: connection refused" Nov 04 03:09:44.359525 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:09:44.359486 4405 patch_prober.go:29] interesting pod/e2e-inf-a-22e72e-openshift-default-64656d9fb9-26n5c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.120:15021/healthz/ready\": dial tcp 10.134.0.120:15021: connect: connection refused" start-of-body= Nov 04 03:09:44.359954 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:09:44.359544 4405 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-22e72e-openshift-default-64656d9fb9-26n5c" podUID="fd747189-5360-4360-83cd-07a4e414fda4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.120:15021/healthz/ready\": dial tcp 10.134.0.120:15021: connect: connection refused" Nov 04 03:09:45.359012 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:09:45.358973 4405 patch_prober.go:29] interesting pod/e2e-inf-a-22e72e-openshift-default-64656d9fb9-26n5c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.120:15021/healthz/ready\": dial tcp 10.134.0.120:15021: connect: connection refused" start-of-body= Nov 04 03:09:45.359204 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:09:45.359046 4405 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-22e72e-openshift-default-64656d9fb9-26n5c" podUID="fd747189-5360-4360-83cd-07a4e414fda4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.120:15021/healthz/ready\": dial tcp 10.134.0.120:15021: connect: connection refused" Nov 04 03:09:46.359048 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:09:46.359006 4405 patch_prober.go:29] interesting pod/e2e-inf-a-22e72e-openshift-default-64656d9fb9-26n5c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.120:15021/healthz/ready\": dial tcp 10.134.0.120:15021: connect: connection refused" start-of-body= Nov 04 03:09:46.359465 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:09:46.359061 4405 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-22e72e-openshift-default-64656d9fb9-26n5c" podUID="fd747189-5360-4360-83cd-07a4e414fda4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.120:15021/healthz/ready\": dial tcp 10.134.0.120:15021: connect: connection refused" Nov 04 03:09:47.147813 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:09:47.147756 4405 patch_prober.go:29] interesting pod/e2e-worker-w6-ebb7fd-openshift-default-544f5768f-mngkl container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 04 03:09:47.148173 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:09:47.148142 4405 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w6-ebb7fd-openshift-default-544f5768f-mngkl" podUID="f55884d8-38e1-416b-853d-9c09082444bf" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 04 03:09:47.360169 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:09:47.360128 4405 patch_prober.go:29] interesting pod/e2e-inf-a-22e72e-openshift-default-64656d9fb9-26n5c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.120:15021/healthz/ready\": dial tcp 10.134.0.120:15021: connect: connection refused" start-of-body= Nov 04 03:09:47.360716 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:09:47.360214 4405 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-22e72e-openshift-default-64656d9fb9-26n5c" podUID="fd747189-5360-4360-83cd-07a4e414fda4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.120:15021/healthz/ready\": dial tcp 10.134.0.120:15021: connect: connection refused" Nov 04 03:09:47.859258 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:09:47.859222 4405 generic.go:356] "Generic (PLEG): container finished" podID="f55884d8-38e1-416b-853d-9c09082444bf" containerID="a6f9bfbf61349ba11b6f631437bbf329fa68d34d50afb063d904657629571c89" exitCode=0 Nov 04 03:09:47.859453 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:09:47.859291 4405 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-ebb7fd-openshift-default-544f5768f-mngkl" event={"ID":"f55884d8-38e1-416b-853d-9c09082444bf","Type":"ContainerDied","Data":"a6f9bfbf61349ba11b6f631437bbf329fa68d34d50afb063d904657629571c89"} Nov 04 03:09:48.364406 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:09:48.364373 4405 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-22e72e-openshift-default-64656d9fb9-26n5c" Nov 04 03:09:48.364964 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:09:48.364780 4405 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-22e72e-openshift-default-64656d9fb9-26n5c" Nov 04 03:09:48.365714 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:09:48.365693 4405 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-22e72e-openshift-default-64656d9fb9-26n5c" Nov 04 03:09:48.378258 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:09:48.378236 4405 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-ebb7fd-openshift-default-544f5768f-mngkl" Nov 04 03:09:48.472821 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:09:48.472727 4405 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f55884d8-38e1-416b-853d-9c09082444bf-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f55884d8-38e1-416b-853d-9c09082444bf-istiod-ca-cert\") pod \"f55884d8-38e1-416b-853d-9c09082444bf\" (UID: \"f55884d8-38e1-416b-853d-9c09082444bf\") " Nov 04 03:09:48.472821 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:09:48.472775 4405 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f55884d8-38e1-416b-853d-9c09082444bf-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f55884d8-38e1-416b-853d-9c09082444bf-workload-socket\") pod \"f55884d8-38e1-416b-853d-9c09082444bf\" (UID: \"f55884d8-38e1-416b-853d-9c09082444bf\") " Nov 04 03:09:48.473062 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:09:48.472828 4405 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f55884d8-38e1-416b-853d-9c09082444bf-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f55884d8-38e1-416b-853d-9c09082444bf-istio-data\") pod \"f55884d8-38e1-416b-853d-9c09082444bf\" (UID: \"f55884d8-38e1-416b-853d-9c09082444bf\") " Nov 04 03:09:48.473062 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:09:48.472866 4405 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f55884d8-38e1-416b-853d-9c09082444bf-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f55884d8-38e1-416b-853d-9c09082444bf-workload-certs\") pod \"f55884d8-38e1-416b-853d-9c09082444bf\" (UID: \"f55884d8-38e1-416b-853d-9c09082444bf\") " Nov 04 03:09:48.473062 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:09:48.472898 4405 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/f55884d8-38e1-416b-853d-9c09082444bf-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f55884d8-38e1-416b-853d-9c09082444bf-istio-podinfo\") pod \"f55884d8-38e1-416b-853d-9c09082444bf\" (UID: \"f55884d8-38e1-416b-853d-9c09082444bf\") " Nov 04 03:09:48.473062 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:09:48.472938 4405 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f55884d8-38e1-416b-853d-9c09082444bf-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f55884d8-38e1-416b-853d-9c09082444bf-credential-socket\") pod \"f55884d8-38e1-416b-853d-9c09082444bf\" (UID: \"f55884d8-38e1-416b-853d-9c09082444bf\") " Nov 04 03:09:48.473062 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:09:48.472997 4405 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f55884d8-38e1-416b-853d-9c09082444bf-kube-api-access-sjjlx\" (UniqueName: \"kubernetes.io/projected/f55884d8-38e1-416b-853d-9c09082444bf-kube-api-access-sjjlx\") pod \"f55884d8-38e1-416b-853d-9c09082444bf\" (UID: \"f55884d8-38e1-416b-853d-9c09082444bf\") " Nov 04 03:09:48.473329 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:09:48.473084 4405 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f55884d8-38e1-416b-853d-9c09082444bf-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f55884d8-38e1-416b-853d-9c09082444bf-istio-envoy\") pod \"f55884d8-38e1-416b-853d-9c09082444bf\" (UID: \"f55884d8-38e1-416b-853d-9c09082444bf\") " Nov 04 03:09:48.473329 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:09:48.473159 4405 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f55884d8-38e1-416b-853d-9c09082444bf-istio-token\" (UniqueName: \"kubernetes.io/projected/f55884d8-38e1-416b-853d-9c09082444bf-istio-token\") pod \"f55884d8-38e1-416b-853d-9c09082444bf\" (UID: \"f55884d8-38e1-416b-853d-9c09082444bf\") " Nov 04 03:09:48.473329 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:09:48.473194 4405 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f55884d8-38e1-416b-853d-9c09082444bf-workload-socket" pod "f55884d8-38e1-416b-853d-9c09082444bf" (UID: "f55884d8-38e1-416b-853d-9c09082444bf"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 03:09:48.473329 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:09:48.473262 4405 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f55884d8-38e1-416b-853d-9c09082444bf-istiod-ca-cert" pod "f55884d8-38e1-416b-853d-9c09082444bf" (UID: "f55884d8-38e1-416b-853d-9c09082444bf"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 03:09:48.473329 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:09:48.473267 4405 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f55884d8-38e1-416b-853d-9c09082444bf-workload-certs" pod "f55884d8-38e1-416b-853d-9c09082444bf" (UID: "f55884d8-38e1-416b-853d-9c09082444bf"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 03:09:48.473601 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:09:48.473533 4405 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f55884d8-38e1-416b-853d-9c09082444bf-credential-socket" pod "f55884d8-38e1-416b-853d-9c09082444bf" (UID: "f55884d8-38e1-416b-853d-9c09082444bf"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 03:09:48.473735 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:09:48.473708 4405 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f55884d8-38e1-416b-853d-9c09082444bf-istio-data" pod "f55884d8-38e1-416b-853d-9c09082444bf" (UID: "f55884d8-38e1-416b-853d-9c09082444bf"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 03:09:48.473895 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:09:48.473797 4405 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f55884d8-38e1-416b-853d-9c09082444bf-istiod-ca-cert\") on node \"ip-10-0-134-125.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:09:48.473984 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:09:48.473905 4405 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f55884d8-38e1-416b-853d-9c09082444bf-workload-socket\") on node \"ip-10-0-134-125.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:09:48.473984 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:09:48.473920 4405 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f55884d8-38e1-416b-853d-9c09082444bf-workload-certs\") on node \"ip-10-0-134-125.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:09:48.473984 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:09:48.473933 4405 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f55884d8-38e1-416b-853d-9c09082444bf-credential-socket\") on node \"ip-10-0-134-125.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:09:48.475770 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:09:48.475741 4405 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f55884d8-38e1-416b-853d-9c09082444bf-istio-envoy" pod "f55884d8-38e1-416b-853d-9c09082444bf" (UID: "f55884d8-38e1-416b-853d-9c09082444bf"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 03:09:48.475950 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:09:48.475929 4405 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f55884d8-38e1-416b-853d-9c09082444bf-kube-api-access-sjjlx" pod "f55884d8-38e1-416b-853d-9c09082444bf" (UID: "f55884d8-38e1-416b-853d-9c09082444bf"). InnerVolumeSpecName "kube-api-access-sjjlx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 03:09:48.476036 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:09:48.476022 4405 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f55884d8-38e1-416b-853d-9c09082444bf-istio-token" pod "f55884d8-38e1-416b-853d-9c09082444bf" (UID: "f55884d8-38e1-416b-853d-9c09082444bf"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 03:09:48.476152 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:09:48.476128 4405 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f55884d8-38e1-416b-853d-9c09082444bf-istio-podinfo" pod "f55884d8-38e1-416b-853d-9c09082444bf" (UID: "f55884d8-38e1-416b-853d-9c09082444bf"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 04 03:09:48.574992 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:09:48.574958 4405 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f55884d8-38e1-416b-853d-9c09082444bf-istio-envoy\") on node \"ip-10-0-134-125.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:09:48.574992 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:09:48.574987 4405 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f55884d8-38e1-416b-853d-9c09082444bf-istio-token\") on node \"ip-10-0-134-125.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:09:48.574992 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:09:48.574996 4405 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f55884d8-38e1-416b-853d-9c09082444bf-istio-data\") on node \"ip-10-0-134-125.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:09:48.575243 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:09:48.575006 4405 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f55884d8-38e1-416b-853d-9c09082444bf-istio-podinfo\") on node \"ip-10-0-134-125.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:09:48.575243 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:09:48.575017 4405 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sjjlx\" (UniqueName: \"kubernetes.io/projected/f55884d8-38e1-416b-853d-9c09082444bf-kube-api-access-sjjlx\") on node \"ip-10-0-134-125.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:09:48.868422 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:09:48.868372 4405 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-ebb7fd-openshift-default-544f5768f-mngkl" Nov 04 03:09:48.868422 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:09:48.868393 4405 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-ebb7fd-openshift-default-544f5768f-mngkl" event={"ID":"f55884d8-38e1-416b-853d-9c09082444bf","Type":"ContainerDied","Data":"5165ac3190835fadeb23be42ae6025ae7e78deea8a4ea0f4df4ab91c817ecaef"} Nov 04 03:09:48.868739 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:09:48.868465 4405 scope.go:122] "RemoveContainer" containerID="a6f9bfbf61349ba11b6f631437bbf329fa68d34d50afb063d904657629571c89" Nov 04 03:09:49.038493 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:09:49.038462 4405 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-ebb7fd-openshift-default-544f5768f-mngkl"] Nov 04 03:09:49.043391 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:09:49.043361 4405 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w6-ebb7fd-openshift-default-544f5768f-mngkl"] Nov 04 03:09:49.702619 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:09:49.702583 4405 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f55884d8-38e1-416b-853d-9c09082444bf" path="/var/lib/kubelet/pods/f55884d8-38e1-416b-853d-9c09082444bf/volumes" Nov 04 03:09:53.608802 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:09:53.608763 4405 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-57f116-79648fc5c8-pg42j"] Nov 04 03:09:53.609243 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:09:53.608995 4405 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-57f116-79648fc5c8-pg42j" podUID="d2227982-1e00-4ef1-9858-1abfe945bb84" containerName="maas-api" containerID="cri-o://34dd768ead4d27389133974c1b192d6e5118c0022b823eea38e38136c1babd2a" gracePeriod=30 Nov 04 03:09:53.906324 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:09:53.906295 4405 generic.go:356] "Generic (PLEG): container finished" podID="d2227982-1e00-4ef1-9858-1abfe945bb84" containerID="34dd768ead4d27389133974c1b192d6e5118c0022b823eea38e38136c1babd2a" exitCode=0 Nov 04 03:09:53.906460 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:09:53.906366 4405 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-57f116-79648fc5c8-pg42j" event={"ID":"d2227982-1e00-4ef1-9858-1abfe945bb84","Type":"ContainerDied","Data":"34dd768ead4d27389133974c1b192d6e5118c0022b823eea38e38136c1babd2a"} Nov 04 03:09:53.906460 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:09:53.906418 4405 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-57f116-79648fc5c8-pg42j" event={"ID":"d2227982-1e00-4ef1-9858-1abfe945bb84","Type":"ContainerDied","Data":"97eb9647ef385aae404d394cfb76300e79c69d53f8480a8ffd9fe90a0b00b578"} Nov 04 03:09:53.906460 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:09:53.906440 4405 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="97eb9647ef385aae404d394cfb76300e79c69d53f8480a8ffd9fe90a0b00b578" Nov 04 03:09:53.920779 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:09:53.920755 4405 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-57f116-79648fc5c8-pg42j" Nov 04 03:09:53.950486 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:09:53.950454 4405 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d2227982-1e00-4ef1-9858-1abfe945bb84-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d2227982-1e00-4ef1-9858-1abfe945bb84-maas-api-tls\") pod \"d2227982-1e00-4ef1-9858-1abfe945bb84\" (UID: \"d2227982-1e00-4ef1-9858-1abfe945bb84\") " Nov 04 03:09:53.950699 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:09:53.950657 4405 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d2227982-1e00-4ef1-9858-1abfe945bb84-metrics-certs\" (UniqueName: \"kubernetes.io/secret/d2227982-1e00-4ef1-9858-1abfe945bb84-metrics-certs\") pod \"d2227982-1e00-4ef1-9858-1abfe945bb84\" (UID: \"d2227982-1e00-4ef1-9858-1abfe945bb84\") " Nov 04 03:09:53.950810 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:09:53.950792 4405 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d2227982-1e00-4ef1-9858-1abfe945bb84-kube-api-access-6m4bx\" (UniqueName: \"kubernetes.io/projected/d2227982-1e00-4ef1-9858-1abfe945bb84-kube-api-access-6m4bx\") pod \"d2227982-1e00-4ef1-9858-1abfe945bb84\" (UID: \"d2227982-1e00-4ef1-9858-1abfe945bb84\") " Nov 04 03:09:53.950919 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:09:53.950900 4405 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d2227982-1e00-4ef1-9858-1abfe945bb84-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d2227982-1e00-4ef1-9858-1abfe945bb84-openshift-service-ca\") pod \"d2227982-1e00-4ef1-9858-1abfe945bb84\" (UID: \"d2227982-1e00-4ef1-9858-1abfe945bb84\") " Nov 04 03:09:53.951307 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:09:53.951268 4405 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d2227982-1e00-4ef1-9858-1abfe945bb84-openshift-service-ca" pod "d2227982-1e00-4ef1-9858-1abfe945bb84" (UID: "d2227982-1e00-4ef1-9858-1abfe945bb84"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 03:09:53.951491 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:09:53.951474 4405 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d2227982-1e00-4ef1-9858-1abfe945bb84-openshift-service-ca\") on node \"ip-10-0-134-125.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:09:53.953138 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:09:53.953110 4405 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d2227982-1e00-4ef1-9858-1abfe945bb84-kube-api-access-6m4bx" pod "d2227982-1e00-4ef1-9858-1abfe945bb84" (UID: "d2227982-1e00-4ef1-9858-1abfe945bb84"). InnerVolumeSpecName "kube-api-access-6m4bx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 03:09:53.953402 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:09:53.953372 4405 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d2227982-1e00-4ef1-9858-1abfe945bb84-maas-api-tls" pod "d2227982-1e00-4ef1-9858-1abfe945bb84" (UID: "d2227982-1e00-4ef1-9858-1abfe945bb84"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 04 03:09:53.953497 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:09:53.953398 4405 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d2227982-1e00-4ef1-9858-1abfe945bb84-metrics-certs" pod "d2227982-1e00-4ef1-9858-1abfe945bb84" (UID: "d2227982-1e00-4ef1-9858-1abfe945bb84"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 04 03:09:54.053107 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:09:54.053071 4405 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d2227982-1e00-4ef1-9858-1abfe945bb84-maas-api-tls\") on node \"ip-10-0-134-125.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:09:54.053107 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:09:54.053104 4405 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d2227982-1e00-4ef1-9858-1abfe945bb84-metrics-certs\") on node \"ip-10-0-134-125.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:09:54.053319 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:09:54.053118 4405 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6m4bx\" (UniqueName: \"kubernetes.io/projected/d2227982-1e00-4ef1-9858-1abfe945bb84-kube-api-access-6m4bx\") on node \"ip-10-0-134-125.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:09:54.913435 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:09:54.913393 4405 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-57f116-79648fc5c8-pg42j" Nov 04 03:09:55.078849 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:09:55.078811 4405 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-57f116-79648fc5c8-pg42j"] Nov 04 03:09:55.129036 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:09:55.129005 4405 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-57f116-79648fc5c8-pg42j"] Nov 04 03:09:55.430466 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:09:55.430428 4405 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-db5ef5-5d5584b699-rjmpj"] Nov 04 03:09:55.431035 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:09:55.431019 4405 cpu_manager.go:414] "containerMap: removing container" podUID="e21df403-3af7-4a07-8793-452782d57a3b" containerName="istio-proxy" Nov 04 03:09:55.431035 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:09:55.431035 4405 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e21df403-3af7-4a07-8793-452782d57a3b" containerName="istio-proxy" Nov 04 03:09:55.431133 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:09:55.431049 4405 cpu_manager.go:414] "containerMap: removing container" podUID="595ef150-1fae-478f-8974-0b0efc667be8" containerName="payload-processing" Nov 04 03:09:55.431133 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:09:55.431054 4405 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="595ef150-1fae-478f-8974-0b0efc667be8" containerName="payload-processing" Nov 04 03:09:55.431133 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:09:55.431061 4405 cpu_manager.go:414] "containerMap: removing container" podUID="541d536e-5ac3-4f98-9f6b-a827a5ef340a" containerName="maas-api" Nov 04 03:09:55.431133 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:09:55.431067 4405 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="541d536e-5ac3-4f98-9f6b-a827a5ef340a" containerName="maas-api" Nov 04 03:09:55.431133 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:09:55.431076 4405 cpu_manager.go:414] "containerMap: removing container" podUID="05f6d23b-ab70-4e62-9065-41f075c26ab6" containerName="istio-proxy" Nov 04 03:09:55.431133 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:09:55.431082 4405 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="05f6d23b-ab70-4e62-9065-41f075c26ab6" containerName="istio-proxy" Nov 04 03:09:55.431133 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:09:55.431091 4405 cpu_manager.go:414] "containerMap: removing container" podUID="3918cd06-f979-4c62-a75f-916151257784" containerName="istio-proxy" Nov 04 03:09:55.431133 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:09:55.431095 4405 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3918cd06-f979-4c62-a75f-916151257784" containerName="istio-proxy" Nov 04 03:09:55.431133 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:09:55.431103 4405 cpu_manager.go:414] "containerMap: removing container" podUID="d2227982-1e00-4ef1-9858-1abfe945bb84" containerName="maas-api" Nov 04 03:09:55.431133 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:09:55.431107 4405 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d2227982-1e00-4ef1-9858-1abfe945bb84" containerName="maas-api" Nov 04 03:09:55.431133 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:09:55.431116 4405 cpu_manager.go:414] "containerMap: removing container" podUID="a6be9827-0b28-45b8-bcbe-242840761447" containerName="payload-pre-processing" Nov 04 03:09:55.431133 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:09:55.431121 4405 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a6be9827-0b28-45b8-bcbe-242840761447" containerName="payload-pre-processing" Nov 04 03:09:55.431133 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:09:55.431136 4405 cpu_manager.go:414] "containerMap: removing container" podUID="ef8afda8-47fe-4cab-85a0-01397c9ab56b" containerName="maas-api" Nov 04 03:09:55.431515 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:09:55.431142 4405 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ef8afda8-47fe-4cab-85a0-01397c9ab56b" containerName="maas-api" Nov 04 03:09:55.431515 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:09:55.431156 4405 cpu_manager.go:414] "containerMap: removing container" podUID="121fbac1-908f-4c37-8efa-f17d449ed6b9" containerName="istio-proxy" Nov 04 03:09:55.431515 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:09:55.431160 4405 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="121fbac1-908f-4c37-8efa-f17d449ed6b9" containerName="istio-proxy" Nov 04 03:09:55.431515 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:09:55.431170 4405 cpu_manager.go:414] "containerMap: removing container" podUID="f55884d8-38e1-416b-853d-9c09082444bf" containerName="istio-proxy" Nov 04 03:09:55.431515 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:09:55.431174 4405 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f55884d8-38e1-416b-853d-9c09082444bf" containerName="istio-proxy" Nov 04 03:09:55.431515 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:09:55.431183 4405 cpu_manager.go:414] "containerMap: removing container" podUID="01be67f1-0362-4960-af5b-4ddad895ab4a" containerName="payload-processing" Nov 04 03:09:55.431515 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:09:55.431189 4405 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="01be67f1-0362-4960-af5b-4ddad895ab4a" containerName="payload-processing" Nov 04 03:09:55.431515 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:09:55.431272 4405 memory_manager.go:365] "RemoveStaleState removing state" podUID="121fbac1-908f-4c37-8efa-f17d449ed6b9" containerName="istio-proxy" Nov 04 03:09:55.431515 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:09:55.431282 4405 memory_manager.go:365] "RemoveStaleState removing state" podUID="f55884d8-38e1-416b-853d-9c09082444bf" containerName="istio-proxy" Nov 04 03:09:55.431515 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:09:55.431288 4405 memory_manager.go:365] "RemoveStaleState removing state" podUID="01be67f1-0362-4960-af5b-4ddad895ab4a" containerName="payload-processing" Nov 04 03:09:55.431515 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:09:55.431295 4405 memory_manager.go:365] "RemoveStaleState removing state" podUID="e21df403-3af7-4a07-8793-452782d57a3b" containerName="istio-proxy" Nov 04 03:09:55.431515 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:09:55.431300 4405 memory_manager.go:365] "RemoveStaleState removing state" podUID="595ef150-1fae-478f-8974-0b0efc667be8" containerName="payload-processing" Nov 04 03:09:55.431515 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:09:55.431306 4405 memory_manager.go:365] "RemoveStaleState removing state" podUID="541d536e-5ac3-4f98-9f6b-a827a5ef340a" containerName="maas-api" Nov 04 03:09:55.431515 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:09:55.431311 4405 memory_manager.go:365] "RemoveStaleState removing state" podUID="05f6d23b-ab70-4e62-9065-41f075c26ab6" containerName="istio-proxy" Nov 04 03:09:55.431515 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:09:55.431317 4405 memory_manager.go:365] "RemoveStaleState removing state" podUID="3918cd06-f979-4c62-a75f-916151257784" containerName="istio-proxy" Nov 04 03:09:55.431515 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:09:55.431322 4405 memory_manager.go:365] "RemoveStaleState removing state" podUID="d2227982-1e00-4ef1-9858-1abfe945bb84" containerName="maas-api" Nov 04 03:09:55.431515 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:09:55.431327 4405 memory_manager.go:365] "RemoveStaleState removing state" podUID="a6be9827-0b28-45b8-bcbe-242840761447" containerName="payload-pre-processing" Nov 04 03:09:55.431515 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:09:55.431335 4405 memory_manager.go:365] "RemoveStaleState removing state" podUID="ef8afda8-47fe-4cab-85a0-01397c9ab56b" containerName="maas-api" Nov 04 03:09:55.434859 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:09:55.434842 4405 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-db5ef5-5d5584b699-rjmpj" Nov 04 03:09:55.476055 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:09:55.476011 4405 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3d80d037-3e20-4898-855b-86fbcf2bb9aa-metrics-certs\") pod \"maas-api-e2e-inf-a-db5ef5-5d5584b699-rjmpj\" (UID: \"3d80d037-3e20-4898-855b-86fbcf2bb9aa\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-db5ef5-5d5584b699-rjmpj" Nov 04 03:09:55.476238 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:09:55.476159 4405 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3d80d037-3e20-4898-855b-86fbcf2bb9aa-openshift-service-ca\") pod \"maas-api-e2e-inf-a-db5ef5-5d5584b699-rjmpj\" (UID: \"3d80d037-3e20-4898-855b-86fbcf2bb9aa\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-db5ef5-5d5584b699-rjmpj" Nov 04 03:09:55.476238 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:09:55.476215 4405 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5zqv5\" (UniqueName: \"kubernetes.io/projected/3d80d037-3e20-4898-855b-86fbcf2bb9aa-kube-api-access-5zqv5\") pod \"maas-api-e2e-inf-a-db5ef5-5d5584b699-rjmpj\" (UID: \"3d80d037-3e20-4898-855b-86fbcf2bb9aa\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-db5ef5-5d5584b699-rjmpj" Nov 04 03:09:55.476424 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:09:55.476403 4405 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3d80d037-3e20-4898-855b-86fbcf2bb9aa-maas-api-tls\") pod \"maas-api-e2e-inf-a-db5ef5-5d5584b699-rjmpj\" (UID: \"3d80d037-3e20-4898-855b-86fbcf2bb9aa\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-db5ef5-5d5584b699-rjmpj" Nov 04 03:09:55.488758 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:09:55.488545 4405 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-a-db5ef5-serving-cert\"" Nov 04 03:09:55.492767 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:09:55.492744 4405 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-db5ef5-5d5584b699-rjmpj"] Nov 04 03:09:55.582035 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:09:55.581994 4405 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3d80d037-3e20-4898-855b-86fbcf2bb9aa-openshift-service-ca\") pod \"maas-api-e2e-inf-a-db5ef5-5d5584b699-rjmpj\" (UID: \"3d80d037-3e20-4898-855b-86fbcf2bb9aa\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-db5ef5-5d5584b699-rjmpj" Nov 04 03:09:55.582248 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:09:55.582069 4405 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5zqv5\" (UniqueName: \"kubernetes.io/projected/3d80d037-3e20-4898-855b-86fbcf2bb9aa-kube-api-access-5zqv5\") pod \"maas-api-e2e-inf-a-db5ef5-5d5584b699-rjmpj\" (UID: \"3d80d037-3e20-4898-855b-86fbcf2bb9aa\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-db5ef5-5d5584b699-rjmpj" Nov 04 03:09:55.582248 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:09:55.582170 4405 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3d80d037-3e20-4898-855b-86fbcf2bb9aa-maas-api-tls\") pod \"maas-api-e2e-inf-a-db5ef5-5d5584b699-rjmpj\" (UID: \"3d80d037-3e20-4898-855b-86fbcf2bb9aa\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-db5ef5-5d5584b699-rjmpj" Nov 04 03:09:55.582248 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:09:55.582239 4405 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3d80d037-3e20-4898-855b-86fbcf2bb9aa-metrics-certs\") pod \"maas-api-e2e-inf-a-db5ef5-5d5584b699-rjmpj\" (UID: \"3d80d037-3e20-4898-855b-86fbcf2bb9aa\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-db5ef5-5d5584b699-rjmpj" Nov 04 03:09:55.582525 ip-10-0-134-125 kubenswrapper[4405]: E1104 03:09:55.582501 4405 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-inf-a-db5ef5-serving-cert: secret "maas-api-e2e-inf-a-db5ef5-serving-cert" not found Nov 04 03:09:55.582616 ip-10-0-134-125 kubenswrapper[4405]: E1104 03:09:55.582577 4405 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3d80d037-3e20-4898-855b-86fbcf2bb9aa-maas-api-tls podName:3d80d037-3e20-4898-855b-86fbcf2bb9aa nodeName:}" failed. No retries permitted until 2026-11-04 03:09:56.08255543 +0000 UTC m=+2242.513255538 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/3d80d037-3e20-4898-855b-86fbcf2bb9aa-maas-api-tls") pod "maas-api-e2e-inf-a-db5ef5-5d5584b699-rjmpj" (UID: "3d80d037-3e20-4898-855b-86fbcf2bb9aa") : secret "maas-api-e2e-inf-a-db5ef5-serving-cert" not found Nov 04 03:09:55.582942 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:09:55.582924 4405 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3d80d037-3e20-4898-855b-86fbcf2bb9aa-openshift-service-ca\") pod \"maas-api-e2e-inf-a-db5ef5-5d5584b699-rjmpj\" (UID: \"3d80d037-3e20-4898-855b-86fbcf2bb9aa\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-db5ef5-5d5584b699-rjmpj" Nov 04 03:09:55.585222 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:09:55.585196 4405 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3d80d037-3e20-4898-855b-86fbcf2bb9aa-metrics-certs\") pod \"maas-api-e2e-inf-a-db5ef5-5d5584b699-rjmpj\" (UID: \"3d80d037-3e20-4898-855b-86fbcf2bb9aa\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-db5ef5-5d5584b699-rjmpj" Nov 04 03:09:55.644126 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:09:55.644094 4405 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5zqv5\" (UniqueName: \"kubernetes.io/projected/3d80d037-3e20-4898-855b-86fbcf2bb9aa-kube-api-access-5zqv5\") pod \"maas-api-e2e-inf-a-db5ef5-5d5584b699-rjmpj\" (UID: \"3d80d037-3e20-4898-855b-86fbcf2bb9aa\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-db5ef5-5d5584b699-rjmpj" Nov 04 03:09:55.702954 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:09:55.702868 4405 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d2227982-1e00-4ef1-9858-1abfe945bb84" path="/var/lib/kubelet/pods/d2227982-1e00-4ef1-9858-1abfe945bb84/volumes" Nov 04 03:09:55.964821 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:09:55.964734 4405 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-57f116-openshift-default-75f4fc499-znglh"] Nov 04 03:09:55.965302 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:09:55.964995 4405 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w2-57f116-openshift-default-75f4fc499-znglh" podUID="fd2b4daf-d65f-44ca-8939-67778fe8ef5a" containerName="istio-proxy" containerID="cri-o://8990f651f55bcfb5f4dd6208c0a5730f4a31e23bad1d255db7883f28f1975764" gracePeriod=30 Nov 04 03:09:56.131699 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:09:56.117001 4405 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3d80d037-3e20-4898-855b-86fbcf2bb9aa-maas-api-tls\") pod \"maas-api-e2e-inf-a-db5ef5-5d5584b699-rjmpj\" (UID: \"3d80d037-3e20-4898-855b-86fbcf2bb9aa\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-db5ef5-5d5584b699-rjmpj" Nov 04 03:09:56.153293 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:09:56.153009 4405 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3d80d037-3e20-4898-855b-86fbcf2bb9aa-maas-api-tls\") pod \"maas-api-e2e-inf-a-db5ef5-5d5584b699-rjmpj\" (UID: \"3d80d037-3e20-4898-855b-86fbcf2bb9aa\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-db5ef5-5d5584b699-rjmpj" Nov 04 03:09:56.360176 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:09:56.360143 4405 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-db5ef5-5d5584b699-rjmpj" Nov 04 03:09:56.749363 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:09:56.749338 4405 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-db5ef5-5d5584b699-rjmpj"] Nov 04 03:09:56.751770 ip-10-0-134-125 kubenswrapper[4405]: W1104 03:09:56.751730 4405 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3d80d037_3e20_4898_855b_86fbcf2bb9aa.slice/crio-45324060e1268053f6c60d52c17dc63e8035ae528abac61a297ca157efa255d9 WatchSource:0}: Error finding container 45324060e1268053f6c60d52c17dc63e8035ae528abac61a297ca157efa255d9: Status 404 returned error can't find the container with id 45324060e1268053f6c60d52c17dc63e8035ae528abac61a297ca157efa255d9 Nov 04 03:09:56.928912 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:09:56.928869 4405 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-db5ef5-5d5584b699-rjmpj" event={"ID":"3d80d037-3e20-4898-855b-86fbcf2bb9aa","Type":"ContainerStarted","Data":"45324060e1268053f6c60d52c17dc63e8035ae528abac61a297ca157efa255d9"} Nov 04 03:09:57.092787 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:09:57.092751 4405 patch_prober.go:29] interesting pod/e2e-shared-a-w2-57f116-openshift-default-75f4fc499-znglh container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 04 03:09:57.093206 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:09:57.092813 4405 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-w2-57f116-openshift-default-75f4fc499-znglh" podUID="fd2b4daf-d65f-44ca-8939-67778fe8ef5a" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 04 03:09:57.442401 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:09:57.442311 4405 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-76597d6855-vxzxx"] Nov 04 03:09:57.442989 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:09:57.442971 4405 cpu_manager.go:414] "containerMap: removing container" podUID="e21df403-3af7-4a07-8793-452782d57a3b" containerName="istio-proxy" Nov 04 03:09:57.443042 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:09:57.442992 4405 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e21df403-3af7-4a07-8793-452782d57a3b" containerName="istio-proxy" Nov 04 03:09:57.443116 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:09:57.443106 4405 memory_manager.go:365] "RemoveStaleState removing state" podUID="e21df403-3af7-4a07-8793-452782d57a3b" containerName="istio-proxy" Nov 04 03:09:57.448038 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:09:57.448018 4405 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w5-3b7476-openshift-default-59975457bf-q7gfm"] Nov 04 03:09:57.448154 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:09:57.448134 4405 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-vxzxx" Nov 04 03:09:57.452140 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:09:57.452123 4405 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-3b7476-openshift-default-59975457bf-q7gfm" Nov 04 03:09:57.537135 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:09:57.537102 4405 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e6cb6771-a9e6-452f-a0b6-30ffb955feb8-istiod-ca-cert\") pod \"e2e-worker-w5-3b7476-openshift-default-59975457bf-q7gfm\" (UID: \"e6cb6771-a9e6-452f-a0b6-30ffb955feb8\") " pod="openshift-ingress/e2e-worker-w5-3b7476-openshift-default-59975457bf-q7gfm" Nov 04 03:09:57.537336 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:09:57.537174 4405 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e6cb6771-a9e6-452f-a0b6-30ffb955feb8-credential-socket\") pod \"e2e-worker-w5-3b7476-openshift-default-59975457bf-q7gfm\" (UID: \"e6cb6771-a9e6-452f-a0b6-30ffb955feb8\") " pod="openshift-ingress/e2e-worker-w5-3b7476-openshift-default-59975457bf-q7gfm" Nov 04 03:09:57.537336 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:09:57.537263 4405 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/eeec2ff5-38c0-4e67-aeed-9f0baa9fa1d1-workload-certs\") pod \"maas-default-gateway-openshift-default-76597d6855-vxzxx\" (UID: \"eeec2ff5-38c0-4e67-aeed-9f0baa9fa1d1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-vxzxx" Nov 04 03:09:57.537336 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:09:57.537331 4405 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e6cb6771-a9e6-452f-a0b6-30ffb955feb8-istio-data\") pod \"e2e-worker-w5-3b7476-openshift-default-59975457bf-q7gfm\" (UID: \"e6cb6771-a9e6-452f-a0b6-30ffb955feb8\") " pod="openshift-ingress/e2e-worker-w5-3b7476-openshift-default-59975457bf-q7gfm" Nov 04 03:09:57.537483 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:09:57.537372 4405 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/eeec2ff5-38c0-4e67-aeed-9f0baa9fa1d1-istio-podinfo\") pod \"maas-default-gateway-openshift-default-76597d6855-vxzxx\" (UID: \"eeec2ff5-38c0-4e67-aeed-9f0baa9fa1d1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-vxzxx" Nov 04 03:09:57.537483 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:09:57.537466 4405 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e6cb6771-a9e6-452f-a0b6-30ffb955feb8-workload-certs\") pod \"e2e-worker-w5-3b7476-openshift-default-59975457bf-q7gfm\" (UID: \"e6cb6771-a9e6-452f-a0b6-30ffb955feb8\") " pod="openshift-ingress/e2e-worker-w5-3b7476-openshift-default-59975457bf-q7gfm" Nov 04 03:09:57.537572 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:09:57.537515 4405 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e6cb6771-a9e6-452f-a0b6-30ffb955feb8-istio-envoy\") pod \"e2e-worker-w5-3b7476-openshift-default-59975457bf-q7gfm\" (UID: \"e6cb6771-a9e6-452f-a0b6-30ffb955feb8\") " pod="openshift-ingress/e2e-worker-w5-3b7476-openshift-default-59975457bf-q7gfm" Nov 04 03:09:57.537630 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:09:57.537604 4405 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/eeec2ff5-38c0-4e67-aeed-9f0baa9fa1d1-istio-token\") pod \"maas-default-gateway-openshift-default-76597d6855-vxzxx\" (UID: \"eeec2ff5-38c0-4e67-aeed-9f0baa9fa1d1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-vxzxx" Nov 04 03:09:57.537714 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:09:57.537640 4405 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/eeec2ff5-38c0-4e67-aeed-9f0baa9fa1d1-istio-data\") pod \"maas-default-gateway-openshift-default-76597d6855-vxzxx\" (UID: \"eeec2ff5-38c0-4e67-aeed-9f0baa9fa1d1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-vxzxx" Nov 04 03:09:57.537753 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:09:57.537714 4405 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e6cb6771-a9e6-452f-a0b6-30ffb955feb8-istio-podinfo\") pod \"e2e-worker-w5-3b7476-openshift-default-59975457bf-q7gfm\" (UID: \"e6cb6771-a9e6-452f-a0b6-30ffb955feb8\") " pod="openshift-ingress/e2e-worker-w5-3b7476-openshift-default-59975457bf-q7gfm" Nov 04 03:09:57.537799 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:09:57.537751 4405 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gxct2\" (UniqueName: \"kubernetes.io/projected/e6cb6771-a9e6-452f-a0b6-30ffb955feb8-kube-api-access-gxct2\") pod \"e2e-worker-w5-3b7476-openshift-default-59975457bf-q7gfm\" (UID: \"e6cb6771-a9e6-452f-a0b6-30ffb955feb8\") " pod="openshift-ingress/e2e-worker-w5-3b7476-openshift-default-59975457bf-q7gfm" Nov 04 03:09:57.537799 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:09:57.537767 4405 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/eeec2ff5-38c0-4e67-aeed-9f0baa9fa1d1-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-76597d6855-vxzxx\" (UID: \"eeec2ff5-38c0-4e67-aeed-9f0baa9fa1d1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-vxzxx" Nov 04 03:09:57.537881 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:09:57.537868 4405 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/eeec2ff5-38c0-4e67-aeed-9f0baa9fa1d1-workload-socket\") pod \"maas-default-gateway-openshift-default-76597d6855-vxzxx\" (UID: \"eeec2ff5-38c0-4e67-aeed-9f0baa9fa1d1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-vxzxx" Nov 04 03:09:57.537913 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:09:57.537902 4405 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/eeec2ff5-38c0-4e67-aeed-9f0baa9fa1d1-credential-socket\") pod \"maas-default-gateway-openshift-default-76597d6855-vxzxx\" (UID: \"eeec2ff5-38c0-4e67-aeed-9f0baa9fa1d1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-vxzxx" Nov 04 03:09:57.537954 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:09:57.537927 4405 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/eeec2ff5-38c0-4e67-aeed-9f0baa9fa1d1-istio-envoy\") pod \"maas-default-gateway-openshift-default-76597d6855-vxzxx\" (UID: \"eeec2ff5-38c0-4e67-aeed-9f0baa9fa1d1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-vxzxx" Nov 04 03:09:57.538032 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:09:57.538016 4405 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e6cb6771-a9e6-452f-a0b6-30ffb955feb8-istio-token\") pod \"e2e-worker-w5-3b7476-openshift-default-59975457bf-q7gfm\" (UID: \"e6cb6771-a9e6-452f-a0b6-30ffb955feb8\") " pod="openshift-ingress/e2e-worker-w5-3b7476-openshift-default-59975457bf-q7gfm" Nov 04 03:09:57.538114 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:09:57.538098 4405 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cjnvx\" (UniqueName: \"kubernetes.io/projected/eeec2ff5-38c0-4e67-aeed-9f0baa9fa1d1-kube-api-access-cjnvx\") pod \"maas-default-gateway-openshift-default-76597d6855-vxzxx\" (UID: \"eeec2ff5-38c0-4e67-aeed-9f0baa9fa1d1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-vxzxx" Nov 04 03:09:57.538173 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:09:57.538152 4405 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e6cb6771-a9e6-452f-a0b6-30ffb955feb8-workload-socket\") pod \"e2e-worker-w5-3b7476-openshift-default-59975457bf-q7gfm\" (UID: \"e6cb6771-a9e6-452f-a0b6-30ffb955feb8\") " pod="openshift-ingress/e2e-worker-w5-3b7476-openshift-default-59975457bf-q7gfm" Nov 04 03:09:57.595875 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:09:57.595846 4405 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-3b7476-openshift-default-59975457bf-q7gfm"] Nov 04 03:09:57.639393 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:09:57.639361 4405 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cjnvx\" (UniqueName: \"kubernetes.io/projected/eeec2ff5-38c0-4e67-aeed-9f0baa9fa1d1-kube-api-access-cjnvx\") pod \"maas-default-gateway-openshift-default-76597d6855-vxzxx\" (UID: \"eeec2ff5-38c0-4e67-aeed-9f0baa9fa1d1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-vxzxx" Nov 04 03:09:57.639519 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:09:57.639409 4405 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e6cb6771-a9e6-452f-a0b6-30ffb955feb8-workload-socket\") pod \"e2e-worker-w5-3b7476-openshift-default-59975457bf-q7gfm\" (UID: \"e6cb6771-a9e6-452f-a0b6-30ffb955feb8\") " pod="openshift-ingress/e2e-worker-w5-3b7476-openshift-default-59975457bf-q7gfm" Nov 04 03:09:57.639519 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:09:57.639445 4405 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e6cb6771-a9e6-452f-a0b6-30ffb955feb8-istiod-ca-cert\") pod \"e2e-worker-w5-3b7476-openshift-default-59975457bf-q7gfm\" (UID: \"e6cb6771-a9e6-452f-a0b6-30ffb955feb8\") " pod="openshift-ingress/e2e-worker-w5-3b7476-openshift-default-59975457bf-q7gfm" Nov 04 03:09:57.639519 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:09:57.639493 4405 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e6cb6771-a9e6-452f-a0b6-30ffb955feb8-credential-socket\") pod \"e2e-worker-w5-3b7476-openshift-default-59975457bf-q7gfm\" (UID: \"e6cb6771-a9e6-452f-a0b6-30ffb955feb8\") " pod="openshift-ingress/e2e-worker-w5-3b7476-openshift-default-59975457bf-q7gfm" Nov 04 03:09:57.639696 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:09:57.639521 4405 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/eeec2ff5-38c0-4e67-aeed-9f0baa9fa1d1-workload-certs\") pod \"maas-default-gateway-openshift-default-76597d6855-vxzxx\" (UID: \"eeec2ff5-38c0-4e67-aeed-9f0baa9fa1d1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-vxzxx" Nov 04 03:09:57.639696 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:09:57.639575 4405 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e6cb6771-a9e6-452f-a0b6-30ffb955feb8-istio-data\") pod \"e2e-worker-w5-3b7476-openshift-default-59975457bf-q7gfm\" (UID: \"e6cb6771-a9e6-452f-a0b6-30ffb955feb8\") " pod="openshift-ingress/e2e-worker-w5-3b7476-openshift-default-59975457bf-q7gfm" Nov 04 03:09:57.639696 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:09:57.639609 4405 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/eeec2ff5-38c0-4e67-aeed-9f0baa9fa1d1-istio-podinfo\") pod \"maas-default-gateway-openshift-default-76597d6855-vxzxx\" (UID: \"eeec2ff5-38c0-4e67-aeed-9f0baa9fa1d1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-vxzxx" Nov 04 03:09:57.639863 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:09:57.639648 4405 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e6cb6771-a9e6-452f-a0b6-30ffb955feb8-workload-certs\") pod \"e2e-worker-w5-3b7476-openshift-default-59975457bf-q7gfm\" (UID: \"e6cb6771-a9e6-452f-a0b6-30ffb955feb8\") " pod="openshift-ingress/e2e-worker-w5-3b7476-openshift-default-59975457bf-q7gfm" Nov 04 03:09:57.639863 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:09:57.639760 4405 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e6cb6771-a9e6-452f-a0b6-30ffb955feb8-istio-envoy\") pod \"e2e-worker-w5-3b7476-openshift-default-59975457bf-q7gfm\" (UID: \"e6cb6771-a9e6-452f-a0b6-30ffb955feb8\") " pod="openshift-ingress/e2e-worker-w5-3b7476-openshift-default-59975457bf-q7gfm" Nov 04 03:09:57.639863 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:09:57.639804 4405 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/eeec2ff5-38c0-4e67-aeed-9f0baa9fa1d1-istio-token\") pod \"maas-default-gateway-openshift-default-76597d6855-vxzxx\" (UID: \"eeec2ff5-38c0-4e67-aeed-9f0baa9fa1d1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-vxzxx" Nov 04 03:09:57.639863 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:09:57.639831 4405 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/eeec2ff5-38c0-4e67-aeed-9f0baa9fa1d1-istio-data\") pod \"maas-default-gateway-openshift-default-76597d6855-vxzxx\" (UID: \"eeec2ff5-38c0-4e67-aeed-9f0baa9fa1d1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-vxzxx" Nov 04 03:09:57.640071 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:09:57.639864 4405 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e6cb6771-a9e6-452f-a0b6-30ffb955feb8-istio-podinfo\") pod \"e2e-worker-w5-3b7476-openshift-default-59975457bf-q7gfm\" (UID: \"e6cb6771-a9e6-452f-a0b6-30ffb955feb8\") " pod="openshift-ingress/e2e-worker-w5-3b7476-openshift-default-59975457bf-q7gfm" Nov 04 03:09:57.640071 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:09:57.639902 4405 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gxct2\" (UniqueName: \"kubernetes.io/projected/e6cb6771-a9e6-452f-a0b6-30ffb955feb8-kube-api-access-gxct2\") pod \"e2e-worker-w5-3b7476-openshift-default-59975457bf-q7gfm\" (UID: \"e6cb6771-a9e6-452f-a0b6-30ffb955feb8\") " pod="openshift-ingress/e2e-worker-w5-3b7476-openshift-default-59975457bf-q7gfm" Nov 04 03:09:57.640071 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:09:57.639926 4405 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/eeec2ff5-38c0-4e67-aeed-9f0baa9fa1d1-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-76597d6855-vxzxx\" (UID: \"eeec2ff5-38c0-4e67-aeed-9f0baa9fa1d1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-vxzxx" Nov 04 03:09:57.640071 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:09:57.639966 4405 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e6cb6771-a9e6-452f-a0b6-30ffb955feb8-credential-socket\") pod \"e2e-worker-w5-3b7476-openshift-default-59975457bf-q7gfm\" (UID: \"e6cb6771-a9e6-452f-a0b6-30ffb955feb8\") " pod="openshift-ingress/e2e-worker-w5-3b7476-openshift-default-59975457bf-q7gfm" Nov 04 03:09:57.640071 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:09:57.639991 4405 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/eeec2ff5-38c0-4e67-aeed-9f0baa9fa1d1-workload-socket\") pod \"maas-default-gateway-openshift-default-76597d6855-vxzxx\" (UID: \"eeec2ff5-38c0-4e67-aeed-9f0baa9fa1d1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-vxzxx" Nov 04 03:09:57.640071 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:09:57.640026 4405 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/eeec2ff5-38c0-4e67-aeed-9f0baa9fa1d1-credential-socket\") pod \"maas-default-gateway-openshift-default-76597d6855-vxzxx\" (UID: \"eeec2ff5-38c0-4e67-aeed-9f0baa9fa1d1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-vxzxx" Nov 04 03:09:57.640071 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:09:57.640062 4405 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/eeec2ff5-38c0-4e67-aeed-9f0baa9fa1d1-istio-envoy\") pod \"maas-default-gateway-openshift-default-76597d6855-vxzxx\" (UID: \"eeec2ff5-38c0-4e67-aeed-9f0baa9fa1d1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-vxzxx" Nov 04 03:09:57.640437 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:09:57.640101 4405 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e6cb6771-a9e6-452f-a0b6-30ffb955feb8-istio-token\") pod \"e2e-worker-w5-3b7476-openshift-default-59975457bf-q7gfm\" (UID: \"e6cb6771-a9e6-452f-a0b6-30ffb955feb8\") " pod="openshift-ingress/e2e-worker-w5-3b7476-openshift-default-59975457bf-q7gfm" Nov 04 03:09:57.640437 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:09:57.640116 4405 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e6cb6771-a9e6-452f-a0b6-30ffb955feb8-istio-data\") pod \"e2e-worker-w5-3b7476-openshift-default-59975457bf-q7gfm\" (UID: \"e6cb6771-a9e6-452f-a0b6-30ffb955feb8\") " pod="openshift-ingress/e2e-worker-w5-3b7476-openshift-default-59975457bf-q7gfm" Nov 04 03:09:57.640437 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:09:57.639986 4405 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/eeec2ff5-38c0-4e67-aeed-9f0baa9fa1d1-workload-certs\") pod \"maas-default-gateway-openshift-default-76597d6855-vxzxx\" (UID: \"eeec2ff5-38c0-4e67-aeed-9f0baa9fa1d1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-vxzxx" Nov 04 03:09:57.640437 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:09:57.640203 4405 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/eeec2ff5-38c0-4e67-aeed-9f0baa9fa1d1-istio-data\") pod \"maas-default-gateway-openshift-default-76597d6855-vxzxx\" (UID: \"eeec2ff5-38c0-4e67-aeed-9f0baa9fa1d1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-vxzxx" Nov 04 03:09:57.640768 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:09:57.640740 4405 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/eeec2ff5-38c0-4e67-aeed-9f0baa9fa1d1-workload-socket\") pod \"maas-default-gateway-openshift-default-76597d6855-vxzxx\" (UID: \"eeec2ff5-38c0-4e67-aeed-9f0baa9fa1d1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-vxzxx" Nov 04 03:09:57.640891 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:09:57.640868 4405 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/eeec2ff5-38c0-4e67-aeed-9f0baa9fa1d1-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-76597d6855-vxzxx\" (UID: \"eeec2ff5-38c0-4e67-aeed-9f0baa9fa1d1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-vxzxx" Nov 04 03:09:57.640993 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:09:57.640913 4405 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e6cb6771-a9e6-452f-a0b6-30ffb955feb8-workload-certs\") pod \"e2e-worker-w5-3b7476-openshift-default-59975457bf-q7gfm\" (UID: \"e6cb6771-a9e6-452f-a0b6-30ffb955feb8\") " pod="openshift-ingress/e2e-worker-w5-3b7476-openshift-default-59975457bf-q7gfm" Nov 04 03:09:57.641071 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:09:57.641015 4405 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/eeec2ff5-38c0-4e67-aeed-9f0baa9fa1d1-credential-socket\") pod \"maas-default-gateway-openshift-default-76597d6855-vxzxx\" (UID: \"eeec2ff5-38c0-4e67-aeed-9f0baa9fa1d1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-vxzxx" Nov 04 03:09:57.641121 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:09:57.641088 4405 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e6cb6771-a9e6-452f-a0b6-30ffb955feb8-workload-socket\") pod \"e2e-worker-w5-3b7476-openshift-default-59975457bf-q7gfm\" (UID: \"e6cb6771-a9e6-452f-a0b6-30ffb955feb8\") " pod="openshift-ingress/e2e-worker-w5-3b7476-openshift-default-59975457bf-q7gfm" Nov 04 03:09:57.641696 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:09:57.641643 4405 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e6cb6771-a9e6-452f-a0b6-30ffb955feb8-istiod-ca-cert\") pod \"e2e-worker-w5-3b7476-openshift-default-59975457bf-q7gfm\" (UID: \"e6cb6771-a9e6-452f-a0b6-30ffb955feb8\") " pod="openshift-ingress/e2e-worker-w5-3b7476-openshift-default-59975457bf-q7gfm" Nov 04 03:09:57.643143 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:09:57.643115 4405 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e6cb6771-a9e6-452f-a0b6-30ffb955feb8-istio-envoy\") pod \"e2e-worker-w5-3b7476-openshift-default-59975457bf-q7gfm\" (UID: \"e6cb6771-a9e6-452f-a0b6-30ffb955feb8\") " pod="openshift-ingress/e2e-worker-w5-3b7476-openshift-default-59975457bf-q7gfm" Nov 04 03:09:57.643483 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:09:57.643466 4405 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/eeec2ff5-38c0-4e67-aeed-9f0baa9fa1d1-istio-podinfo\") pod \"maas-default-gateway-openshift-default-76597d6855-vxzxx\" (UID: \"eeec2ff5-38c0-4e67-aeed-9f0baa9fa1d1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-vxzxx" Nov 04 03:09:57.643877 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:09:57.643854 4405 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/eeec2ff5-38c0-4e67-aeed-9f0baa9fa1d1-istio-envoy\") pod \"maas-default-gateway-openshift-default-76597d6855-vxzxx\" (UID: \"eeec2ff5-38c0-4e67-aeed-9f0baa9fa1d1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-vxzxx" Nov 04 03:09:57.644058 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:09:57.644041 4405 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e6cb6771-a9e6-452f-a0b6-30ffb955feb8-istio-podinfo\") pod \"e2e-worker-w5-3b7476-openshift-default-59975457bf-q7gfm\" (UID: \"e6cb6771-a9e6-452f-a0b6-30ffb955feb8\") " pod="openshift-ingress/e2e-worker-w5-3b7476-openshift-default-59975457bf-q7gfm" Nov 04 03:09:57.666250 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:09:57.666221 4405 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-76597d6855-vxzxx"] Nov 04 03:09:57.711617 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:09:57.711586 4405 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cjnvx\" (UniqueName: \"kubernetes.io/projected/eeec2ff5-38c0-4e67-aeed-9f0baa9fa1d1-kube-api-access-cjnvx\") pod \"maas-default-gateway-openshift-default-76597d6855-vxzxx\" (UID: \"eeec2ff5-38c0-4e67-aeed-9f0baa9fa1d1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-vxzxx" Nov 04 03:09:57.721198 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:09:57.721169 4405 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/eeec2ff5-38c0-4e67-aeed-9f0baa9fa1d1-istio-token\") pod \"maas-default-gateway-openshift-default-76597d6855-vxzxx\" (UID: \"eeec2ff5-38c0-4e67-aeed-9f0baa9fa1d1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-vxzxx" Nov 04 03:09:57.721332 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:09:57.721169 4405 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e6cb6771-a9e6-452f-a0b6-30ffb955feb8-istio-token\") pod \"e2e-worker-w5-3b7476-openshift-default-59975457bf-q7gfm\" (UID: \"e6cb6771-a9e6-452f-a0b6-30ffb955feb8\") " pod="openshift-ingress/e2e-worker-w5-3b7476-openshift-default-59975457bf-q7gfm" Nov 04 03:09:57.724443 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:09:57.724050 4405 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gxct2\" (UniqueName: \"kubernetes.io/projected/e6cb6771-a9e6-452f-a0b6-30ffb955feb8-kube-api-access-gxct2\") pod \"e2e-worker-w5-3b7476-openshift-default-59975457bf-q7gfm\" (UID: \"e6cb6771-a9e6-452f-a0b6-30ffb955feb8\") " pod="openshift-ingress/e2e-worker-w5-3b7476-openshift-default-59975457bf-q7gfm" Nov 04 03:09:57.764401 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:09:57.764371 4405 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-vxzxx" Nov 04 03:09:57.773209 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:09:57.773181 4405 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-3b7476-openshift-default-59975457bf-q7gfm" Nov 04 03:09:57.950229 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:09:57.950137 4405 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-db5ef5-5d5584b699-rjmpj" event={"ID":"3d80d037-3e20-4898-855b-86fbcf2bb9aa","Type":"ContainerStarted","Data":"e079012dbf9f0a256f342f2823ed63641442c2d25d24122f14a1ce243691052f"} Nov 04 03:09:57.950409 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:09:57.950257 4405 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-db5ef5-5d5584b699-rjmpj" Nov 04 03:09:58.070059 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:09:58.070008 4405 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-db5ef5-5d5584b699-rjmpj" podStartSLOduration=2.211645484 podStartE2EDuration="3.069992928s" totalImagesPullingTime="858.347444ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 03:09:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 03:09:58.066926579 +0000 UTC m=+2244.497626718" watchObservedRunningTime="2026-11-04 03:09:58.069992928 +0000 UTC m=+2244.500693042" Nov 04 03:09:58.188479 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:09:58.187964 4405 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-76597d6855-vxzxx"] Nov 04 03:09:58.197013 ip-10-0-134-125 kubenswrapper[4405]: W1104 03:09:58.196968 4405 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeeec2ff5_38c0_4e67_aeed_9f0baa9fa1d1.slice/crio-8075b1c999bf29c05d87e2d09c8ec401fc9bc6518cc5a89a82f2cc9bd5d31eed WatchSource:0}: Error finding container 8075b1c999bf29c05d87e2d09c8ec401fc9bc6518cc5a89a82f2cc9bd5d31eed: Status 404 returned error can't find the container with id 8075b1c999bf29c05d87e2d09c8ec401fc9bc6518cc5a89a82f2cc9bd5d31eed Nov 04 03:09:58.201162 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:09:58.201069 4405 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 04 03:09:58.201286 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:09:58.201215 4405 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 04 03:09:58.201286 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:09:58.201265 4405 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 04 03:09:58.229797 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:09:58.216768 4405 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-3b7476-openshift-default-59975457bf-q7gfm"] Nov 04 03:09:58.229797 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:09:58.228729 4405 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 04 03:09:58.229797 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:09:58.228826 4405 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 04 03:09:58.229797 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:09:58.228870 4405 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 04 03:09:58.959681 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:09:58.959632 4405 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-3b7476-openshift-default-59975457bf-q7gfm" event={"ID":"e6cb6771-a9e6-452f-a0b6-30ffb955feb8","Type":"ContainerStarted","Data":"4b1aee4d81d9c94a20df3fd373faab5077813f49eb41d3fc655645a2cc12c416"} Nov 04 03:09:58.959896 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:09:58.959707 4405 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-3b7476-openshift-default-59975457bf-q7gfm" event={"ID":"e6cb6771-a9e6-452f-a0b6-30ffb955feb8","Type":"ContainerStarted","Data":"ee2ba2f6c28f1cde4528f0e550ae387746e5fd964bd85dc6660bb7d34000c1cf"} Nov 04 03:09:58.961804 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:09:58.961756 4405 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-vxzxx" event={"ID":"eeec2ff5-38c0-4e67-aeed-9f0baa9fa1d1","Type":"ContainerStarted","Data":"1bdc0173c3101f7a1fe376035b7e0c6f0f0caceddaf8617db2c2b539c4019cfc"} Nov 04 03:09:58.961804 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:09:58.961796 4405 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-vxzxx" event={"ID":"eeec2ff5-38c0-4e67-aeed-9f0baa9fa1d1","Type":"ContainerStarted","Data":"8075b1c999bf29c05d87e2d09c8ec401fc9bc6518cc5a89a82f2cc9bd5d31eed"} Nov 04 03:09:59.090116 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:09:59.090053 4405 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w5-3b7476-openshift-default-59975457bf-q7gfm" podStartSLOduration=2.090032193 podStartE2EDuration="2.090032193s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 03:09:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 03:09:59.088469098 +0000 UTC m=+2245.519169218" watchObservedRunningTime="2026-11-04 03:09:59.090032193 +0000 UTC m=+2245.520732313" Nov 04 03:09:59.756369 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:09:59.756301 4405 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-vxzxx" podStartSLOduration=2.7562832999999998 podStartE2EDuration="2.7562833s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 03:09:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 03:09:59.223041353 +0000 UTC m=+2245.653741475" watchObservedRunningTime="2026-11-04 03:09:59.7562833 +0000 UTC m=+2246.186983421" Nov 04 03:09:59.760203 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:09:59.760176 4405 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-9d2b96-7d8dd46f5b-7rbgg"] Nov 04 03:09:59.765209 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:09:59.765183 4405 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-vxzxx" Nov 04 03:09:59.766725 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:09:59.766697 4405 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76597d6855-vxzxx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.125:15021/healthz/ready\": dial tcp 10.134.0.125:15021: connect: connection refused" start-of-body= Nov 04 03:09:59.766922 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:09:59.766896 4405 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-vxzxx" podUID="eeec2ff5-38c0-4e67-aeed-9f0baa9fa1d1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.125:15021/healthz/ready\": dial tcp 10.134.0.125:15021: connect: connection refused" Nov 04 03:09:59.774054 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:09:59.774026 4405 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w5-3b7476-openshift-default-59975457bf-q7gfm" Nov 04 03:09:59.775656 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:09:59.775630 4405 patch_prober.go:29] interesting pod/e2e-worker-w5-3b7476-openshift-default-59975457bf-q7gfm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.126:15021/healthz/ready\": dial tcp 10.134.0.126:15021: connect: connection refused" start-of-body= Nov 04 03:09:59.775774 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:09:59.775718 4405 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-3b7476-openshift-default-59975457bf-q7gfm" podUID="e6cb6771-a9e6-452f-a0b6-30ffb955feb8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.126:15021/healthz/ready\": dial tcp 10.134.0.126:15021: connect: connection refused" Nov 04 03:10:00.765248 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:10:00.765203 4405 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76597d6855-vxzxx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.125:15021/healthz/ready\": dial tcp 10.134.0.125:15021: connect: connection refused" start-of-body= Nov 04 03:10:00.765694 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:10:00.765273 4405 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-vxzxx" podUID="eeec2ff5-38c0-4e67-aeed-9f0baa9fa1d1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.125:15021/healthz/ready\": dial tcp 10.134.0.125:15021: connect: connection refused" Nov 04 03:10:00.773567 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:10:00.773534 4405 patch_prober.go:29] interesting pod/e2e-worker-w5-3b7476-openshift-default-59975457bf-q7gfm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.126:15021/healthz/ready\": dial tcp 10.134.0.126:15021: connect: connection refused" start-of-body= Nov 04 03:10:00.773655 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:10:00.773590 4405 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-3b7476-openshift-default-59975457bf-q7gfm" podUID="e6cb6771-a9e6-452f-a0b6-30ffb955feb8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.126:15021/healthz/ready\": dial tcp 10.134.0.126:15021: connect: connection refused" Nov 04 03:10:01.762719 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:10:01.762301 4405 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-57f116-openshift-default-75f4fc499-znglh" Nov 04 03:10:01.766908 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:10:01.766599 4405 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76597d6855-vxzxx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.125:15021/healthz/ready\": dial tcp 10.134.0.125:15021: connect: connection refused" start-of-body= Nov 04 03:10:01.766908 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:10:01.766647 4405 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-vxzxx" podUID="eeec2ff5-38c0-4e67-aeed-9f0baa9fa1d1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.125:15021/healthz/ready\": dial tcp 10.134.0.125:15021: connect: connection refused" Nov 04 03:10:01.773982 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:10:01.773953 4405 patch_prober.go:29] interesting pod/e2e-worker-w5-3b7476-openshift-default-59975457bf-q7gfm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.126:15021/healthz/ready\": dial tcp 10.134.0.126:15021: connect: connection refused" start-of-body= Nov 04 03:10:01.774200 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:10:01.774161 4405 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-3b7476-openshift-default-59975457bf-q7gfm" podUID="e6cb6771-a9e6-452f-a0b6-30ffb955feb8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.126:15021/healthz/ready\": dial tcp 10.134.0.126:15021: connect: connection refused" Nov 04 03:10:01.854777 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:10:01.854739 4405 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fd2b4daf-d65f-44ca-8939-67778fe8ef5a-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fd2b4daf-d65f-44ca-8939-67778fe8ef5a-istiod-ca-cert\") pod \"fd2b4daf-d65f-44ca-8939-67778fe8ef5a\" (UID: \"fd2b4daf-d65f-44ca-8939-67778fe8ef5a\") " Nov 04 03:10:01.854984 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:10:01.854806 4405 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fd2b4daf-d65f-44ca-8939-67778fe8ef5a-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fd2b4daf-d65f-44ca-8939-67778fe8ef5a-credential-socket\") pod \"fd2b4daf-d65f-44ca-8939-67778fe8ef5a\" (UID: \"fd2b4daf-d65f-44ca-8939-67778fe8ef5a\") " Nov 04 03:10:01.854984 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:10:01.854838 4405 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fd2b4daf-d65f-44ca-8939-67778fe8ef5a-kube-api-access-zrt6m\" (UniqueName: \"kubernetes.io/projected/fd2b4daf-d65f-44ca-8939-67778fe8ef5a-kube-api-access-zrt6m\") pod \"fd2b4daf-d65f-44ca-8939-67778fe8ef5a\" (UID: \"fd2b4daf-d65f-44ca-8939-67778fe8ef5a\") " Nov 04 03:10:01.854984 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:10:01.854910 4405 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fd2b4daf-d65f-44ca-8939-67778fe8ef5a-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fd2b4daf-d65f-44ca-8939-67778fe8ef5a-workload-certs\") pod \"fd2b4daf-d65f-44ca-8939-67778fe8ef5a\" (UID: \"fd2b4daf-d65f-44ca-8939-67778fe8ef5a\") " Nov 04 03:10:01.855183 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:10:01.855031 4405 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fd2b4daf-d65f-44ca-8939-67778fe8ef5a-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fd2b4daf-d65f-44ca-8939-67778fe8ef5a-istio-envoy\") pod \"fd2b4daf-d65f-44ca-8939-67778fe8ef5a\" (UID: \"fd2b4daf-d65f-44ca-8939-67778fe8ef5a\") " Nov 04 03:10:01.855183 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:10:01.855071 4405 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fd2b4daf-d65f-44ca-8939-67778fe8ef5a-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fd2b4daf-d65f-44ca-8939-67778fe8ef5a-istio-data\") pod \"fd2b4daf-d65f-44ca-8939-67778fe8ef5a\" (UID: \"fd2b4daf-d65f-44ca-8939-67778fe8ef5a\") " Nov 04 03:10:01.855183 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:10:01.855142 4405 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/fd2b4daf-d65f-44ca-8939-67778fe8ef5a-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fd2b4daf-d65f-44ca-8939-67778fe8ef5a-istio-podinfo\") pod \"fd2b4daf-d65f-44ca-8939-67778fe8ef5a\" (UID: \"fd2b4daf-d65f-44ca-8939-67778fe8ef5a\") " Nov 04 03:10:01.855183 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:10:01.855176 4405 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fd2b4daf-d65f-44ca-8939-67778fe8ef5a-istio-token\" (UniqueName: \"kubernetes.io/projected/fd2b4daf-d65f-44ca-8939-67778fe8ef5a-istio-token\") pod \"fd2b4daf-d65f-44ca-8939-67778fe8ef5a\" (UID: \"fd2b4daf-d65f-44ca-8939-67778fe8ef5a\") " Nov 04 03:10:01.855392 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:10:01.855224 4405 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fd2b4daf-d65f-44ca-8939-67778fe8ef5a-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fd2b4daf-d65f-44ca-8939-67778fe8ef5a-workload-socket\") pod \"fd2b4daf-d65f-44ca-8939-67778fe8ef5a\" (UID: \"fd2b4daf-d65f-44ca-8939-67778fe8ef5a\") " Nov 04 03:10:01.855659 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:10:01.855628 4405 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fd2b4daf-d65f-44ca-8939-67778fe8ef5a-istiod-ca-cert" pod "fd2b4daf-d65f-44ca-8939-67778fe8ef5a" (UID: "fd2b4daf-d65f-44ca-8939-67778fe8ef5a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 03:10:01.855934 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:10:01.855902 4405 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fd2b4daf-d65f-44ca-8939-67778fe8ef5a-istio-data" pod "fd2b4daf-d65f-44ca-8939-67778fe8ef5a" (UID: "fd2b4daf-d65f-44ca-8939-67778fe8ef5a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 03:10:01.855934 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:10:01.855918 4405 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fd2b4daf-d65f-44ca-8939-67778fe8ef5a-workload-certs" pod "fd2b4daf-d65f-44ca-8939-67778fe8ef5a" (UID: "fd2b4daf-d65f-44ca-8939-67778fe8ef5a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 03:10:01.856108 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:10:01.856028 4405 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fd2b4daf-d65f-44ca-8939-67778fe8ef5a-workload-socket" pod "fd2b4daf-d65f-44ca-8939-67778fe8ef5a" (UID: "fd2b4daf-d65f-44ca-8939-67778fe8ef5a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 03:10:01.856711 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:10:01.856599 4405 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fd2b4daf-d65f-44ca-8939-67778fe8ef5a-credential-socket" pod "fd2b4daf-d65f-44ca-8939-67778fe8ef5a" (UID: "fd2b4daf-d65f-44ca-8939-67778fe8ef5a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 03:10:01.857946 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:10:01.857919 4405 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fd2b4daf-d65f-44ca-8939-67778fe8ef5a-kube-api-access-zrt6m" pod "fd2b4daf-d65f-44ca-8939-67778fe8ef5a" (UID: "fd2b4daf-d65f-44ca-8939-67778fe8ef5a"). InnerVolumeSpecName "kube-api-access-zrt6m". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 03:10:01.858226 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:10:01.858209 4405 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fd2b4daf-d65f-44ca-8939-67778fe8ef5a-istio-envoy" pod "fd2b4daf-d65f-44ca-8939-67778fe8ef5a" (UID: "fd2b4daf-d65f-44ca-8939-67778fe8ef5a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 03:10:01.858496 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:10:01.858469 4405 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/fd2b4daf-d65f-44ca-8939-67778fe8ef5a-istio-podinfo" pod "fd2b4daf-d65f-44ca-8939-67778fe8ef5a" (UID: "fd2b4daf-d65f-44ca-8939-67778fe8ef5a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 04 03:10:01.858730 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:10:01.858710 4405 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fd2b4daf-d65f-44ca-8939-67778fe8ef5a-istio-token" pod "fd2b4daf-d65f-44ca-8939-67778fe8ef5a" (UID: "fd2b4daf-d65f-44ca-8939-67778fe8ef5a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 03:10:01.957198 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:10:01.957164 4405 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fd2b4daf-d65f-44ca-8939-67778fe8ef5a-workload-certs\") on node \"ip-10-0-134-125.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:10:01.957198 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:10:01.957196 4405 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fd2b4daf-d65f-44ca-8939-67778fe8ef5a-istio-envoy\") on node \"ip-10-0-134-125.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:10:01.957198 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:10:01.957205 4405 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fd2b4daf-d65f-44ca-8939-67778fe8ef5a-istio-data\") on node \"ip-10-0-134-125.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:10:01.957456 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:10:01.957215 4405 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fd2b4daf-d65f-44ca-8939-67778fe8ef5a-istio-podinfo\") on node \"ip-10-0-134-125.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:10:01.957456 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:10:01.957223 4405 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fd2b4daf-d65f-44ca-8939-67778fe8ef5a-istio-token\") on node \"ip-10-0-134-125.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:10:01.957456 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:10:01.957232 4405 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fd2b4daf-d65f-44ca-8939-67778fe8ef5a-workload-socket\") on node \"ip-10-0-134-125.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:10:01.957456 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:10:01.957240 4405 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fd2b4daf-d65f-44ca-8939-67778fe8ef5a-istiod-ca-cert\") on node \"ip-10-0-134-125.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:10:01.957456 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:10:01.957249 4405 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fd2b4daf-d65f-44ca-8939-67778fe8ef5a-credential-socket\") on node \"ip-10-0-134-125.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:10:01.957456 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:10:01.957258 4405 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zrt6m\" (UniqueName: \"kubernetes.io/projected/fd2b4daf-d65f-44ca-8939-67778fe8ef5a-kube-api-access-zrt6m\") on node \"ip-10-0-134-125.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:10:02.043143 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:10:02.043094 4405 generic.go:356] "Generic (PLEG): container finished" podID="fd2b4daf-d65f-44ca-8939-67778fe8ef5a" containerID="8990f651f55bcfb5f4dd6208c0a5730f4a31e23bad1d255db7883f28f1975764" exitCode=0 Nov 04 03:10:02.043758 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:10:02.043724 4405 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-57f116-openshift-default-75f4fc499-znglh" event={"ID":"fd2b4daf-d65f-44ca-8939-67778fe8ef5a","Type":"ContainerDied","Data":"8990f651f55bcfb5f4dd6208c0a5730f4a31e23bad1d255db7883f28f1975764"} Nov 04 03:10:02.044032 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:10:02.043990 4405 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-57f116-openshift-default-75f4fc499-znglh" event={"ID":"fd2b4daf-d65f-44ca-8939-67778fe8ef5a","Type":"ContainerDied","Data":"db1b50b8c67c645f7848e2e9bc5f8ed67e7a4841e0f82ea9ef1698e9a03c89ab"} Nov 04 03:10:02.044162 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:10:02.044148 4405 scope.go:122] "RemoveContainer" containerID="8990f651f55bcfb5f4dd6208c0a5730f4a31e23bad1d255db7883f28f1975764" Nov 04 03:10:02.044532 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:10:02.044513 4405 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-57f116-openshift-default-75f4fc499-znglh" Nov 04 03:10:02.078647 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:10:02.078603 4405 scope.go:122] "RemoveContainer" containerID="7192f537f98158d936b5e94e6c021bd5b8a8ba642b9a06dea35cddcc897bf35f" Nov 04 03:10:02.135755 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:10:02.129962 4405 scope.go:122] "RemoveContainer" containerID="8990f651f55bcfb5f4dd6208c0a5730f4a31e23bad1d255db7883f28f1975764" Nov 04 03:10:02.135755 ip-10-0-134-125 kubenswrapper[4405]: E1104 03:10:02.130754 4405 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8990f651f55bcfb5f4dd6208c0a5730f4a31e23bad1d255db7883f28f1975764\": container with ID starting with 8990f651f55bcfb5f4dd6208c0a5730f4a31e23bad1d255db7883f28f1975764 not found: ID does not exist" containerID="8990f651f55bcfb5f4dd6208c0a5730f4a31e23bad1d255db7883f28f1975764" Nov 04 03:10:02.135755 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:10:02.130795 4405 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8990f651f55bcfb5f4dd6208c0a5730f4a31e23bad1d255db7883f28f1975764"} err="failed to get container status \"8990f651f55bcfb5f4dd6208c0a5730f4a31e23bad1d255db7883f28f1975764\": rpc error: code = NotFound desc = could not find container \"8990f651f55bcfb5f4dd6208c0a5730f4a31e23bad1d255db7883f28f1975764\": container with ID starting with 8990f651f55bcfb5f4dd6208c0a5730f4a31e23bad1d255db7883f28f1975764 not found: ID does not exist" Nov 04 03:10:02.135755 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:10:02.130823 4405 scope.go:122] "RemoveContainer" containerID="7192f537f98158d936b5e94e6c021bd5b8a8ba642b9a06dea35cddcc897bf35f" Nov 04 03:10:02.135755 ip-10-0-134-125 kubenswrapper[4405]: E1104 03:10:02.131137 4405 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7192f537f98158d936b5e94e6c021bd5b8a8ba642b9a06dea35cddcc897bf35f\": container with ID starting with 7192f537f98158d936b5e94e6c021bd5b8a8ba642b9a06dea35cddcc897bf35f not found: ID does not exist" containerID="7192f537f98158d936b5e94e6c021bd5b8a8ba642b9a06dea35cddcc897bf35f" Nov 04 03:10:02.135755 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:10:02.131163 4405 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7192f537f98158d936b5e94e6c021bd5b8a8ba642b9a06dea35cddcc897bf35f"} err="failed to get container status \"7192f537f98158d936b5e94e6c021bd5b8a8ba642b9a06dea35cddcc897bf35f\": rpc error: code = NotFound desc = could not find container \"7192f537f98158d936b5e94e6c021bd5b8a8ba642b9a06dea35cddcc897bf35f\": container with ID starting with 7192f537f98158d936b5e94e6c021bd5b8a8ba642b9a06dea35cddcc897bf35f not found: ID does not exist" Nov 04 03:10:02.253255 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:10:02.251333 4405 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-57f116-openshift-default-75f4fc499-znglh"] Nov 04 03:10:02.256261 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:10:02.256220 4405 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-57f116-openshift-default-75f4fc499-znglh"] Nov 04 03:10:02.764848 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:10:02.764817 4405 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76597d6855-vxzxx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.125:15021/healthz/ready\": dial tcp 10.134.0.125:15021: connect: connection refused" start-of-body= Nov 04 03:10:02.765040 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:10:02.764880 4405 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-vxzxx" podUID="eeec2ff5-38c0-4e67-aeed-9f0baa9fa1d1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.125:15021/healthz/ready\": dial tcp 10.134.0.125:15021: connect: connection refused" Nov 04 03:10:02.774407 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:10:02.774372 4405 patch_prober.go:29] interesting pod/e2e-worker-w5-3b7476-openshift-default-59975457bf-q7gfm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.126:15021/healthz/ready\": dial tcp 10.134.0.126:15021: connect: connection refused" start-of-body= Nov 04 03:10:02.774807 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:10:02.774437 4405 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-3b7476-openshift-default-59975457bf-q7gfm" podUID="e6cb6771-a9e6-452f-a0b6-30ffb955feb8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.126:15021/healthz/ready\": dial tcp 10.134.0.126:15021: connect: connection refused" Nov 04 03:10:03.757472 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:10:03.757436 4405 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fd2b4daf-d65f-44ca-8939-67778fe8ef5a" path="/var/lib/kubelet/pods/fd2b4daf-d65f-44ca-8939-67778fe8ef5a/volumes" Nov 04 03:10:03.765200 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:10:03.765171 4405 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76597d6855-vxzxx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.125:15021/healthz/ready\": dial tcp 10.134.0.125:15021: connect: connection refused" start-of-body= Nov 04 03:10:03.765311 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:10:03.765241 4405 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-vxzxx" podUID="eeec2ff5-38c0-4e67-aeed-9f0baa9fa1d1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.125:15021/healthz/ready\": dial tcp 10.134.0.125:15021: connect: connection refused" Nov 04 03:10:03.773781 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:10:03.773753 4405 patch_prober.go:29] interesting pod/e2e-worker-w5-3b7476-openshift-default-59975457bf-q7gfm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.126:15021/healthz/ready\": dial tcp 10.134.0.126:15021: connect: connection refused" start-of-body= Nov 04 03:10:03.773924 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:10:03.773802 4405 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-3b7476-openshift-default-59975457bf-q7gfm" podUID="e6cb6771-a9e6-452f-a0b6-30ffb955feb8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.126:15021/healthz/ready\": dial tcp 10.134.0.126:15021: connect: connection refused" Nov 04 03:10:03.968464 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:10:03.968117 4405 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-db5ef5-5d5584b699-rjmpj" Nov 04 03:10:04.728401 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:10:04.727969 4405 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-00c846-6865789c84-c24hp"] Nov 04 03:10:04.760659 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:10:04.760611 4405 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-b-9d2b96-7d8dd46f5b-7rbgg" podUID="24a56d15-dba0-46bc-aca3-f6fabdf7c354" containerName="payload-processing" containerID="cri-o://b7442d178bb44a6ff44f2c82bb2146204a34f48330ffe0999fb2aac2ce7cd281" gracePeriod=25 Nov 04 03:10:04.765723 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:10:04.765645 4405 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76597d6855-vxzxx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.125:15021/healthz/ready\": dial tcp 10.134.0.125:15021: connect: connection refused" start-of-body= Nov 04 03:10:04.765834 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:10:04.765773 4405 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-vxzxx" podUID="eeec2ff5-38c0-4e67-aeed-9f0baa9fa1d1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.125:15021/healthz/ready\": dial tcp 10.134.0.125:15021: connect: connection refused" Nov 04 03:10:04.774077 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:10:04.774040 4405 patch_prober.go:29] interesting pod/e2e-worker-w5-3b7476-openshift-default-59975457bf-q7gfm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.126:15021/healthz/ready\": dial tcp 10.134.0.126:15021: connect: connection refused" start-of-body= Nov 04 03:10:04.774252 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:10:04.774097 4405 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-3b7476-openshift-default-59975457bf-q7gfm" podUID="e6cb6771-a9e6-452f-a0b6-30ffb955feb8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.126:15021/healthz/ready\": dial tcp 10.134.0.126:15021: connect: connection refused" Nov 04 03:10:04.835253 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:10:04.835150 4405 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-00c846-5d7b447995-p8qhc"] Nov 04 03:10:05.089205 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:10:05.089177 4405 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-9d2b96-7d8dd46f5b-7rbgg" Nov 04 03:10:05.107189 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:10:05.107159 4405 generic.go:356] "Generic (PLEG): container finished" podID="24a56d15-dba0-46bc-aca3-f6fabdf7c354" containerID="b7442d178bb44a6ff44f2c82bb2146204a34f48330ffe0999fb2aac2ce7cd281" exitCode=0 Nov 04 03:10:05.107378 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:10:05.107204 4405 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-9d2b96-7d8dd46f5b-7rbgg" Nov 04 03:10:05.107378 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:10:05.107248 4405 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-9d2b96-7d8dd46f5b-7rbgg" event={"ID":"24a56d15-dba0-46bc-aca3-f6fabdf7c354","Type":"ContainerDied","Data":"b7442d178bb44a6ff44f2c82bb2146204a34f48330ffe0999fb2aac2ce7cd281"} Nov 04 03:10:05.107378 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:10:05.107305 4405 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-9d2b96-7d8dd46f5b-7rbgg" event={"ID":"24a56d15-dba0-46bc-aca3-f6fabdf7c354","Type":"ContainerDied","Data":"93927ced226f2285e3d490d2a8c9cf79d121fbbf6ee31245235760129d6ba624"} Nov 04 03:10:05.107378 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:10:05.107333 4405 scope.go:122] "RemoveContainer" containerID="b7442d178bb44a6ff44f2c82bb2146204a34f48330ffe0999fb2aac2ce7cd281" Nov 04 03:10:05.117991 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:10:05.117971 4405 scope.go:122] "RemoveContainer" containerID="b7442d178bb44a6ff44f2c82bb2146204a34f48330ffe0999fb2aac2ce7cd281" Nov 04 03:10:05.118320 ip-10-0-134-125 kubenswrapper[4405]: E1104 03:10:05.118300 4405 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b7442d178bb44a6ff44f2c82bb2146204a34f48330ffe0999fb2aac2ce7cd281\": container with ID starting with b7442d178bb44a6ff44f2c82bb2146204a34f48330ffe0999fb2aac2ce7cd281 not found: ID does not exist" containerID="b7442d178bb44a6ff44f2c82bb2146204a34f48330ffe0999fb2aac2ce7cd281" Nov 04 03:10:05.118408 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:10:05.118327 4405 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b7442d178bb44a6ff44f2c82bb2146204a34f48330ffe0999fb2aac2ce7cd281"} err="failed to get container status \"b7442d178bb44a6ff44f2c82bb2146204a34f48330ffe0999fb2aac2ce7cd281\": rpc error: code = NotFound desc = could not find container \"b7442d178bb44a6ff44f2c82bb2146204a34f48330ffe0999fb2aac2ce7cd281\": container with ID starting with b7442d178bb44a6ff44f2c82bb2146204a34f48330ffe0999fb2aac2ce7cd281 not found: ID does not exist" Nov 04 03:10:05.176355 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:10:05.163158 4405 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/24a56d15-dba0-46bc-aca3-f6fabdf7c354-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/24a56d15-dba0-46bc-aca3-f6fabdf7c354-plugins-config-volume\") pod \"24a56d15-dba0-46bc-aca3-f6fabdf7c354\" (UID: \"24a56d15-dba0-46bc-aca3-f6fabdf7c354\") " Nov 04 03:10:05.176355 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:10:05.163250 4405 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/24a56d15-dba0-46bc-aca3-f6fabdf7c354-kube-api-access-hmbw8\" (UniqueName: \"kubernetes.io/projected/24a56d15-dba0-46bc-aca3-f6fabdf7c354-kube-api-access-hmbw8\") pod \"24a56d15-dba0-46bc-aca3-f6fabdf7c354\" (UID: \"24a56d15-dba0-46bc-aca3-f6fabdf7c354\") " Nov 04 03:10:05.176355 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:10:05.175332 4405 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/24a56d15-dba0-46bc-aca3-f6fabdf7c354-plugins-config-volume" pod "24a56d15-dba0-46bc-aca3-f6fabdf7c354" (UID: "24a56d15-dba0-46bc-aca3-f6fabdf7c354"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 03:10:05.192625 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:10:05.192015 4405 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/24a56d15-dba0-46bc-aca3-f6fabdf7c354-plugins-config-volume\") on node \"ip-10-0-134-125.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:10:05.199177 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:10:05.199133 4405 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/24a56d15-dba0-46bc-aca3-f6fabdf7c354-kube-api-access-hmbw8" pod "24a56d15-dba0-46bc-aca3-f6fabdf7c354" (UID: "24a56d15-dba0-46bc-aca3-f6fabdf7c354"). InnerVolumeSpecName "kube-api-access-hmbw8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 03:10:05.294344 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:10:05.294300 4405 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hmbw8\" (UniqueName: \"kubernetes.io/projected/24a56d15-dba0-46bc-aca3-f6fabdf7c354-kube-api-access-hmbw8\") on node \"ip-10-0-134-125.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:10:05.541527 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:10:05.541489 4405 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-9d2b96-7d8dd46f5b-7rbgg"] Nov 04 03:10:05.548483 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:10:05.548442 4405 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-9d2b96-7d8dd46f5b-7rbgg"] Nov 04 03:10:05.711595 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:10:05.711492 4405 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="24a56d15-dba0-46bc-aca3-f6fabdf7c354" path="/var/lib/kubelet/pods/24a56d15-dba0-46bc-aca3-f6fabdf7c354/volumes" Nov 04 03:10:05.765828 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:10:05.765788 4405 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76597d6855-vxzxx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.125:15021/healthz/ready\": dial tcp 10.134.0.125:15021: connect: connection refused" start-of-body= Nov 04 03:10:05.766016 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:10:05.765866 4405 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-vxzxx" podUID="eeec2ff5-38c0-4e67-aeed-9f0baa9fa1d1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.125:15021/healthz/ready\": dial tcp 10.134.0.125:15021: connect: connection refused" Nov 04 03:10:05.774401 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:10:05.774338 4405 patch_prober.go:29] interesting pod/e2e-worker-w5-3b7476-openshift-default-59975457bf-q7gfm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.126:15021/healthz/ready\": dial tcp 10.134.0.126:15021: connect: connection refused" start-of-body= Nov 04 03:10:05.774527 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:10:05.774422 4405 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-3b7476-openshift-default-59975457bf-q7gfm" podUID="e6cb6771-a9e6-452f-a0b6-30ffb955feb8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.126:15021/healthz/ready\": dial tcp 10.134.0.126:15021: connect: connection refused" Nov 04 03:10:06.765134 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:10:06.765096 4405 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76597d6855-vxzxx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.125:15021/healthz/ready\": dial tcp 10.134.0.125:15021: connect: connection refused" start-of-body= Nov 04 03:10:06.765617 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:10:06.765176 4405 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-vxzxx" podUID="eeec2ff5-38c0-4e67-aeed-9f0baa9fa1d1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.125:15021/healthz/ready\": dial tcp 10.134.0.125:15021: connect: connection refused" Nov 04 03:10:06.779726 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:10:06.779694 4405 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w5-3b7476-openshift-default-59975457bf-q7gfm" Nov 04 03:10:06.780041 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:10:06.780018 4405 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w5-3b7476-openshift-default-59975457bf-q7gfm" Nov 04 03:10:06.780909 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:10:06.780888 4405 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w5-3b7476-openshift-default-59975457bf-q7gfm" Nov 04 03:10:06.903030 ip-10-0-134-125 kubenswrapper[4405]: E1104 03:10:06.902976 4405 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5f2vb" containerName="extractor" Nov 04 03:10:08.145546 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:10:08.143964 4405 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-vxzxx" Nov 04 03:10:08.145546 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:10:08.145389 4405 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-vxzxx" Nov 04 03:10:08.146687 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:10:08.146636 4405 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-vxzxx" Nov 04 03:10:09.728746 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:10:09.728706 4405 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-00c846-6865789c84-c24hp" podUID="f30fbe83-4589-49c6-bdd9-3e994b1a1d4d" containerName="payload-processing" containerID="cri-o://8ab91cade19aa24aa2f656d0bb1ca1713f4db2280c4b685520ac1f8ca946c11d" gracePeriod=25 Nov 04 03:10:09.836020 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:10:09.835961 4405 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-00c846-5d7b447995-p8qhc" podUID="10ad282e-7474-4938-8dee-81c65db8f4ae" containerName="payload-pre-processing" containerID="cri-o://e7275d6a058359b7202bdfb6abb769d805a8392045b32389e04c7c2c02a39681" gracePeriod=25 Nov 04 03:10:10.020301 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:10:10.020274 4405 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-00c846-6865789c84-c24hp" Nov 04 03:10:10.066797 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:10:10.066758 4405 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f30fbe83-4589-49c6-bdd9-3e994b1a1d4d-kube-api-access-vbcd5\" (UniqueName: \"kubernetes.io/projected/f30fbe83-4589-49c6-bdd9-3e994b1a1d4d-kube-api-access-vbcd5\") pod \"f30fbe83-4589-49c6-bdd9-3e994b1a1d4d\" (UID: \"f30fbe83-4589-49c6-bdd9-3e994b1a1d4d\") " Nov 04 03:10:10.066998 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:10:10.066832 4405 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f30fbe83-4589-49c6-bdd9-3e994b1a1d4d-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f30fbe83-4589-49c6-bdd9-3e994b1a1d4d-plugins-config-volume\") pod \"f30fbe83-4589-49c6-bdd9-3e994b1a1d4d\" (UID: \"f30fbe83-4589-49c6-bdd9-3e994b1a1d4d\") " Nov 04 03:10:10.067304 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:10:10.067258 4405 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f30fbe83-4589-49c6-bdd9-3e994b1a1d4d-plugins-config-volume" pod "f30fbe83-4589-49c6-bdd9-3e994b1a1d4d" (UID: "f30fbe83-4589-49c6-bdd9-3e994b1a1d4d"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 03:10:10.067541 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:10:10.067433 4405 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f30fbe83-4589-49c6-bdd9-3e994b1a1d4d-plugins-config-volume\") on node \"ip-10-0-134-125.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:10:10.068993 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:10:10.068969 4405 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f30fbe83-4589-49c6-bdd9-3e994b1a1d4d-kube-api-access-vbcd5" pod "f30fbe83-4589-49c6-bdd9-3e994b1a1d4d" (UID: "f30fbe83-4589-49c6-bdd9-3e994b1a1d4d"). InnerVolumeSpecName "kube-api-access-vbcd5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 03:10:10.109556 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:10:10.109533 4405 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-00c846-5d7b447995-p8qhc" Nov 04 03:10:10.161589 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:10:10.161553 4405 generic.go:356] "Generic (PLEG): container finished" podID="10ad282e-7474-4938-8dee-81c65db8f4ae" containerID="e7275d6a058359b7202bdfb6abb769d805a8392045b32389e04c7c2c02a39681" exitCode=0 Nov 04 03:10:10.161813 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:10:10.161602 4405 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-00c846-5d7b447995-p8qhc" Nov 04 03:10:10.161813 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:10:10.161643 4405 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-00c846-5d7b447995-p8qhc" event={"ID":"10ad282e-7474-4938-8dee-81c65db8f4ae","Type":"ContainerDied","Data":"e7275d6a058359b7202bdfb6abb769d805a8392045b32389e04c7c2c02a39681"} Nov 04 03:10:10.161813 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:10:10.161730 4405 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-00c846-5d7b447995-p8qhc" event={"ID":"10ad282e-7474-4938-8dee-81c65db8f4ae","Type":"ContainerDied","Data":"9b2bf6d510325c789cedde851d20ebe281921f13afc6af2ff5efc349d3bb93a8"} Nov 04 03:10:10.161813 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:10:10.161764 4405 scope.go:122] "RemoveContainer" containerID="e7275d6a058359b7202bdfb6abb769d805a8392045b32389e04c7c2c02a39681" Nov 04 03:10:10.163182 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:10:10.163158 4405 generic.go:356] "Generic (PLEG): container finished" podID="f30fbe83-4589-49c6-bdd9-3e994b1a1d4d" containerID="8ab91cade19aa24aa2f656d0bb1ca1713f4db2280c4b685520ac1f8ca946c11d" exitCode=0 Nov 04 03:10:10.163273 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:10:10.163193 4405 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-00c846-6865789c84-c24hp" event={"ID":"f30fbe83-4589-49c6-bdd9-3e994b1a1d4d","Type":"ContainerDied","Data":"8ab91cade19aa24aa2f656d0bb1ca1713f4db2280c4b685520ac1f8ca946c11d"} Nov 04 03:10:10.163273 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:10:10.163252 4405 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-00c846-6865789c84-c24hp" event={"ID":"f30fbe83-4589-49c6-bdd9-3e994b1a1d4d","Type":"ContainerDied","Data":"c1edee9b7bb51c538d0b4baa897d010abdabb3b0f283a21455f67916d965df8c"} Nov 04 03:10:10.163388 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:10:10.163286 4405 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-00c846-6865789c84-c24hp" Nov 04 03:10:10.168223 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:10:10.168194 4405 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/10ad282e-7474-4938-8dee-81c65db8f4ae-kube-api-access-97n2r\" (UniqueName: \"kubernetes.io/projected/10ad282e-7474-4938-8dee-81c65db8f4ae-kube-api-access-97n2r\") pod \"10ad282e-7474-4938-8dee-81c65db8f4ae\" (UID: \"10ad282e-7474-4938-8dee-81c65db8f4ae\") " Nov 04 03:10:10.168481 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:10:10.168463 4405 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/10ad282e-7474-4938-8dee-81c65db8f4ae-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/10ad282e-7474-4938-8dee-81c65db8f4ae-plugins-config-volume\") pod \"10ad282e-7474-4938-8dee-81c65db8f4ae\" (UID: \"10ad282e-7474-4938-8dee-81c65db8f4ae\") " Nov 04 03:10:10.168940 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:10:10.168877 4405 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/10ad282e-7474-4938-8dee-81c65db8f4ae-plugins-config-volume" pod "10ad282e-7474-4938-8dee-81c65db8f4ae" (UID: "10ad282e-7474-4938-8dee-81c65db8f4ae"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 03:10:10.169033 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:10:10.169010 4405 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vbcd5\" (UniqueName: \"kubernetes.io/projected/f30fbe83-4589-49c6-bdd9-3e994b1a1d4d-kube-api-access-vbcd5\") on node \"ip-10-0-134-125.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:10:10.169075 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:10:10.169030 4405 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/10ad282e-7474-4938-8dee-81c65db8f4ae-plugins-config-volume\") on node \"ip-10-0-134-125.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:10:10.171118 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:10:10.171083 4405 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/10ad282e-7474-4938-8dee-81c65db8f4ae-kube-api-access-97n2r" pod "10ad282e-7474-4938-8dee-81c65db8f4ae" (UID: "10ad282e-7474-4938-8dee-81c65db8f4ae"). InnerVolumeSpecName "kube-api-access-97n2r". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 03:10:10.172820 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:10:10.172804 4405 scope.go:122] "RemoveContainer" containerID="e7275d6a058359b7202bdfb6abb769d805a8392045b32389e04c7c2c02a39681" Nov 04 03:10:10.173166 ip-10-0-134-125 kubenswrapper[4405]: E1104 03:10:10.173145 4405 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e7275d6a058359b7202bdfb6abb769d805a8392045b32389e04c7c2c02a39681\": container with ID starting with e7275d6a058359b7202bdfb6abb769d805a8392045b32389e04c7c2c02a39681 not found: ID does not exist" containerID="e7275d6a058359b7202bdfb6abb769d805a8392045b32389e04c7c2c02a39681" Nov 04 03:10:10.173261 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:10:10.173176 4405 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e7275d6a058359b7202bdfb6abb769d805a8392045b32389e04c7c2c02a39681"} err="failed to get container status \"e7275d6a058359b7202bdfb6abb769d805a8392045b32389e04c7c2c02a39681\": rpc error: code = NotFound desc = could not find container \"e7275d6a058359b7202bdfb6abb769d805a8392045b32389e04c7c2c02a39681\": container with ID starting with e7275d6a058359b7202bdfb6abb769d805a8392045b32389e04c7c2c02a39681 not found: ID does not exist" Nov 04 03:10:10.173261 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:10:10.173194 4405 scope.go:122] "RemoveContainer" containerID="8ab91cade19aa24aa2f656d0bb1ca1713f4db2280c4b685520ac1f8ca946c11d" Nov 04 03:10:10.185502 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:10:10.185479 4405 scope.go:122] "RemoveContainer" containerID="8ab91cade19aa24aa2f656d0bb1ca1713f4db2280c4b685520ac1f8ca946c11d" Nov 04 03:10:10.185780 ip-10-0-134-125 kubenswrapper[4405]: E1104 03:10:10.185761 4405 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8ab91cade19aa24aa2f656d0bb1ca1713f4db2280c4b685520ac1f8ca946c11d\": container with ID starting with 8ab91cade19aa24aa2f656d0bb1ca1713f4db2280c4b685520ac1f8ca946c11d not found: ID does not exist" containerID="8ab91cade19aa24aa2f656d0bb1ca1713f4db2280c4b685520ac1f8ca946c11d" Nov 04 03:10:10.185840 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:10:10.185786 4405 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8ab91cade19aa24aa2f656d0bb1ca1713f4db2280c4b685520ac1f8ca946c11d"} err="failed to get container status \"8ab91cade19aa24aa2f656d0bb1ca1713f4db2280c4b685520ac1f8ca946c11d\": rpc error: code = NotFound desc = could not find container \"8ab91cade19aa24aa2f656d0bb1ca1713f4db2280c4b685520ac1f8ca946c11d\": container with ID starting with 8ab91cade19aa24aa2f656d0bb1ca1713f4db2280c4b685520ac1f8ca946c11d not found: ID does not exist" Nov 04 03:10:10.270837 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:10:10.270745 4405 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-97n2r\" (UniqueName: \"kubernetes.io/projected/10ad282e-7474-4938-8dee-81c65db8f4ae-kube-api-access-97n2r\") on node \"ip-10-0-134-125.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:10:10.365815 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:10:10.365779 4405 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-00c846-6865789c84-c24hp"] Nov 04 03:10:10.370267 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:10:10.370241 4405 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-00c846-6865789c84-c24hp"] Nov 04 03:10:10.587182 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:10:10.587144 4405 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-00c846-5d7b447995-p8qhc"] Nov 04 03:10:10.590914 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:10:10.590889 4405 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-00c846-5d7b447995-p8qhc"] Nov 04 03:10:11.427308 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:10:11.427277 4405 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-22e72e-85b6d5f746-nw6tv"] Nov 04 03:10:11.552971 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:10:11.552938 4405 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-22e72e-7ffdf7c747-vbt9t"] Nov 04 03:10:11.710581 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:10:11.710485 4405 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="10ad282e-7474-4938-8dee-81c65db8f4ae" path="/var/lib/kubelet/pods/10ad282e-7474-4938-8dee-81c65db8f4ae/volumes" Nov 04 03:10:11.711150 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:10:11.711131 4405 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f30fbe83-4589-49c6-bdd9-3e994b1a1d4d" path="/var/lib/kubelet/pods/f30fbe83-4589-49c6-bdd9-3e994b1a1d4d/volumes" Nov 04 03:10:16.428143 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:10:16.428090 4405 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-a-22e72e-85b6d5f746-nw6tv" podUID="e6996ff6-b08c-4c58-916d-b79ae32ade57" containerName="payload-processing" containerID="cri-o://0756f8b9248fc773c1c0c7399ff1099b004d30c313df24cc1c5bb041feb84ebb" gracePeriod=25 Nov 04 03:10:16.554178 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:10:16.554139 4405 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-22e72e-7ffdf7c747-vbt9t" podUID="3dcb7be0-d703-40eb-8bc4-9228972e6376" containerName="payload-pre-processing" containerID="cri-o://f0bee4ce4f85cd635f14d3de269c65283086c9c46e53e9d981264dbb778f4b4d" gracePeriod=25 Nov 04 03:10:16.721850 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:10:16.721819 4405 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-22e72e-85b6d5f746-nw6tv" Nov 04 03:10:16.831716 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:10:16.831657 4405 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-22e72e-7ffdf7c747-vbt9t" Nov 04 03:10:16.873346 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:10:16.873295 4405 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e6996ff6-b08c-4c58-916d-b79ae32ade57-kube-api-access-h4f65\" (UniqueName: \"kubernetes.io/projected/e6996ff6-b08c-4c58-916d-b79ae32ade57-kube-api-access-h4f65\") pod \"e6996ff6-b08c-4c58-916d-b79ae32ade57\" (UID: \"e6996ff6-b08c-4c58-916d-b79ae32ade57\") " Nov 04 03:10:16.873690 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:10:16.873653 4405 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e6996ff6-b08c-4c58-916d-b79ae32ade57-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e6996ff6-b08c-4c58-916d-b79ae32ade57-plugins-config-volume\") pod \"e6996ff6-b08c-4c58-916d-b79ae32ade57\" (UID: \"e6996ff6-b08c-4c58-916d-b79ae32ade57\") " Nov 04 03:10:16.874024 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:10:16.873998 4405 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e6996ff6-b08c-4c58-916d-b79ae32ade57-plugins-config-volume" pod "e6996ff6-b08c-4c58-916d-b79ae32ade57" (UID: "e6996ff6-b08c-4c58-916d-b79ae32ade57"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 03:10:16.874750 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:10:16.874723 4405 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e6996ff6-b08c-4c58-916d-b79ae32ade57-plugins-config-volume\") on node \"ip-10-0-134-125.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:10:16.876344 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:10:16.876323 4405 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e6996ff6-b08c-4c58-916d-b79ae32ade57-kube-api-access-h4f65" pod "e6996ff6-b08c-4c58-916d-b79ae32ade57" (UID: "e6996ff6-b08c-4c58-916d-b79ae32ade57"). InnerVolumeSpecName "kube-api-access-h4f65". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 03:10:16.976229 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:10:16.976139 4405 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3dcb7be0-d703-40eb-8bc4-9228972e6376-kube-api-access-jjnr9\" (UniqueName: \"kubernetes.io/projected/3dcb7be0-d703-40eb-8bc4-9228972e6376-kube-api-access-jjnr9\") pod \"3dcb7be0-d703-40eb-8bc4-9228972e6376\" (UID: \"3dcb7be0-d703-40eb-8bc4-9228972e6376\") " Nov 04 03:10:16.976402 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:10:16.976269 4405 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3dcb7be0-d703-40eb-8bc4-9228972e6376-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3dcb7be0-d703-40eb-8bc4-9228972e6376-plugins-config-volume\") pod \"3dcb7be0-d703-40eb-8bc4-9228972e6376\" (UID: \"3dcb7be0-d703-40eb-8bc4-9228972e6376\") " Nov 04 03:10:16.976716 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:10:16.976681 4405 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3dcb7be0-d703-40eb-8bc4-9228972e6376-plugins-config-volume" pod "3dcb7be0-d703-40eb-8bc4-9228972e6376" (UID: "3dcb7be0-d703-40eb-8bc4-9228972e6376"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 03:10:16.976983 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:10:16.976963 4405 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h4f65\" (UniqueName: \"kubernetes.io/projected/e6996ff6-b08c-4c58-916d-b79ae32ade57-kube-api-access-h4f65\") on node \"ip-10-0-134-125.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:10:16.977075 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:10:16.976989 4405 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3dcb7be0-d703-40eb-8bc4-9228972e6376-plugins-config-volume\") on node \"ip-10-0-134-125.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:10:16.978594 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:10:16.978574 4405 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3dcb7be0-d703-40eb-8bc4-9228972e6376-kube-api-access-jjnr9" pod "3dcb7be0-d703-40eb-8bc4-9228972e6376" (UID: "3dcb7be0-d703-40eb-8bc4-9228972e6376"). InnerVolumeSpecName "kube-api-access-jjnr9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 03:10:17.078584 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:10:17.078548 4405 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jjnr9\" (UniqueName: \"kubernetes.io/projected/3dcb7be0-d703-40eb-8bc4-9228972e6376-kube-api-access-jjnr9\") on node \"ip-10-0-134-125.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:10:17.233031 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:10:17.232931 4405 generic.go:356] "Generic (PLEG): container finished" podID="e6996ff6-b08c-4c58-916d-b79ae32ade57" containerID="0756f8b9248fc773c1c0c7399ff1099b004d30c313df24cc1c5bb041feb84ebb" exitCode=0 Nov 04 03:10:17.233031 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:10:17.232994 4405 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-22e72e-85b6d5f746-nw6tv" Nov 04 03:10:17.233301 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:10:17.233028 4405 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-22e72e-85b6d5f746-nw6tv" event={"ID":"e6996ff6-b08c-4c58-916d-b79ae32ade57","Type":"ContainerDied","Data":"0756f8b9248fc773c1c0c7399ff1099b004d30c313df24cc1c5bb041feb84ebb"} Nov 04 03:10:17.233301 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:10:17.233099 4405 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-22e72e-85b6d5f746-nw6tv" event={"ID":"e6996ff6-b08c-4c58-916d-b79ae32ade57","Type":"ContainerDied","Data":"20cc72c3e791df4ce671a0dbcd24e6874a54634394fa0778496eacffcde0f531"} Nov 04 03:10:17.233301 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:10:17.233133 4405 scope.go:122] "RemoveContainer" containerID="0756f8b9248fc773c1c0c7399ff1099b004d30c313df24cc1c5bb041feb84ebb" Nov 04 03:10:17.239087 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:10:17.239058 4405 generic.go:356] "Generic (PLEG): container finished" podID="3dcb7be0-d703-40eb-8bc4-9228972e6376" containerID="f0bee4ce4f85cd635f14d3de269c65283086c9c46e53e9d981264dbb778f4b4d" exitCode=0 Nov 04 03:10:17.239239 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:10:17.239133 4405 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-22e72e-7ffdf7c747-vbt9t" event={"ID":"3dcb7be0-d703-40eb-8bc4-9228972e6376","Type":"ContainerDied","Data":"f0bee4ce4f85cd635f14d3de269c65283086c9c46e53e9d981264dbb778f4b4d"} Nov 04 03:10:17.239239 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:10:17.239138 4405 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-22e72e-7ffdf7c747-vbt9t" Nov 04 03:10:17.239239 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:10:17.239183 4405 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-22e72e-7ffdf7c747-vbt9t" event={"ID":"3dcb7be0-d703-40eb-8bc4-9228972e6376","Type":"ContainerDied","Data":"1eb8986d7347b5293c0d5c33ea356ba6f5370ce3399ccc750b5317d035cb51cc"} Nov 04 03:10:17.247151 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:10:17.247130 4405 scope.go:122] "RemoveContainer" containerID="0756f8b9248fc773c1c0c7399ff1099b004d30c313df24cc1c5bb041feb84ebb" Nov 04 03:10:17.247478 ip-10-0-134-125 kubenswrapper[4405]: E1104 03:10:17.247450 4405 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0756f8b9248fc773c1c0c7399ff1099b004d30c313df24cc1c5bb041feb84ebb\": container with ID starting with 0756f8b9248fc773c1c0c7399ff1099b004d30c313df24cc1c5bb041feb84ebb not found: ID does not exist" containerID="0756f8b9248fc773c1c0c7399ff1099b004d30c313df24cc1c5bb041feb84ebb" Nov 04 03:10:17.247588 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:10:17.247488 4405 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0756f8b9248fc773c1c0c7399ff1099b004d30c313df24cc1c5bb041feb84ebb"} err="failed to get container status \"0756f8b9248fc773c1c0c7399ff1099b004d30c313df24cc1c5bb041feb84ebb\": rpc error: code = NotFound desc = could not find container \"0756f8b9248fc773c1c0c7399ff1099b004d30c313df24cc1c5bb041feb84ebb\": container with ID starting with 0756f8b9248fc773c1c0c7399ff1099b004d30c313df24cc1c5bb041feb84ebb not found: ID does not exist" Nov 04 03:10:17.247588 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:10:17.247510 4405 scope.go:122] "RemoveContainer" containerID="f0bee4ce4f85cd635f14d3de269c65283086c9c46e53e9d981264dbb778f4b4d" Nov 04 03:10:17.257598 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:10:17.257576 4405 scope.go:122] "RemoveContainer" containerID="f0bee4ce4f85cd635f14d3de269c65283086c9c46e53e9d981264dbb778f4b4d" Nov 04 03:10:17.257911 ip-10-0-134-125 kubenswrapper[4405]: E1104 03:10:17.257890 4405 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f0bee4ce4f85cd635f14d3de269c65283086c9c46e53e9d981264dbb778f4b4d\": container with ID starting with f0bee4ce4f85cd635f14d3de269c65283086c9c46e53e9d981264dbb778f4b4d not found: ID does not exist" containerID="f0bee4ce4f85cd635f14d3de269c65283086c9c46e53e9d981264dbb778f4b4d" Nov 04 03:10:17.257975 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:10:17.257921 4405 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f0bee4ce4f85cd635f14d3de269c65283086c9c46e53e9d981264dbb778f4b4d"} err="failed to get container status \"f0bee4ce4f85cd635f14d3de269c65283086c9c46e53e9d981264dbb778f4b4d\": rpc error: code = NotFound desc = could not find container \"f0bee4ce4f85cd635f14d3de269c65283086c9c46e53e9d981264dbb778f4b4d\": container with ID starting with f0bee4ce4f85cd635f14d3de269c65283086c9c46e53e9d981264dbb778f4b4d not found: ID does not exist" Nov 04 03:10:17.397480 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:10:17.397439 4405 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-22e72e-85b6d5f746-nw6tv"] Nov 04 03:10:17.401972 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:10:17.401943 4405 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-22e72e-85b6d5f746-nw6tv"] Nov 04 03:10:17.511114 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:10:17.511024 4405 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-22e72e-7ffdf7c747-vbt9t"] Nov 04 03:10:17.516551 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:10:17.516520 4405 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-22e72e-7ffdf7c747-vbt9t"] Nov 04 03:10:17.703761 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:10:17.703719 4405 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3dcb7be0-d703-40eb-8bc4-9228972e6376" path="/var/lib/kubelet/pods/3dcb7be0-d703-40eb-8bc4-9228972e6376/volumes" Nov 04 03:10:17.704256 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:10:17.704234 4405 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e6996ff6-b08c-4c58-916d-b79ae32ade57" path="/var/lib/kubelet/pods/e6996ff6-b08c-4c58-916d-b79ae32ade57/volumes" Nov 04 03:10:19.822323 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:10:19.822285 4405 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-4cbcf738-openshift-default-68fb497c9c-2ndqr"] Nov 04 03:10:19.822910 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:10:19.822895 4405 cpu_manager.go:414] "containerMap: removing container" podUID="fd2b4daf-d65f-44ca-8939-67778fe8ef5a" containerName="istio-proxy" Nov 04 03:10:19.822966 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:10:19.822912 4405 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fd2b4daf-d65f-44ca-8939-67778fe8ef5a" containerName="istio-proxy" Nov 04 03:10:19.822966 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:10:19.822928 4405 cpu_manager.go:414] "containerMap: removing container" podUID="f30fbe83-4589-49c6-bdd9-3e994b1a1d4d" containerName="payload-processing" Nov 04 03:10:19.822966 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:10:19.822934 4405 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f30fbe83-4589-49c6-bdd9-3e994b1a1d4d" containerName="payload-processing" Nov 04 03:10:19.822966 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:10:19.822957 4405 cpu_manager.go:414] "containerMap: removing container" podUID="3dcb7be0-d703-40eb-8bc4-9228972e6376" containerName="payload-pre-processing" Nov 04 03:10:19.822966 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:10:19.822963 4405 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3dcb7be0-d703-40eb-8bc4-9228972e6376" containerName="payload-pre-processing" Nov 04 03:10:19.823128 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:10:19.822980 4405 cpu_manager.go:414] "containerMap: removing container" podUID="e6996ff6-b08c-4c58-916d-b79ae32ade57" containerName="payload-processing" Nov 04 03:10:19.823128 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:10:19.822985 4405 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e6996ff6-b08c-4c58-916d-b79ae32ade57" containerName="payload-processing" Nov 04 03:10:19.823128 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:10:19.822991 4405 cpu_manager.go:414] "containerMap: removing container" podUID="10ad282e-7474-4938-8dee-81c65db8f4ae" containerName="payload-pre-processing" Nov 04 03:10:19.823128 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:10:19.822996 4405 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="10ad282e-7474-4938-8dee-81c65db8f4ae" containerName="payload-pre-processing" Nov 04 03:10:19.823128 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:10:19.823008 4405 cpu_manager.go:414] "containerMap: removing container" podUID="24a56d15-dba0-46bc-aca3-f6fabdf7c354" containerName="payload-processing" Nov 04 03:10:19.823128 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:10:19.823013 4405 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="24a56d15-dba0-46bc-aca3-f6fabdf7c354" containerName="payload-processing" Nov 04 03:10:19.823128 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:10:19.823097 4405 memory_manager.go:365] "RemoveStaleState removing state" podUID="3dcb7be0-d703-40eb-8bc4-9228972e6376" containerName="payload-pre-processing" Nov 04 03:10:19.823128 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:10:19.823107 4405 memory_manager.go:365] "RemoveStaleState removing state" podUID="e6996ff6-b08c-4c58-916d-b79ae32ade57" containerName="payload-processing" Nov 04 03:10:19.823128 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:10:19.823113 4405 memory_manager.go:365] "RemoveStaleState removing state" podUID="10ad282e-7474-4938-8dee-81c65db8f4ae" containerName="payload-pre-processing" Nov 04 03:10:19.823128 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:10:19.823119 4405 memory_manager.go:365] "RemoveStaleState removing state" podUID="fd2b4daf-d65f-44ca-8939-67778fe8ef5a" containerName="istio-proxy" Nov 04 03:10:19.823128 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:10:19.823125 4405 memory_manager.go:365] "RemoveStaleState removing state" podUID="24a56d15-dba0-46bc-aca3-f6fabdf7c354" containerName="payload-processing" Nov 04 03:10:19.823465 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:10:19.823135 4405 memory_manager.go:365] "RemoveStaleState removing state" podUID="fd2b4daf-d65f-44ca-8939-67778fe8ef5a" containerName="istio-proxy" Nov 04 03:10:19.823465 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:10:19.823141 4405 memory_manager.go:365] "RemoveStaleState removing state" podUID="f30fbe83-4589-49c6-bdd9-3e994b1a1d4d" containerName="payload-processing" Nov 04 03:10:19.828004 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:10:19.827984 4405 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-4cbcf738-openshift-default-68fb497c9c-2ndqr" Nov 04 03:10:19.867863 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:10:19.867793 4405 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a77157ba-0aed-46cf-90de-ffc15074a50f-istio-token\") pod \"e2e-mt-4cbcf738-openshift-default-68fb497c9c-2ndqr\" (UID: \"a77157ba-0aed-46cf-90de-ffc15074a50f\") " pod="openshift-ingress/e2e-mt-4cbcf738-openshift-default-68fb497c9c-2ndqr" Nov 04 03:10:19.868176 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:10:19.868152 4405 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a77157ba-0aed-46cf-90de-ffc15074a50f-istio-data\") pod \"e2e-mt-4cbcf738-openshift-default-68fb497c9c-2ndqr\" (UID: \"a77157ba-0aed-46cf-90de-ffc15074a50f\") " pod="openshift-ingress/e2e-mt-4cbcf738-openshift-default-68fb497c9c-2ndqr" Nov 04 03:10:19.868530 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:10:19.868473 4405 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a77157ba-0aed-46cf-90de-ffc15074a50f-workload-socket\") pod \"e2e-mt-4cbcf738-openshift-default-68fb497c9c-2ndqr\" (UID: \"a77157ba-0aed-46cf-90de-ffc15074a50f\") " pod="openshift-ingress/e2e-mt-4cbcf738-openshift-default-68fb497c9c-2ndqr" Nov 04 03:10:19.868908 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:10:19.868888 4405 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a77157ba-0aed-46cf-90de-ffc15074a50f-credential-socket\") pod \"e2e-mt-4cbcf738-openshift-default-68fb497c9c-2ndqr\" (UID: \"a77157ba-0aed-46cf-90de-ffc15074a50f\") " pod="openshift-ingress/e2e-mt-4cbcf738-openshift-default-68fb497c9c-2ndqr" Nov 04 03:10:19.869111 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:10:19.869096 4405 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a77157ba-0aed-46cf-90de-ffc15074a50f-workload-certs\") pod \"e2e-mt-4cbcf738-openshift-default-68fb497c9c-2ndqr\" (UID: \"a77157ba-0aed-46cf-90de-ffc15074a50f\") " pod="openshift-ingress/e2e-mt-4cbcf738-openshift-default-68fb497c9c-2ndqr" Nov 04 03:10:19.869270 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:10:19.869255 4405 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a77157ba-0aed-46cf-90de-ffc15074a50f-istio-envoy\") pod \"e2e-mt-4cbcf738-openshift-default-68fb497c9c-2ndqr\" (UID: \"a77157ba-0aed-46cf-90de-ffc15074a50f\") " pod="openshift-ingress/e2e-mt-4cbcf738-openshift-default-68fb497c9c-2ndqr" Nov 04 03:10:19.869431 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:10:19.869415 4405 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a77157ba-0aed-46cf-90de-ffc15074a50f-istiod-ca-cert\") pod \"e2e-mt-4cbcf738-openshift-default-68fb497c9c-2ndqr\" (UID: \"a77157ba-0aed-46cf-90de-ffc15074a50f\") " pod="openshift-ingress/e2e-mt-4cbcf738-openshift-default-68fb497c9c-2ndqr" Nov 04 03:10:19.869588 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:10:19.869571 4405 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a77157ba-0aed-46cf-90de-ffc15074a50f-istio-podinfo\") pod \"e2e-mt-4cbcf738-openshift-default-68fb497c9c-2ndqr\" (UID: \"a77157ba-0aed-46cf-90de-ffc15074a50f\") " pod="openshift-ingress/e2e-mt-4cbcf738-openshift-default-68fb497c9c-2ndqr" Nov 04 03:10:19.869769 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:10:19.869750 4405 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xxs4z\" (UniqueName: \"kubernetes.io/projected/a77157ba-0aed-46cf-90de-ffc15074a50f-kube-api-access-xxs4z\") pod \"e2e-mt-4cbcf738-openshift-default-68fb497c9c-2ndqr\" (UID: \"a77157ba-0aed-46cf-90de-ffc15074a50f\") " pod="openshift-ingress/e2e-mt-4cbcf738-openshift-default-68fb497c9c-2ndqr" Nov 04 03:10:19.916719 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:10:19.911646 4405 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-4cbcf738-openshift-default-dockercfg-czrdg\"" Nov 04 03:10:19.967104 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:10:19.967067 4405 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-4cbcf738-openshift-default-68fb497c9c-2ndqr"] Nov 04 03:10:19.971727 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:10:19.971042 4405 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a77157ba-0aed-46cf-90de-ffc15074a50f-credential-socket\") pod \"e2e-mt-4cbcf738-openshift-default-68fb497c9c-2ndqr\" (UID: \"a77157ba-0aed-46cf-90de-ffc15074a50f\") " pod="openshift-ingress/e2e-mt-4cbcf738-openshift-default-68fb497c9c-2ndqr" Nov 04 03:10:19.971727 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:10:19.971116 4405 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a77157ba-0aed-46cf-90de-ffc15074a50f-workload-certs\") pod \"e2e-mt-4cbcf738-openshift-default-68fb497c9c-2ndqr\" (UID: \"a77157ba-0aed-46cf-90de-ffc15074a50f\") " pod="openshift-ingress/e2e-mt-4cbcf738-openshift-default-68fb497c9c-2ndqr" Nov 04 03:10:19.971727 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:10:19.971162 4405 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a77157ba-0aed-46cf-90de-ffc15074a50f-istio-envoy\") pod \"e2e-mt-4cbcf738-openshift-default-68fb497c9c-2ndqr\" (UID: \"a77157ba-0aed-46cf-90de-ffc15074a50f\") " pod="openshift-ingress/e2e-mt-4cbcf738-openshift-default-68fb497c9c-2ndqr" Nov 04 03:10:19.971727 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:10:19.971206 4405 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a77157ba-0aed-46cf-90de-ffc15074a50f-istiod-ca-cert\") pod \"e2e-mt-4cbcf738-openshift-default-68fb497c9c-2ndqr\" (UID: \"a77157ba-0aed-46cf-90de-ffc15074a50f\") " pod="openshift-ingress/e2e-mt-4cbcf738-openshift-default-68fb497c9c-2ndqr" Nov 04 03:10:19.971727 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:10:19.971259 4405 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a77157ba-0aed-46cf-90de-ffc15074a50f-istio-podinfo\") pod \"e2e-mt-4cbcf738-openshift-default-68fb497c9c-2ndqr\" (UID: \"a77157ba-0aed-46cf-90de-ffc15074a50f\") " pod="openshift-ingress/e2e-mt-4cbcf738-openshift-default-68fb497c9c-2ndqr" Nov 04 03:10:19.971727 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:10:19.971293 4405 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xxs4z\" (UniqueName: \"kubernetes.io/projected/a77157ba-0aed-46cf-90de-ffc15074a50f-kube-api-access-xxs4z\") pod \"e2e-mt-4cbcf738-openshift-default-68fb497c9c-2ndqr\" (UID: \"a77157ba-0aed-46cf-90de-ffc15074a50f\") " pod="openshift-ingress/e2e-mt-4cbcf738-openshift-default-68fb497c9c-2ndqr" Nov 04 03:10:19.971727 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:10:19.971377 4405 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a77157ba-0aed-46cf-90de-ffc15074a50f-istio-token\") pod \"e2e-mt-4cbcf738-openshift-default-68fb497c9c-2ndqr\" (UID: \"a77157ba-0aed-46cf-90de-ffc15074a50f\") " pod="openshift-ingress/e2e-mt-4cbcf738-openshift-default-68fb497c9c-2ndqr" Nov 04 03:10:19.971727 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:10:19.971416 4405 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a77157ba-0aed-46cf-90de-ffc15074a50f-istio-data\") pod \"e2e-mt-4cbcf738-openshift-default-68fb497c9c-2ndqr\" (UID: \"a77157ba-0aed-46cf-90de-ffc15074a50f\") " pod="openshift-ingress/e2e-mt-4cbcf738-openshift-default-68fb497c9c-2ndqr" Nov 04 03:10:19.971727 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:10:19.971547 4405 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a77157ba-0aed-46cf-90de-ffc15074a50f-workload-socket\") pod \"e2e-mt-4cbcf738-openshift-default-68fb497c9c-2ndqr\" (UID: \"a77157ba-0aed-46cf-90de-ffc15074a50f\") " pod="openshift-ingress/e2e-mt-4cbcf738-openshift-default-68fb497c9c-2ndqr" Nov 04 03:10:19.972802 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:10:19.972541 4405 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a77157ba-0aed-46cf-90de-ffc15074a50f-istiod-ca-cert\") pod \"e2e-mt-4cbcf738-openshift-default-68fb497c9c-2ndqr\" (UID: \"a77157ba-0aed-46cf-90de-ffc15074a50f\") " pod="openshift-ingress/e2e-mt-4cbcf738-openshift-default-68fb497c9c-2ndqr" Nov 04 03:10:19.973336 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:10:19.972952 4405 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a77157ba-0aed-46cf-90de-ffc15074a50f-workload-socket\") pod \"e2e-mt-4cbcf738-openshift-default-68fb497c9c-2ndqr\" (UID: \"a77157ba-0aed-46cf-90de-ffc15074a50f\") " pod="openshift-ingress/e2e-mt-4cbcf738-openshift-default-68fb497c9c-2ndqr" Nov 04 03:10:19.973336 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:10:19.973006 4405 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a77157ba-0aed-46cf-90de-ffc15074a50f-credential-socket\") pod \"e2e-mt-4cbcf738-openshift-default-68fb497c9c-2ndqr\" (UID: \"a77157ba-0aed-46cf-90de-ffc15074a50f\") " pod="openshift-ingress/e2e-mt-4cbcf738-openshift-default-68fb497c9c-2ndqr" Nov 04 03:10:19.973336 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:10:19.973235 4405 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a77157ba-0aed-46cf-90de-ffc15074a50f-workload-certs\") pod \"e2e-mt-4cbcf738-openshift-default-68fb497c9c-2ndqr\" (UID: \"a77157ba-0aed-46cf-90de-ffc15074a50f\") " pod="openshift-ingress/e2e-mt-4cbcf738-openshift-default-68fb497c9c-2ndqr" Nov 04 03:10:19.974545 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:10:19.974522 4405 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a77157ba-0aed-46cf-90de-ffc15074a50f-istio-data\") pod \"e2e-mt-4cbcf738-openshift-default-68fb497c9c-2ndqr\" (UID: \"a77157ba-0aed-46cf-90de-ffc15074a50f\") " pod="openshift-ingress/e2e-mt-4cbcf738-openshift-default-68fb497c9c-2ndqr" Nov 04 03:10:19.978035 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:10:19.978014 4405 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a77157ba-0aed-46cf-90de-ffc15074a50f-istio-envoy\") pod \"e2e-mt-4cbcf738-openshift-default-68fb497c9c-2ndqr\" (UID: \"a77157ba-0aed-46cf-90de-ffc15074a50f\") " pod="openshift-ingress/e2e-mt-4cbcf738-openshift-default-68fb497c9c-2ndqr" Nov 04 03:10:19.981587 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:10:19.981544 4405 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a77157ba-0aed-46cf-90de-ffc15074a50f-istio-podinfo\") pod \"e2e-mt-4cbcf738-openshift-default-68fb497c9c-2ndqr\" (UID: \"a77157ba-0aed-46cf-90de-ffc15074a50f\") " pod="openshift-ingress/e2e-mt-4cbcf738-openshift-default-68fb497c9c-2ndqr" Nov 04 03:10:20.036035 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:10:20.036005 4405 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xxs4z\" (UniqueName: \"kubernetes.io/projected/a77157ba-0aed-46cf-90de-ffc15074a50f-kube-api-access-xxs4z\") pod \"e2e-mt-4cbcf738-openshift-default-68fb497c9c-2ndqr\" (UID: \"a77157ba-0aed-46cf-90de-ffc15074a50f\") " pod="openshift-ingress/e2e-mt-4cbcf738-openshift-default-68fb497c9c-2ndqr" Nov 04 03:10:20.039391 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:10:20.039345 4405 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a77157ba-0aed-46cf-90de-ffc15074a50f-istio-token\") pod \"e2e-mt-4cbcf738-openshift-default-68fb497c9c-2ndqr\" (UID: \"a77157ba-0aed-46cf-90de-ffc15074a50f\") " pod="openshift-ingress/e2e-mt-4cbcf738-openshift-default-68fb497c9c-2ndqr" Nov 04 03:10:20.162744 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:10:20.162712 4405 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-4cbcf738-openshift-default-68fb497c9c-2ndqr" Nov 04 03:10:20.603150 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:10:20.603118 4405 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-4cbcf738-openshift-default-68fb497c9c-2ndqr"] Nov 04 03:10:20.604295 ip-10-0-134-125 kubenswrapper[4405]: W1104 03:10:20.604270 4405 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda77157ba_0aed_46cf_90de_ffc15074a50f.slice/crio-e57f052b1c1e308e3db04076ce75ab0841ad6d56803824fa0ad5b0ee0a050b4b WatchSource:0}: Error finding container e57f052b1c1e308e3db04076ce75ab0841ad6d56803824fa0ad5b0ee0a050b4b: Status 404 returned error can't find the container with id e57f052b1c1e308e3db04076ce75ab0841ad6d56803824fa0ad5b0ee0a050b4b Nov 04 03:10:20.607964 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:10:20.607932 4405 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 04 03:10:20.608057 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:10:20.608003 4405 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 04 03:10:20.608057 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:10:20.608036 4405 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 04 03:10:21.302783 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:10:21.302685 4405 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-4cbcf738-openshift-default-68fb497c9c-2ndqr" event={"ID":"a77157ba-0aed-46cf-90de-ffc15074a50f","Type":"ContainerStarted","Data":"81aa272b24db3e2aac8307e85ec1c538754b7333cddcdf89cdfbbdaaf0382d0e"} Nov 04 03:10:21.302783 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:10:21.302747 4405 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-4cbcf738-openshift-default-68fb497c9c-2ndqr" event={"ID":"a77157ba-0aed-46cf-90de-ffc15074a50f","Type":"ContainerStarted","Data":"e57f052b1c1e308e3db04076ce75ab0841ad6d56803824fa0ad5b0ee0a050b4b"} Nov 04 03:10:21.476550 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:10:21.476495 4405 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-4cbcf738-openshift-default-68fb497c9c-2ndqr" podStartSLOduration=2.476480282 podStartE2EDuration="2.476480282s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 03:10:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 03:10:21.474817216 +0000 UTC m=+2267.905517347" watchObservedRunningTime="2026-11-04 03:10:21.476480282 +0000 UTC m=+2267.907180403" Nov 04 03:10:21.562396 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:10:21.562299 4405 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-bl5mg"] Nov 04 03:10:22.163094 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:10:22.163062 4405 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-4cbcf738-openshift-default-68fb497c9c-2ndqr" Nov 04 03:10:22.169120 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:10:22.169093 4405 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-4cbcf738-openshift-default-68fb497c9c-2ndqr" Nov 04 03:10:22.350726 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:10:22.342190 4405 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-4cbcf738-openshift-default-68fb497c9c-2ndqr" Nov 04 03:10:22.350726 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:10:22.345331 4405 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-4cbcf738-openshift-default-68fb497c9c-2ndqr" Nov 04 03:10:26.536169 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:10:26.536136 4405 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-3b7476-openshift-default-59975457bf-msl8m"] Nov 04 03:10:26.536620 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:10:26.536345 4405 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w5-3b7476-openshift-default-59975457bf-msl8m" podUID="8a0008f3-4ab9-4ceb-8feb-0f536776610a" containerName="istio-proxy" containerID="cri-o://a0089c3a7587fb582b2f368dfc4276ef22829f9419fca354c13b5d981a281fb0" gracePeriod=30 Nov 04 03:10:26.550246 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:10:26.550216 4405 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-3b7476-openshift-default-59975457bf-q7gfm"] Nov 04 03:10:26.550424 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:10:26.550404 4405 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w5-3b7476-openshift-default-59975457bf-q7gfm" podUID="e6cb6771-a9e6-452f-a0b6-30ffb955feb8" containerName="istio-proxy" containerID="cri-o://4b1aee4d81d9c94a20df3fd373faab5077813f49eb41d3fc655645a2cc12c416" gracePeriod=30 Nov 04 03:10:27.927553 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:10:27.927512 4405 patch_prober.go:29] interesting pod/e2e-worker-w5-3b7476-openshift-default-59975457bf-msl8m container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 04 03:10:27.928075 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:10:27.927576 4405 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w5-3b7476-openshift-default-59975457bf-msl8m" podUID="8a0008f3-4ab9-4ceb-8feb-0f536776610a" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 04 03:10:29.084258 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:10:29.084221 4405 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-4cbcf738-67bd6b6f59-mfssb"] Nov 04 03:10:29.084854 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:10:29.084838 4405 cpu_manager.go:414] "containerMap: removing container" podUID="fd2b4daf-d65f-44ca-8939-67778fe8ef5a" containerName="istio-proxy" Nov 04 03:10:29.084914 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:10:29.084855 4405 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fd2b4daf-d65f-44ca-8939-67778fe8ef5a" containerName="istio-proxy" Nov 04 03:10:29.088195 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:10:29.088176 4405 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-4cbcf738-67bd6b6f59-mfssb" Nov 04 03:10:29.193148 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:10:29.193110 4405 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-4cbcf738\"" Nov 04 03:10:29.193386 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:10:29.193364 4405 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-mt-4cbcf738-dockercfg-pnsst\"" Nov 04 03:10:29.196528 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:10:29.196502 4405 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-4cbcf738-67bd6b6f59-mfssb"] Nov 04 03:10:29.214797 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:10:29.214771 4405 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-szbfw\" (UniqueName: \"kubernetes.io/projected/8b2f727a-24b4-4768-81c5-4ee22b3b5f26-kube-api-access-szbfw\") pod \"payload-pre-processing-e2e-mt-4cbcf738-67bd6b6f59-mfssb\" (UID: \"8b2f727a-24b4-4768-81c5-4ee22b3b5f26\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-4cbcf738-67bd6b6f59-mfssb" Nov 04 03:10:29.214906 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:10:29.214861 4405 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8b2f727a-24b4-4768-81c5-4ee22b3b5f26-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-4cbcf738-67bd6b6f59-mfssb\" (UID: \"8b2f727a-24b4-4768-81c5-4ee22b3b5f26\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-4cbcf738-67bd6b6f59-mfssb" Nov 04 03:10:29.316589 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:10:29.316548 4405 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-szbfw\" (UniqueName: \"kubernetes.io/projected/8b2f727a-24b4-4768-81c5-4ee22b3b5f26-kube-api-access-szbfw\") pod \"payload-pre-processing-e2e-mt-4cbcf738-67bd6b6f59-mfssb\" (UID: \"8b2f727a-24b4-4768-81c5-4ee22b3b5f26\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-4cbcf738-67bd6b6f59-mfssb" Nov 04 03:10:29.316795 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:10:29.316642 4405 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8b2f727a-24b4-4768-81c5-4ee22b3b5f26-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-4cbcf738-67bd6b6f59-mfssb\" (UID: \"8b2f727a-24b4-4768-81c5-4ee22b3b5f26\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-4cbcf738-67bd6b6f59-mfssb" Nov 04 03:10:29.317470 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:10:29.317451 4405 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8b2f727a-24b4-4768-81c5-4ee22b3b5f26-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-4cbcf738-67bd6b6f59-mfssb\" (UID: \"8b2f727a-24b4-4768-81c5-4ee22b3b5f26\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-4cbcf738-67bd6b6f59-mfssb" Nov 04 03:10:29.378605 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:10:29.378527 4405 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-szbfw\" (UniqueName: \"kubernetes.io/projected/8b2f727a-24b4-4768-81c5-4ee22b3b5f26-kube-api-access-szbfw\") pod \"payload-pre-processing-e2e-mt-4cbcf738-67bd6b6f59-mfssb\" (UID: \"8b2f727a-24b4-4768-81c5-4ee22b3b5f26\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-4cbcf738-67bd6b6f59-mfssb" Nov 04 03:10:29.405209 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:10:29.405178 4405 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-4cbcf738-67bd6b6f59-mfssb" Nov 04 03:10:29.784739 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:10:29.784705 4405 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-4cbcf738-67bd6b6f59-mfssb"] Nov 04 03:10:29.785858 ip-10-0-134-125 kubenswrapper[4405]: W1104 03:10:29.785828 4405 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8b2f727a_24b4_4768_81c5_4ee22b3b5f26.slice/crio-c183efe5ade9477d7d4812ce256d8cd28f0fe93bb1b34b9baf1b6d46fdca3efb WatchSource:0}: Error finding container c183efe5ade9477d7d4812ce256d8cd28f0fe93bb1b34b9baf1b6d46fdca3efb: Status 404 returned error can't find the container with id c183efe5ade9477d7d4812ce256d8cd28f0fe93bb1b34b9baf1b6d46fdca3efb Nov 04 03:10:30.390480 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:10:30.390441 4405 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-4cbcf738-67bd6b6f59-mfssb" event={"ID":"8b2f727a-24b4-4768-81c5-4ee22b3b5f26","Type":"ContainerStarted","Data":"81f9bc01d27cc3812c541659169ce05bc5d3299d4c26694bc621a3a2b120a8bf"} Nov 04 03:10:30.390956 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:10:30.390536 4405 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-4cbcf738-67bd6b6f59-mfssb" event={"ID":"8b2f727a-24b4-4768-81c5-4ee22b3b5f26","Type":"ContainerStarted","Data":"c183efe5ade9477d7d4812ce256d8cd28f0fe93bb1b34b9baf1b6d46fdca3efb"} Nov 04 03:10:30.390956 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:10:30.390696 4405 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-4cbcf738-67bd6b6f59-mfssb" Nov 04 03:10:31.855392 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:10:31.855364 4405 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-3b7476-openshift-default-59975457bf-msl8m" Nov 04 03:10:31.871456 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:10:31.871431 4405 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-3b7476-openshift-default-59975457bf-q7gfm" Nov 04 03:10:31.959442 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:10:31.959411 4405 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8a0008f3-4ab9-4ceb-8feb-0f536776610a-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8a0008f3-4ab9-4ceb-8feb-0f536776610a-istiod-ca-cert\") pod \"8a0008f3-4ab9-4ceb-8feb-0f536776610a\" (UID: \"8a0008f3-4ab9-4ceb-8feb-0f536776610a\") " Nov 04 03:10:31.959619 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:10:31.959448 4405 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e6cb6771-a9e6-452f-a0b6-30ffb955feb8-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e6cb6771-a9e6-452f-a0b6-30ffb955feb8-credential-socket\") pod \"e6cb6771-a9e6-452f-a0b6-30ffb955feb8\" (UID: \"e6cb6771-a9e6-452f-a0b6-30ffb955feb8\") " Nov 04 03:10:31.959619 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:10:31.959478 4405 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e6cb6771-a9e6-452f-a0b6-30ffb955feb8-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e6cb6771-a9e6-452f-a0b6-30ffb955feb8-workload-certs\") pod \"e6cb6771-a9e6-452f-a0b6-30ffb955feb8\" (UID: \"e6cb6771-a9e6-452f-a0b6-30ffb955feb8\") " Nov 04 03:10:31.959619 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:10:31.959532 4405 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e6cb6771-a9e6-452f-a0b6-30ffb955feb8-istio-token\" (UniqueName: \"kubernetes.io/projected/e6cb6771-a9e6-452f-a0b6-30ffb955feb8-istio-token\") pod \"e6cb6771-a9e6-452f-a0b6-30ffb955feb8\" (UID: \"e6cb6771-a9e6-452f-a0b6-30ffb955feb8\") " Nov 04 03:10:31.959619 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:10:31.959581 4405 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e6cb6771-a9e6-452f-a0b6-30ffb955feb8-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e6cb6771-a9e6-452f-a0b6-30ffb955feb8-istiod-ca-cert\") pod \"e6cb6771-a9e6-452f-a0b6-30ffb955feb8\" (UID: \"e6cb6771-a9e6-452f-a0b6-30ffb955feb8\") " Nov 04 03:10:31.959856 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:10:31.959641 4405 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8a0008f3-4ab9-4ceb-8feb-0f536776610a-kube-api-access-74xm5\" (UniqueName: \"kubernetes.io/projected/8a0008f3-4ab9-4ceb-8feb-0f536776610a-kube-api-access-74xm5\") pod \"8a0008f3-4ab9-4ceb-8feb-0f536776610a\" (UID: \"8a0008f3-4ab9-4ceb-8feb-0f536776610a\") " Nov 04 03:10:31.959856 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:10:31.959714 4405 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e6cb6771-a9e6-452f-a0b6-30ffb955feb8-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e6cb6771-a9e6-452f-a0b6-30ffb955feb8-istio-data\") pod \"e6cb6771-a9e6-452f-a0b6-30ffb955feb8\" (UID: \"e6cb6771-a9e6-452f-a0b6-30ffb955feb8\") " Nov 04 03:10:31.959856 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:10:31.959763 4405 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e6cb6771-a9e6-452f-a0b6-30ffb955feb8-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e6cb6771-a9e6-452f-a0b6-30ffb955feb8-workload-socket\") pod \"e6cb6771-a9e6-452f-a0b6-30ffb955feb8\" (UID: \"e6cb6771-a9e6-452f-a0b6-30ffb955feb8\") " Nov 04 03:10:31.959856 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:10:31.959792 4405 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/8a0008f3-4ab9-4ceb-8feb-0f536776610a-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8a0008f3-4ab9-4ceb-8feb-0f536776610a-istio-podinfo\") pod \"8a0008f3-4ab9-4ceb-8feb-0f536776610a\" (UID: \"8a0008f3-4ab9-4ceb-8feb-0f536776610a\") " Nov 04 03:10:31.959856 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:10:31.959798 4405 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e6cb6771-a9e6-452f-a0b6-30ffb955feb8-credential-socket" pod "e6cb6771-a9e6-452f-a0b6-30ffb955feb8" (UID: "e6cb6771-a9e6-452f-a0b6-30ffb955feb8"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 03:10:31.959856 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:10:31.959832 4405 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e6cb6771-a9e6-452f-a0b6-30ffb955feb8-kube-api-access-gxct2\" (UniqueName: \"kubernetes.io/projected/e6cb6771-a9e6-452f-a0b6-30ffb955feb8-kube-api-access-gxct2\") pod \"e6cb6771-a9e6-452f-a0b6-30ffb955feb8\" (UID: \"e6cb6771-a9e6-452f-a0b6-30ffb955feb8\") " Nov 04 03:10:31.960194 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:10:31.959861 4405 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e6cb6771-a9e6-452f-a0b6-30ffb955feb8-workload-certs" pod "e6cb6771-a9e6-452f-a0b6-30ffb955feb8" (UID: "e6cb6771-a9e6-452f-a0b6-30ffb955feb8"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 03:10:31.960194 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:10:31.959899 4405 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/e6cb6771-a9e6-452f-a0b6-30ffb955feb8-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e6cb6771-a9e6-452f-a0b6-30ffb955feb8-istio-podinfo\") pod \"e6cb6771-a9e6-452f-a0b6-30ffb955feb8\" (UID: \"e6cb6771-a9e6-452f-a0b6-30ffb955feb8\") " Nov 04 03:10:31.960194 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:10:31.959901 4405 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8a0008f3-4ab9-4ceb-8feb-0f536776610a-istiod-ca-cert" pod "8a0008f3-4ab9-4ceb-8feb-0f536776610a" (UID: "8a0008f3-4ab9-4ceb-8feb-0f536776610a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 03:10:31.960194 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:10:31.959930 4405 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8a0008f3-4ab9-4ceb-8feb-0f536776610a-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8a0008f3-4ab9-4ceb-8feb-0f536776610a-istio-envoy\") pod \"8a0008f3-4ab9-4ceb-8feb-0f536776610a\" (UID: \"8a0008f3-4ab9-4ceb-8feb-0f536776610a\") " Nov 04 03:10:31.960194 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:10:31.960016 4405 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8a0008f3-4ab9-4ceb-8feb-0f536776610a-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8a0008f3-4ab9-4ceb-8feb-0f536776610a-credential-socket\") pod \"8a0008f3-4ab9-4ceb-8feb-0f536776610a\" (UID: \"8a0008f3-4ab9-4ceb-8feb-0f536776610a\") " Nov 04 03:10:31.960194 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:10:31.960042 4405 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e6cb6771-a9e6-452f-a0b6-30ffb955feb8-workload-socket" pod "e6cb6771-a9e6-452f-a0b6-30ffb955feb8" (UID: "e6cb6771-a9e6-452f-a0b6-30ffb955feb8"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 03:10:31.960194 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:10:31.960066 4405 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e6cb6771-a9e6-452f-a0b6-30ffb955feb8-istiod-ca-cert" pod "e6cb6771-a9e6-452f-a0b6-30ffb955feb8" (UID: "e6cb6771-a9e6-452f-a0b6-30ffb955feb8"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 03:10:31.960194 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:10:31.960074 4405 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8a0008f3-4ab9-4ceb-8feb-0f536776610a-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8a0008f3-4ab9-4ceb-8feb-0f536776610a-workload-certs\") pod \"8a0008f3-4ab9-4ceb-8feb-0f536776610a\" (UID: \"8a0008f3-4ab9-4ceb-8feb-0f536776610a\") " Nov 04 03:10:31.960194 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:10:31.960107 4405 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e6cb6771-a9e6-452f-a0b6-30ffb955feb8-istio-data" pod "e6cb6771-a9e6-452f-a0b6-30ffb955feb8" (UID: "e6cb6771-a9e6-452f-a0b6-30ffb955feb8"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 03:10:31.960194 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:10:31.960160 4405 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8a0008f3-4ab9-4ceb-8feb-0f536776610a-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8a0008f3-4ab9-4ceb-8feb-0f536776610a-istio-data\") pod \"8a0008f3-4ab9-4ceb-8feb-0f536776610a\" (UID: \"8a0008f3-4ab9-4ceb-8feb-0f536776610a\") " Nov 04 03:10:31.960895 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:10:31.960217 4405 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8a0008f3-4ab9-4ceb-8feb-0f536776610a-istio-token\" (UniqueName: \"kubernetes.io/projected/8a0008f3-4ab9-4ceb-8feb-0f536776610a-istio-token\") pod \"8a0008f3-4ab9-4ceb-8feb-0f536776610a\" (UID: \"8a0008f3-4ab9-4ceb-8feb-0f536776610a\") " Nov 04 03:10:31.960895 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:10:31.960265 4405 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8a0008f3-4ab9-4ceb-8feb-0f536776610a-workload-certs" pod "8a0008f3-4ab9-4ceb-8feb-0f536776610a" (UID: "8a0008f3-4ab9-4ceb-8feb-0f536776610a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 03:10:31.960895 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:10:31.960279 4405 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8a0008f3-4ab9-4ceb-8feb-0f536776610a-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8a0008f3-4ab9-4ceb-8feb-0f536776610a-workload-socket\") pod \"8a0008f3-4ab9-4ceb-8feb-0f536776610a\" (UID: \"8a0008f3-4ab9-4ceb-8feb-0f536776610a\") " Nov 04 03:10:31.960895 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:10:31.960335 4405 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e6cb6771-a9e6-452f-a0b6-30ffb955feb8-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e6cb6771-a9e6-452f-a0b6-30ffb955feb8-istio-envoy\") pod \"e6cb6771-a9e6-452f-a0b6-30ffb955feb8\" (UID: \"e6cb6771-a9e6-452f-a0b6-30ffb955feb8\") " Nov 04 03:10:31.960895 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:10:31.960394 4405 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8a0008f3-4ab9-4ceb-8feb-0f536776610a-credential-socket" pod "8a0008f3-4ab9-4ceb-8feb-0f536776610a" (UID: "8a0008f3-4ab9-4ceb-8feb-0f536776610a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 03:10:31.960895 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:10:31.960636 4405 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8a0008f3-4ab9-4ceb-8feb-0f536776610a-workload-socket" pod "8a0008f3-4ab9-4ceb-8feb-0f536776610a" (UID: "8a0008f3-4ab9-4ceb-8feb-0f536776610a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 03:10:31.961227 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:10:31.961054 4405 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e6cb6771-a9e6-452f-a0b6-30ffb955feb8-istio-data\") on node \"ip-10-0-134-125.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:10:31.961227 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:10:31.961081 4405 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e6cb6771-a9e6-452f-a0b6-30ffb955feb8-workload-socket\") on node \"ip-10-0-134-125.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:10:31.961227 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:10:31.961098 4405 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8a0008f3-4ab9-4ceb-8feb-0f536776610a-credential-socket\") on node \"ip-10-0-134-125.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:10:31.961227 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:10:31.961115 4405 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8a0008f3-4ab9-4ceb-8feb-0f536776610a-workload-certs\") on node \"ip-10-0-134-125.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:10:31.961227 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:10:31.961130 4405 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8a0008f3-4ab9-4ceb-8feb-0f536776610a-workload-socket\") on node \"ip-10-0-134-125.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:10:31.961227 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:10:31.961144 4405 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8a0008f3-4ab9-4ceb-8feb-0f536776610a-istiod-ca-cert\") on node \"ip-10-0-134-125.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:10:31.961227 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:10:31.961158 4405 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e6cb6771-a9e6-452f-a0b6-30ffb955feb8-credential-socket\") on node \"ip-10-0-134-125.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:10:31.961227 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:10:31.961174 4405 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e6cb6771-a9e6-452f-a0b6-30ffb955feb8-workload-certs\") on node \"ip-10-0-134-125.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:10:31.961227 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:10:31.961188 4405 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e6cb6771-a9e6-452f-a0b6-30ffb955feb8-istiod-ca-cert\") on node \"ip-10-0-134-125.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:10:31.961869 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:10:31.961369 4405 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8a0008f3-4ab9-4ceb-8feb-0f536776610a-istio-data" pod "8a0008f3-4ab9-4ceb-8feb-0f536776610a" (UID: "8a0008f3-4ab9-4ceb-8feb-0f536776610a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 03:10:31.962801 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:10:31.962772 4405 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8a0008f3-4ab9-4ceb-8feb-0f536776610a-istio-envoy" pod "8a0008f3-4ab9-4ceb-8feb-0f536776610a" (UID: "8a0008f3-4ab9-4ceb-8feb-0f536776610a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 03:10:31.962922 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:10:31.962801 4405 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8a0008f3-4ab9-4ceb-8feb-0f536776610a-kube-api-access-74xm5" pod "8a0008f3-4ab9-4ceb-8feb-0f536776610a" (UID: "8a0008f3-4ab9-4ceb-8feb-0f536776610a"). InnerVolumeSpecName "kube-api-access-74xm5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 03:10:31.963131 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:10:31.963096 4405 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e6cb6771-a9e6-452f-a0b6-30ffb955feb8-istio-podinfo" pod "e6cb6771-a9e6-452f-a0b6-30ffb955feb8" (UID: "e6cb6771-a9e6-452f-a0b6-30ffb955feb8"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 04 03:10:31.963255 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:10:31.963172 4405 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/8a0008f3-4ab9-4ceb-8feb-0f536776610a-istio-podinfo" pod "8a0008f3-4ab9-4ceb-8feb-0f536776610a" (UID: "8a0008f3-4ab9-4ceb-8feb-0f536776610a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 04 03:10:31.963255 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:10:31.963225 4405 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e6cb6771-a9e6-452f-a0b6-30ffb955feb8-kube-api-access-gxct2" pod "e6cb6771-a9e6-452f-a0b6-30ffb955feb8" (UID: "e6cb6771-a9e6-452f-a0b6-30ffb955feb8"). InnerVolumeSpecName "kube-api-access-gxct2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 03:10:31.963725 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:10:31.963659 4405 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e6cb6771-a9e6-452f-a0b6-30ffb955feb8-istio-token" pod "e6cb6771-a9e6-452f-a0b6-30ffb955feb8" (UID: "e6cb6771-a9e6-452f-a0b6-30ffb955feb8"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 03:10:31.964197 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:10:31.964176 4405 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e6cb6771-a9e6-452f-a0b6-30ffb955feb8-istio-envoy" pod "e6cb6771-a9e6-452f-a0b6-30ffb955feb8" (UID: "e6cb6771-a9e6-452f-a0b6-30ffb955feb8"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 03:10:31.964623 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:10:31.964599 4405 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8a0008f3-4ab9-4ceb-8feb-0f536776610a-istio-token" pod "8a0008f3-4ab9-4ceb-8feb-0f536776610a" (UID: "8a0008f3-4ab9-4ceb-8feb-0f536776610a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 03:10:32.062875 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:10:32.062838 4405 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8a0008f3-4ab9-4ceb-8feb-0f536776610a-istio-data\") on node \"ip-10-0-134-125.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:10:32.062875 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:10:32.062876 4405 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8a0008f3-4ab9-4ceb-8feb-0f536776610a-istio-token\") on node \"ip-10-0-134-125.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:10:32.063095 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:10:32.062890 4405 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e6cb6771-a9e6-452f-a0b6-30ffb955feb8-istio-envoy\") on node \"ip-10-0-134-125.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:10:32.063095 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:10:32.062903 4405 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e6cb6771-a9e6-452f-a0b6-30ffb955feb8-istio-token\") on node \"ip-10-0-134-125.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:10:32.063095 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:10:32.062918 4405 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-74xm5\" (UniqueName: \"kubernetes.io/projected/8a0008f3-4ab9-4ceb-8feb-0f536776610a-kube-api-access-74xm5\") on node \"ip-10-0-134-125.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:10:32.063095 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:10:32.062931 4405 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8a0008f3-4ab9-4ceb-8feb-0f536776610a-istio-podinfo\") on node \"ip-10-0-134-125.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:10:32.063095 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:10:32.062944 4405 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gxct2\" (UniqueName: \"kubernetes.io/projected/e6cb6771-a9e6-452f-a0b6-30ffb955feb8-kube-api-access-gxct2\") on node \"ip-10-0-134-125.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:10:32.063095 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:10:32.062956 4405 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e6cb6771-a9e6-452f-a0b6-30ffb955feb8-istio-podinfo\") on node \"ip-10-0-134-125.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:10:32.063095 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:10:32.062968 4405 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8a0008f3-4ab9-4ceb-8feb-0f536776610a-istio-envoy\") on node \"ip-10-0-134-125.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:10:32.068395 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:10:32.068343 4405 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-mt-4cbcf738-67bd6b6f59-mfssb" podStartSLOduration=3.068327349 podStartE2EDuration="3.068327349s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 03:10:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 03:10:30.511228505 +0000 UTC m=+2276.941928625" watchObservedRunningTime="2026-11-04 03:10:32.068327349 +0000 UTC m=+2278.499027469" Nov 04 03:10:32.406217 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:10:32.406178 4405 generic.go:356] "Generic (PLEG): container finished" podID="e6cb6771-a9e6-452f-a0b6-30ffb955feb8" containerID="4b1aee4d81d9c94a20df3fd373faab5077813f49eb41d3fc655645a2cc12c416" exitCode=0 Nov 04 03:10:32.406429 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:10:32.406229 4405 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-3b7476-openshift-default-59975457bf-q7gfm" Nov 04 03:10:32.406429 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:10:32.406262 4405 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-3b7476-openshift-default-59975457bf-q7gfm" event={"ID":"e6cb6771-a9e6-452f-a0b6-30ffb955feb8","Type":"ContainerDied","Data":"4b1aee4d81d9c94a20df3fd373faab5077813f49eb41d3fc655645a2cc12c416"} Nov 04 03:10:32.406429 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:10:32.406324 4405 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-3b7476-openshift-default-59975457bf-q7gfm" event={"ID":"e6cb6771-a9e6-452f-a0b6-30ffb955feb8","Type":"ContainerDied","Data":"ee2ba2f6c28f1cde4528f0e550ae387746e5fd964bd85dc6660bb7d34000c1cf"} Nov 04 03:10:32.406429 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:10:32.406356 4405 scope.go:122] "RemoveContainer" containerID="4b1aee4d81d9c94a20df3fd373faab5077813f49eb41d3fc655645a2cc12c416" Nov 04 03:10:32.408356 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:10:32.408332 4405 generic.go:356] "Generic (PLEG): container finished" podID="8a0008f3-4ab9-4ceb-8feb-0f536776610a" containerID="a0089c3a7587fb582b2f368dfc4276ef22829f9419fca354c13b5d981a281fb0" exitCode=0 Nov 04 03:10:32.408473 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:10:32.408374 4405 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-3b7476-openshift-default-59975457bf-msl8m" event={"ID":"8a0008f3-4ab9-4ceb-8feb-0f536776610a","Type":"ContainerDied","Data":"a0089c3a7587fb582b2f368dfc4276ef22829f9419fca354c13b5d981a281fb0"} Nov 04 03:10:32.408473 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:10:32.408401 4405 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-3b7476-openshift-default-59975457bf-msl8m" Nov 04 03:10:32.408473 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:10:32.408417 4405 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-3b7476-openshift-default-59975457bf-msl8m" event={"ID":"8a0008f3-4ab9-4ceb-8feb-0f536776610a","Type":"ContainerDied","Data":"d58b6d456cfb5364ee59eadf97dfb1e570168cc90911c0197bb520fb34e2cfbb"} Nov 04 03:10:32.423281 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:10:32.423062 4405 scope.go:122] "RemoveContainer" containerID="4b1aee4d81d9c94a20df3fd373faab5077813f49eb41d3fc655645a2cc12c416" Nov 04 03:10:32.423493 ip-10-0-134-125 kubenswrapper[4405]: E1104 03:10:32.423472 4405 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4b1aee4d81d9c94a20df3fd373faab5077813f49eb41d3fc655645a2cc12c416\": container with ID starting with 4b1aee4d81d9c94a20df3fd373faab5077813f49eb41d3fc655645a2cc12c416 not found: ID does not exist" containerID="4b1aee4d81d9c94a20df3fd373faab5077813f49eb41d3fc655645a2cc12c416" Nov 04 03:10:32.423564 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:10:32.423502 4405 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4b1aee4d81d9c94a20df3fd373faab5077813f49eb41d3fc655645a2cc12c416"} err="failed to get container status \"4b1aee4d81d9c94a20df3fd373faab5077813f49eb41d3fc655645a2cc12c416\": rpc error: code = NotFound desc = could not find container \"4b1aee4d81d9c94a20df3fd373faab5077813f49eb41d3fc655645a2cc12c416\": container with ID starting with 4b1aee4d81d9c94a20df3fd373faab5077813f49eb41d3fc655645a2cc12c416 not found: ID does not exist" Nov 04 03:10:32.423564 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:10:32.423527 4405 scope.go:122] "RemoveContainer" containerID="a0089c3a7587fb582b2f368dfc4276ef22829f9419fca354c13b5d981a281fb0" Nov 04 03:10:32.434507 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:10:32.434487 4405 scope.go:122] "RemoveContainer" containerID="b6a2edf16fc64d8d3cfeceaa4932a3db1cdd4fcbfee7de8866224fa041a79b0b" Nov 04 03:10:32.444200 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:10:32.444177 4405 scope.go:122] "RemoveContainer" containerID="a0089c3a7587fb582b2f368dfc4276ef22829f9419fca354c13b5d981a281fb0" Nov 04 03:10:32.444497 ip-10-0-134-125 kubenswrapper[4405]: E1104 03:10:32.444473 4405 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a0089c3a7587fb582b2f368dfc4276ef22829f9419fca354c13b5d981a281fb0\": container with ID starting with a0089c3a7587fb582b2f368dfc4276ef22829f9419fca354c13b5d981a281fb0 not found: ID does not exist" containerID="a0089c3a7587fb582b2f368dfc4276ef22829f9419fca354c13b5d981a281fb0" Nov 04 03:10:32.444551 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:10:32.444507 4405 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a0089c3a7587fb582b2f368dfc4276ef22829f9419fca354c13b5d981a281fb0"} err="failed to get container status \"a0089c3a7587fb582b2f368dfc4276ef22829f9419fca354c13b5d981a281fb0\": rpc error: code = NotFound desc = could not find container \"a0089c3a7587fb582b2f368dfc4276ef22829f9419fca354c13b5d981a281fb0\": container with ID starting with a0089c3a7587fb582b2f368dfc4276ef22829f9419fca354c13b5d981a281fb0 not found: ID does not exist" Nov 04 03:10:32.444551 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:10:32.444528 4405 scope.go:122] "RemoveContainer" containerID="b6a2edf16fc64d8d3cfeceaa4932a3db1cdd4fcbfee7de8866224fa041a79b0b" Nov 04 03:10:32.444874 ip-10-0-134-125 kubenswrapper[4405]: E1104 03:10:32.444851 4405 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b6a2edf16fc64d8d3cfeceaa4932a3db1cdd4fcbfee7de8866224fa041a79b0b\": container with ID starting with b6a2edf16fc64d8d3cfeceaa4932a3db1cdd4fcbfee7de8866224fa041a79b0b not found: ID does not exist" containerID="b6a2edf16fc64d8d3cfeceaa4932a3db1cdd4fcbfee7de8866224fa041a79b0b" Nov 04 03:10:32.444925 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:10:32.444879 4405 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b6a2edf16fc64d8d3cfeceaa4932a3db1cdd4fcbfee7de8866224fa041a79b0b"} err="failed to get container status \"b6a2edf16fc64d8d3cfeceaa4932a3db1cdd4fcbfee7de8866224fa041a79b0b\": rpc error: code = NotFound desc = could not find container \"b6a2edf16fc64d8d3cfeceaa4932a3db1cdd4fcbfee7de8866224fa041a79b0b\": container with ID starting with b6a2edf16fc64d8d3cfeceaa4932a3db1cdd4fcbfee7de8866224fa041a79b0b not found: ID does not exist" Nov 04 03:10:32.574014 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:10:32.573975 4405 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-3b7476-openshift-default-59975457bf-msl8m"] Nov 04 03:10:32.621965 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:10:32.621923 4405 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w5-3b7476-openshift-default-59975457bf-msl8m"] Nov 04 03:10:32.685333 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:10:32.685246 4405 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-3b7476-openshift-default-59975457bf-q7gfm"] Nov 04 03:10:32.690656 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:10:32.690625 4405 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w5-3b7476-openshift-default-59975457bf-q7gfm"] Nov 04 03:10:33.706004 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:10:33.705968 4405 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8a0008f3-4ab9-4ceb-8feb-0f536776610a" path="/var/lib/kubelet/pods/8a0008f3-4ab9-4ceb-8feb-0f536776610a/volumes" Nov 04 03:10:33.706623 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:10:33.706601 4405 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e6cb6771-a9e6-452f-a0b6-30ffb955feb8" path="/var/lib/kubelet/pods/e6cb6771-a9e6-452f-a0b6-30ffb955feb8/volumes" Nov 04 03:10:34.567208 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:10:34.567176 4405 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-9f4199-85bd8d95f6-6k95j"] Nov 04 03:10:34.567418 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:10:34.567397 4405 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-9f4199-85bd8d95f6-6k95j" podUID="97c488a6-ee10-46bf-bc00-ef8578040f88" containerName="maas-api" containerID="cri-o://f71dd36502a0c52e983365f478eca1a00b26224c4ced97a4b3cb302f0660d9ac" gracePeriod=30 Nov 04 03:10:34.697153 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:10:34.697112 4405 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-9f4199-85bd8d95f6-6k95j" podUID="97c488a6-ee10-46bf-bc00-ef8578040f88" containerName="maas-api" probeResult="failure" output="Get \"https://10.134.0.123:8443/health\": dial tcp 10.134.0.123:8443: connect: connection refused" Nov 04 03:10:34.784185 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:10:34.783849 4405 scope.go:122] "RemoveContainer" containerID="34dd768ead4d27389133974c1b192d6e5118c0022b823eea38e38136c1babd2a" Nov 04 03:10:34.907609 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:10:34.907592 4405 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-9f4199-85bd8d95f6-6k95j" Nov 04 03:10:35.013403 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:10:35.013371 4405 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/97c488a6-ee10-46bf-bc00-ef8578040f88-kube-api-access-dkmvj\" (UniqueName: \"kubernetes.io/projected/97c488a6-ee10-46bf-bc00-ef8578040f88-kube-api-access-dkmvj\") pod \"97c488a6-ee10-46bf-bc00-ef8578040f88\" (UID: \"97c488a6-ee10-46bf-bc00-ef8578040f88\") " Nov 04 03:10:35.013625 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:10:35.013491 4405 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/97c488a6-ee10-46bf-bc00-ef8578040f88-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/97c488a6-ee10-46bf-bc00-ef8578040f88-openshift-service-ca\") pod \"97c488a6-ee10-46bf-bc00-ef8578040f88\" (UID: \"97c488a6-ee10-46bf-bc00-ef8578040f88\") " Nov 04 03:10:35.013625 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:10:35.013538 4405 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/97c488a6-ee10-46bf-bc00-ef8578040f88-metrics-certs\" (UniqueName: \"kubernetes.io/secret/97c488a6-ee10-46bf-bc00-ef8578040f88-metrics-certs\") pod \"97c488a6-ee10-46bf-bc00-ef8578040f88\" (UID: \"97c488a6-ee10-46bf-bc00-ef8578040f88\") " Nov 04 03:10:35.013890 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:10:35.013659 4405 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/97c488a6-ee10-46bf-bc00-ef8578040f88-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/97c488a6-ee10-46bf-bc00-ef8578040f88-maas-api-tls\") pod \"97c488a6-ee10-46bf-bc00-ef8578040f88\" (UID: \"97c488a6-ee10-46bf-bc00-ef8578040f88\") " Nov 04 03:10:35.014389 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:10:35.014347 4405 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/97c488a6-ee10-46bf-bc00-ef8578040f88-openshift-service-ca" pod "97c488a6-ee10-46bf-bc00-ef8578040f88" (UID: "97c488a6-ee10-46bf-bc00-ef8578040f88"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 03:10:35.017537 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:10:35.017509 4405 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/97c488a6-ee10-46bf-bc00-ef8578040f88-kube-api-access-dkmvj" pod "97c488a6-ee10-46bf-bc00-ef8578040f88" (UID: "97c488a6-ee10-46bf-bc00-ef8578040f88"). InnerVolumeSpecName "kube-api-access-dkmvj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 03:10:35.017837 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:10:35.017802 4405 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/97c488a6-ee10-46bf-bc00-ef8578040f88-metrics-certs" pod "97c488a6-ee10-46bf-bc00-ef8578040f88" (UID: "97c488a6-ee10-46bf-bc00-ef8578040f88"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 04 03:10:35.018079 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:10:35.018051 4405 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/97c488a6-ee10-46bf-bc00-ef8578040f88-maas-api-tls" pod "97c488a6-ee10-46bf-bc00-ef8578040f88" (UID: "97c488a6-ee10-46bf-bc00-ef8578040f88"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 04 03:10:35.115578 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:10:35.115545 4405 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dkmvj\" (UniqueName: \"kubernetes.io/projected/97c488a6-ee10-46bf-bc00-ef8578040f88-kube-api-access-dkmvj\") on node \"ip-10-0-134-125.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:10:35.115578 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:10:35.115574 4405 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/97c488a6-ee10-46bf-bc00-ef8578040f88-openshift-service-ca\") on node \"ip-10-0-134-125.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:10:35.115578 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:10:35.115586 4405 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/97c488a6-ee10-46bf-bc00-ef8578040f88-metrics-certs\") on node \"ip-10-0-134-125.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:10:35.115905 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:10:35.115604 4405 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/97c488a6-ee10-46bf-bc00-ef8578040f88-maas-api-tls\") on node \"ip-10-0-134-125.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:10:35.435916 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:10:35.435825 4405 generic.go:356] "Generic (PLEG): container finished" podID="97c488a6-ee10-46bf-bc00-ef8578040f88" containerID="f71dd36502a0c52e983365f478eca1a00b26224c4ced97a4b3cb302f0660d9ac" exitCode=0 Nov 04 03:10:35.435916 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:10:35.435902 4405 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-9f4199-85bd8d95f6-6k95j" Nov 04 03:10:35.436152 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:10:35.435913 4405 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-9f4199-85bd8d95f6-6k95j" event={"ID":"97c488a6-ee10-46bf-bc00-ef8578040f88","Type":"ContainerDied","Data":"f71dd36502a0c52e983365f478eca1a00b26224c4ced97a4b3cb302f0660d9ac"} Nov 04 03:10:35.436152 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:10:35.436010 4405 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-9f4199-85bd8d95f6-6k95j" event={"ID":"97c488a6-ee10-46bf-bc00-ef8578040f88","Type":"ContainerDied","Data":"1d6eae8cb911e0e1ff1c3ba64c47be1e99e048cea70937abe27e8a96680ce7de"} Nov 04 03:10:35.436152 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:10:35.436045 4405 scope.go:122] "RemoveContainer" containerID="f71dd36502a0c52e983365f478eca1a00b26224c4ced97a4b3cb302f0660d9ac" Nov 04 03:10:35.446589 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:10:35.446573 4405 scope.go:122] "RemoveContainer" containerID="f71dd36502a0c52e983365f478eca1a00b26224c4ced97a4b3cb302f0660d9ac" Nov 04 03:10:35.446920 ip-10-0-134-125 kubenswrapper[4405]: E1104 03:10:35.446901 4405 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f71dd36502a0c52e983365f478eca1a00b26224c4ced97a4b3cb302f0660d9ac\": container with ID starting with f71dd36502a0c52e983365f478eca1a00b26224c4ced97a4b3cb302f0660d9ac not found: ID does not exist" containerID="f71dd36502a0c52e983365f478eca1a00b26224c4ced97a4b3cb302f0660d9ac" Nov 04 03:10:35.446994 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:10:35.446938 4405 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f71dd36502a0c52e983365f478eca1a00b26224c4ced97a4b3cb302f0660d9ac"} err="failed to get container status \"f71dd36502a0c52e983365f478eca1a00b26224c4ced97a4b3cb302f0660d9ac\": rpc error: code = NotFound desc = could not find container \"f71dd36502a0c52e983365f478eca1a00b26224c4ced97a4b3cb302f0660d9ac\": container with ID starting with f71dd36502a0c52e983365f478eca1a00b26224c4ced97a4b3cb302f0660d9ac not found: ID does not exist" Nov 04 03:10:35.606806 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:10:35.606762 4405 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-9f4199-85bd8d95f6-6k95j"] Nov 04 03:10:35.614527 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:10:35.614478 4405 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-9f4199-85bd8d95f6-6k95j"] Nov 04 03:10:35.703531 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:10:35.703447 4405 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="97c488a6-ee10-46bf-bc00-ef8578040f88" path="/var/lib/kubelet/pods/97c488a6-ee10-46bf-bc00-ef8578040f88/volumes" Nov 04 03:10:36.886149 ip-10-0-134-125 kubenswrapper[4405]: E1104 03:10:36.886099 4405 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5f2vb" containerName="extractor" Nov 04 03:10:41.401100 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:10:41.401068 4405 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-4cbcf738-67bd6b6f59-mfssb" Nov 04 03:10:46.778604 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:10:46.778509 4405 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-fe4e0c/distinct-2-w0-fe4e0c-kserve-579bf4cc77-8pbq8"] Nov 04 03:10:49.085072 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:10:49.085035 4405 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-e516ff/distinct-w1-e516ff-kserve-f4474cb9c-glpzw"] Nov 04 03:10:56.385512 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:10:56.385472 4405 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-9d2b96-7d4bf9f94-959mx"] Nov 04 03:10:56.385974 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:10:56.385697 4405 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-9d2b96-7d4bf9f94-959mx" podUID="43f2feb2-2005-4eca-a400-0d66bd1a4aac" containerName="maas-api" containerID="cri-o://f42f027d5f382c7bc439859a5e1264080e1a2829513ebed784760e7830fef49f" gracePeriod=30 Nov 04 03:10:56.576528 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:10:56.576297 4405 generic.go:356] "Generic (PLEG): container finished" podID="43f2feb2-2005-4eca-a400-0d66bd1a4aac" containerID="f42f027d5f382c7bc439859a5e1264080e1a2829513ebed784760e7830fef49f" exitCode=0 Nov 04 03:10:56.576528 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:10:56.576444 4405 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-9d2b96-7d4bf9f94-959mx" event={"ID":"43f2feb2-2005-4eca-a400-0d66bd1a4aac","Type":"ContainerDied","Data":"f42f027d5f382c7bc439859a5e1264080e1a2829513ebed784760e7830fef49f"} Nov 04 03:10:56.684894 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:10:56.684867 4405 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-9d2b96-7d4bf9f94-959mx" Nov 04 03:10:56.812257 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:10:56.808119 4405 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/43f2feb2-2005-4eca-a400-0d66bd1a4aac-kube-api-access-22mx6\" (UniqueName: \"kubernetes.io/projected/43f2feb2-2005-4eca-a400-0d66bd1a4aac-kube-api-access-22mx6\") pod \"43f2feb2-2005-4eca-a400-0d66bd1a4aac\" (UID: \"43f2feb2-2005-4eca-a400-0d66bd1a4aac\") " Nov 04 03:10:56.812257 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:10:56.808186 4405 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/43f2feb2-2005-4eca-a400-0d66bd1a4aac-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/43f2feb2-2005-4eca-a400-0d66bd1a4aac-maas-api-tls\") pod \"43f2feb2-2005-4eca-a400-0d66bd1a4aac\" (UID: \"43f2feb2-2005-4eca-a400-0d66bd1a4aac\") " Nov 04 03:10:56.812257 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:10:56.808353 4405 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/43f2feb2-2005-4eca-a400-0d66bd1a4aac-metrics-certs\" (UniqueName: \"kubernetes.io/secret/43f2feb2-2005-4eca-a400-0d66bd1a4aac-metrics-certs\") pod \"43f2feb2-2005-4eca-a400-0d66bd1a4aac\" (UID: \"43f2feb2-2005-4eca-a400-0d66bd1a4aac\") " Nov 04 03:10:56.812257 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:10:56.808425 4405 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/43f2feb2-2005-4eca-a400-0d66bd1a4aac-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/43f2feb2-2005-4eca-a400-0d66bd1a4aac-openshift-service-ca\") pod \"43f2feb2-2005-4eca-a400-0d66bd1a4aac\" (UID: \"43f2feb2-2005-4eca-a400-0d66bd1a4aac\") " Nov 04 03:10:56.825342 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:10:56.812811 4405 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/43f2feb2-2005-4eca-a400-0d66bd1a4aac-openshift-service-ca" pod "43f2feb2-2005-4eca-a400-0d66bd1a4aac" (UID: "43f2feb2-2005-4eca-a400-0d66bd1a4aac"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 03:10:56.827153 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:10:56.827051 4405 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/43f2feb2-2005-4eca-a400-0d66bd1a4aac-maas-api-tls" pod "43f2feb2-2005-4eca-a400-0d66bd1a4aac" (UID: "43f2feb2-2005-4eca-a400-0d66bd1a4aac"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 04 03:10:56.844631 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:10:56.844595 4405 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/43f2feb2-2005-4eca-a400-0d66bd1a4aac-kube-api-access-22mx6" pod "43f2feb2-2005-4eca-a400-0d66bd1a4aac" (UID: "43f2feb2-2005-4eca-a400-0d66bd1a4aac"). InnerVolumeSpecName "kube-api-access-22mx6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 03:10:56.844800 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:10:56.844745 4405 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/43f2feb2-2005-4eca-a400-0d66bd1a4aac-metrics-certs" pod "43f2feb2-2005-4eca-a400-0d66bd1a4aac" (UID: "43f2feb2-2005-4eca-a400-0d66bd1a4aac"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 04 03:10:56.910398 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:10:56.910306 4405 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/43f2feb2-2005-4eca-a400-0d66bd1a4aac-metrics-certs\") on node \"ip-10-0-134-125.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:10:56.910398 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:10:56.910349 4405 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/43f2feb2-2005-4eca-a400-0d66bd1a4aac-openshift-service-ca\") on node \"ip-10-0-134-125.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:10:56.910398 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:10:56.910367 4405 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-22mx6\" (UniqueName: \"kubernetes.io/projected/43f2feb2-2005-4eca-a400-0d66bd1a4aac-kube-api-access-22mx6\") on node \"ip-10-0-134-125.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:10:56.910398 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:10:56.910381 4405 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/43f2feb2-2005-4eca-a400-0d66bd1a4aac-maas-api-tls\") on node \"ip-10-0-134-125.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:10:57.583949 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:10:57.583923 4405 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-9d2b96-7d4bf9f94-959mx" Nov 04 03:10:57.583949 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:10:57.583936 4405 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-9d2b96-7d4bf9f94-959mx" event={"ID":"43f2feb2-2005-4eca-a400-0d66bd1a4aac","Type":"ContainerDied","Data":"721d215bc3c00f8300a925455688b77b0b8b8145f25fa4402c56052e1cc41f32"} Nov 04 03:10:57.584456 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:10:57.583992 4405 scope.go:122] "RemoveContainer" containerID="f42f027d5f382c7bc439859a5e1264080e1a2829513ebed784760e7830fef49f" Nov 04 03:10:57.748358 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:10:57.748324 4405 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-9d2b96-7d4bf9f94-959mx"] Nov 04 03:10:57.754183 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:10:57.754157 4405 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-9d2b96-7d4bf9f94-959mx"] Nov 04 03:10:59.702866 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:10:59.702822 4405 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="43f2feb2-2005-4eca-a400-0d66bd1a4aac" path="/var/lib/kubelet/pods/43f2feb2-2005-4eca-a400-0d66bd1a4aac/volumes" Nov 04 03:11:00.234601 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:11:00.234556 4405 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-fe4e0c-7dcb6cb4dd-vdb4p"] Nov 04 03:11:01.659060 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:11:01.659022 4405 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-bl5mg"] Nov 04 03:11:01.805124 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:11:01.805087 4405 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w0-fe4e0c/distinct-2-w0-fe4e0c-kserve-579bf4cc77-8pbq8" podUID="2b2ab722-d1bb-4376-bede-f5f4ee4562aa" containerName="main" containerID="cri-o://f4e91b1a963343e6bc85130338df13366abb078cb3bcc2e9717901c36187b229" gracePeriod=45 Nov 04 03:11:02.141018 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:11:02.140989 4405 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-fe4e0c/distinct-2-w0-fe4e0c-kserve-579bf4cc77-8pbq8" Nov 04 03:11:02.183351 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:11:02.183256 4405 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2b2ab722-d1bb-4376-bede-f5f4ee4562aa-kube-api-access-gdx2g\" (UniqueName: \"kubernetes.io/projected/2b2ab722-d1bb-4376-bede-f5f4ee4562aa-kube-api-access-gdx2g\") pod \"2b2ab722-d1bb-4376-bede-f5f4ee4562aa\" (UID: \"2b2ab722-d1bb-4376-bede-f5f4ee4562aa\") " Nov 04 03:11:02.183351 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:11:02.183327 4405 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2b2ab722-d1bb-4376-bede-f5f4ee4562aa-home\" (UniqueName: \"kubernetes.io/empty-dir/2b2ab722-d1bb-4376-bede-f5f4ee4562aa-home\") pod \"2b2ab722-d1bb-4376-bede-f5f4ee4562aa\" (UID: \"2b2ab722-d1bb-4376-bede-f5f4ee4562aa\") " Nov 04 03:11:02.183593 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:11:02.183374 4405 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2b2ab722-d1bb-4376-bede-f5f4ee4562aa-tls-certs\" (UniqueName: \"kubernetes.io/secret/2b2ab722-d1bb-4376-bede-f5f4ee4562aa-tls-certs\") pod \"2b2ab722-d1bb-4376-bede-f5f4ee4562aa\" (UID: \"2b2ab722-d1bb-4376-bede-f5f4ee4562aa\") " Nov 04 03:11:02.183593 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:11:02.183523 4405 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2b2ab722-d1bb-4376-bede-f5f4ee4562aa-dshm\" (UniqueName: \"kubernetes.io/empty-dir/2b2ab722-d1bb-4376-bede-f5f4ee4562aa-dshm\") pod \"2b2ab722-d1bb-4376-bede-f5f4ee4562aa\" (UID: \"2b2ab722-d1bb-4376-bede-f5f4ee4562aa\") " Nov 04 03:11:02.183711 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:11:02.183608 4405 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2b2ab722-d1bb-4376-bede-f5f4ee4562aa-home" pod "2b2ab722-d1bb-4376-bede-f5f4ee4562aa" (UID: "2b2ab722-d1bb-4376-bede-f5f4ee4562aa"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 03:11:02.183711 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:11:02.183657 4405 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2b2ab722-d1bb-4376-bede-f5f4ee4562aa-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2b2ab722-d1bb-4376-bede-f5f4ee4562aa-tmp-dir\") pod \"2b2ab722-d1bb-4376-bede-f5f4ee4562aa\" (UID: \"2b2ab722-d1bb-4376-bede-f5f4ee4562aa\") " Nov 04 03:11:02.183833 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:11:02.183747 4405 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2b2ab722-d1bb-4376-bede-f5f4ee4562aa-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2b2ab722-d1bb-4376-bede-f5f4ee4562aa-model-cache\") pod \"2b2ab722-d1bb-4376-bede-f5f4ee4562aa\" (UID: \"2b2ab722-d1bb-4376-bede-f5f4ee4562aa\") " Nov 04 03:11:02.183933 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:11:02.183904 4405 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2b2ab722-d1bb-4376-bede-f5f4ee4562aa-tmp-dir" pod "2b2ab722-d1bb-4376-bede-f5f4ee4562aa" (UID: "2b2ab722-d1bb-4376-bede-f5f4ee4562aa"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 03:11:02.184063 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:11:02.184009 4405 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2b2ab722-d1bb-4376-bede-f5f4ee4562aa-model-cache" pod "2b2ab722-d1bb-4376-bede-f5f4ee4562aa" (UID: "2b2ab722-d1bb-4376-bede-f5f4ee4562aa"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 03:11:02.184525 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:11:02.184505 4405 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2b2ab722-d1bb-4376-bede-f5f4ee4562aa-tmp-dir\") on node \"ip-10-0-134-125.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:11:02.184598 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:11:02.184533 4405 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2b2ab722-d1bb-4376-bede-f5f4ee4562aa-model-cache\") on node \"ip-10-0-134-125.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:11:02.184598 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:11:02.184549 4405 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2b2ab722-d1bb-4376-bede-f5f4ee4562aa-home\") on node \"ip-10-0-134-125.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:11:02.186014 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:11:02.185987 4405 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2b2ab722-d1bb-4376-bede-f5f4ee4562aa-dshm" pod "2b2ab722-d1bb-4376-bede-f5f4ee4562aa" (UID: "2b2ab722-d1bb-4376-bede-f5f4ee4562aa"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 03:11:02.186114 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:11:02.186028 4405 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2b2ab722-d1bb-4376-bede-f5f4ee4562aa-kube-api-access-gdx2g" pod "2b2ab722-d1bb-4376-bede-f5f4ee4562aa" (UID: "2b2ab722-d1bb-4376-bede-f5f4ee4562aa"). InnerVolumeSpecName "kube-api-access-gdx2g". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 03:11:02.186114 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:11:02.186068 4405 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2b2ab722-d1bb-4376-bede-f5f4ee4562aa-tls-certs" pod "2b2ab722-d1bb-4376-bede-f5f4ee4562aa" (UID: "2b2ab722-d1bb-4376-bede-f5f4ee4562aa"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 04 03:11:02.286651 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:11:02.286607 4405 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gdx2g\" (UniqueName: \"kubernetes.io/projected/2b2ab722-d1bb-4376-bede-f5f4ee4562aa-kube-api-access-gdx2g\") on node \"ip-10-0-134-125.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:11:02.286651 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:11:02.286654 4405 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2b2ab722-d1bb-4376-bede-f5f4ee4562aa-tls-certs\") on node \"ip-10-0-134-125.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:11:02.286886 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:11:02.286690 4405 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2b2ab722-d1bb-4376-bede-f5f4ee4562aa-dshm\") on node \"ip-10-0-134-125.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:11:02.615618 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:11:02.615582 4405 generic.go:356] "Generic (PLEG): container finished" podID="2b2ab722-d1bb-4376-bede-f5f4ee4562aa" containerID="f4e91b1a963343e6bc85130338df13366abb078cb3bcc2e9717901c36187b229" exitCode=0 Nov 04 03:11:02.615853 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:11:02.615627 4405 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-fe4e0c/distinct-2-w0-fe4e0c-kserve-579bf4cc77-8pbq8" Nov 04 03:11:02.615853 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:11:02.615656 4405 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-fe4e0c/distinct-2-w0-fe4e0c-kserve-579bf4cc77-8pbq8" event={"ID":"2b2ab722-d1bb-4376-bede-f5f4ee4562aa","Type":"ContainerDied","Data":"f4e91b1a963343e6bc85130338df13366abb078cb3bcc2e9717901c36187b229"} Nov 04 03:11:02.615853 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:11:02.615719 4405 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-fe4e0c/distinct-2-w0-fe4e0c-kserve-579bf4cc77-8pbq8" event={"ID":"2b2ab722-d1bb-4376-bede-f5f4ee4562aa","Type":"ContainerDied","Data":"4c0c112228670c01e1b4bea9d2afa162752248b29abe9da1d7bd83caa1a826c2"} Nov 04 03:11:02.615853 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:11:02.615738 4405 scope.go:122] "RemoveContainer" containerID="f4e91b1a963343e6bc85130338df13366abb078cb3bcc2e9717901c36187b229" Nov 04 03:11:02.626249 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:11:02.626227 4405 scope.go:122] "RemoveContainer" containerID="f4e91b1a963343e6bc85130338df13366abb078cb3bcc2e9717901c36187b229" Nov 04 03:11:02.626519 ip-10-0-134-125 kubenswrapper[4405]: E1104 03:11:02.626501 4405 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f4e91b1a963343e6bc85130338df13366abb078cb3bcc2e9717901c36187b229\": container with ID starting with f4e91b1a963343e6bc85130338df13366abb078cb3bcc2e9717901c36187b229 not found: ID does not exist" containerID="f4e91b1a963343e6bc85130338df13366abb078cb3bcc2e9717901c36187b229" Nov 04 03:11:02.626570 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:11:02.626529 4405 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f4e91b1a963343e6bc85130338df13366abb078cb3bcc2e9717901c36187b229"} err="failed to get container status \"f4e91b1a963343e6bc85130338df13366abb078cb3bcc2e9717901c36187b229\": rpc error: code = NotFound desc = could not find container \"f4e91b1a963343e6bc85130338df13366abb078cb3bcc2e9717901c36187b229\": container with ID starting with f4e91b1a963343e6bc85130338df13366abb078cb3bcc2e9717901c36187b229 not found: ID does not exist" Nov 04 03:11:02.784631 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:11:02.784592 4405 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-fe4e0c/distinct-2-w0-fe4e0c-kserve-579bf4cc77-8pbq8"] Nov 04 03:11:02.789914 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:11:02.789875 4405 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w0-fe4e0c/distinct-2-w0-fe4e0c-kserve-579bf4cc77-8pbq8"] Nov 04 03:11:03.136691 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:11:03.131729 4405 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-fe4e0c/distinct-2-w0-fe4e0c-kserve-579bf4cc77-8pbq8" podUID="2b2ab722-d1bb-4376-bede-f5f4ee4562aa" containerName="main" probeResult="failure" output="Get \"https://10.134.0.68:8000/ready\": context deadline exceeded" Nov 04 03:11:03.721695 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:11:03.721612 4405 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2b2ab722-d1bb-4376-bede-f5f4ee4562aa" path="/var/lib/kubelet/pods/2b2ab722-d1bb-4376-bede-f5f4ee4562aa/volumes" Nov 04 03:11:04.114910 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:11:04.114862 4405 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w1-e516ff/distinct-w1-e516ff-kserve-f4474cb9c-glpzw" podUID="8842ad09-22ff-4492-ba5d-d4cacf4eac75" containerName="main" containerID="cri-o://8466c7b2a5aee2d8317eed1fead0e2d0f8a7976eb7f7a3bb8242a77ff6d01666" gracePeriod=45 Nov 04 03:11:04.879275 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:11:04.878988 4405 generic.go:356] "Generic (PLEG): container finished" podID="8842ad09-22ff-4492-ba5d-d4cacf4eac75" containerID="8466c7b2a5aee2d8317eed1fead0e2d0f8a7976eb7f7a3bb8242a77ff6d01666" exitCode=0 Nov 04 03:11:04.879275 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:11:04.879205 4405 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-e516ff/distinct-w1-e516ff-kserve-f4474cb9c-glpzw" event={"ID":"8842ad09-22ff-4492-ba5d-d4cacf4eac75","Type":"ContainerDied","Data":"8466c7b2a5aee2d8317eed1fead0e2d0f8a7976eb7f7a3bb8242a77ff6d01666"} Nov 04 03:11:04.917273 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:11:04.916871 4405 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-e516ff/distinct-w1-e516ff-kserve-f4474cb9c-glpzw" Nov 04 03:11:05.065754 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:11:05.057139 4405 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8842ad09-22ff-4492-ba5d-d4cacf4eac75-kube-api-access-hwc48\" (UniqueName: \"kubernetes.io/projected/8842ad09-22ff-4492-ba5d-d4cacf4eac75-kube-api-access-hwc48\") pod \"8842ad09-22ff-4492-ba5d-d4cacf4eac75\" (UID: \"8842ad09-22ff-4492-ba5d-d4cacf4eac75\") " Nov 04 03:11:05.065754 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:11:05.057210 4405 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8842ad09-22ff-4492-ba5d-d4cacf4eac75-dshm\" (UniqueName: \"kubernetes.io/empty-dir/8842ad09-22ff-4492-ba5d-d4cacf4eac75-dshm\") pod \"8842ad09-22ff-4492-ba5d-d4cacf4eac75\" (UID: \"8842ad09-22ff-4492-ba5d-d4cacf4eac75\") " Nov 04 03:11:05.065754 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:11:05.057314 4405 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8842ad09-22ff-4492-ba5d-d4cacf4eac75-tls-certs\" (UniqueName: \"kubernetes.io/secret/8842ad09-22ff-4492-ba5d-d4cacf4eac75-tls-certs\") pod \"8842ad09-22ff-4492-ba5d-d4cacf4eac75\" (UID: \"8842ad09-22ff-4492-ba5d-d4cacf4eac75\") " Nov 04 03:11:05.065754 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:11:05.057356 4405 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8842ad09-22ff-4492-ba5d-d4cacf4eac75-home\" (UniqueName: \"kubernetes.io/empty-dir/8842ad09-22ff-4492-ba5d-d4cacf4eac75-home\") pod \"8842ad09-22ff-4492-ba5d-d4cacf4eac75\" (UID: \"8842ad09-22ff-4492-ba5d-d4cacf4eac75\") " Nov 04 03:11:05.065754 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:11:05.057475 4405 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8842ad09-22ff-4492-ba5d-d4cacf4eac75-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8842ad09-22ff-4492-ba5d-d4cacf4eac75-model-cache\") pod \"8842ad09-22ff-4492-ba5d-d4cacf4eac75\" (UID: \"8842ad09-22ff-4492-ba5d-d4cacf4eac75\") " Nov 04 03:11:05.065754 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:11:05.057546 4405 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8842ad09-22ff-4492-ba5d-d4cacf4eac75-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8842ad09-22ff-4492-ba5d-d4cacf4eac75-tmp-dir\") pod \"8842ad09-22ff-4492-ba5d-d4cacf4eac75\" (UID: \"8842ad09-22ff-4492-ba5d-d4cacf4eac75\") " Nov 04 03:11:05.065754 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:11:05.058327 4405 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8842ad09-22ff-4492-ba5d-d4cacf4eac75-tmp-dir" pod "8842ad09-22ff-4492-ba5d-d4cacf4eac75" (UID: "8842ad09-22ff-4492-ba5d-d4cacf4eac75"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 03:11:05.066602 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:11:05.066574 4405 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8842ad09-22ff-4492-ba5d-d4cacf4eac75-home" pod "8842ad09-22ff-4492-ba5d-d4cacf4eac75" (UID: "8842ad09-22ff-4492-ba5d-d4cacf4eac75"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 03:11:05.068142 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:11:05.068117 4405 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8842ad09-22ff-4492-ba5d-d4cacf4eac75-model-cache" pod "8842ad09-22ff-4492-ba5d-d4cacf4eac75" (UID: "8842ad09-22ff-4492-ba5d-d4cacf4eac75"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 03:11:05.165757 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:11:05.165696 4405 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8842ad09-22ff-4492-ba5d-d4cacf4eac75-dshm" pod "8842ad09-22ff-4492-ba5d-d4cacf4eac75" (UID: "8842ad09-22ff-4492-ba5d-d4cacf4eac75"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 03:11:05.179459 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:11:05.174328 4405 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8842ad09-22ff-4492-ba5d-d4cacf4eac75-tls-certs" pod "8842ad09-22ff-4492-ba5d-d4cacf4eac75" (UID: "8842ad09-22ff-4492-ba5d-d4cacf4eac75"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 04 03:11:05.179459 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:11:05.174610 4405 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8842ad09-22ff-4492-ba5d-d4cacf4eac75-kube-api-access-hwc48" pod "8842ad09-22ff-4492-ba5d-d4cacf4eac75" (UID: "8842ad09-22ff-4492-ba5d-d4cacf4eac75"). InnerVolumeSpecName "kube-api-access-hwc48". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 03:11:05.179459 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:11:05.176143 4405 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hwc48\" (UniqueName: \"kubernetes.io/projected/8842ad09-22ff-4492-ba5d-d4cacf4eac75-kube-api-access-hwc48\") on node \"ip-10-0-134-125.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:11:05.179459 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:11:05.176173 4405 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8842ad09-22ff-4492-ba5d-d4cacf4eac75-dshm\") on node \"ip-10-0-134-125.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:11:05.179459 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:11:05.176188 4405 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8842ad09-22ff-4492-ba5d-d4cacf4eac75-tls-certs\") on node \"ip-10-0-134-125.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:11:05.179459 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:11:05.176203 4405 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8842ad09-22ff-4492-ba5d-d4cacf4eac75-home\") on node \"ip-10-0-134-125.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:11:05.179459 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:11:05.176217 4405 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8842ad09-22ff-4492-ba5d-d4cacf4eac75-model-cache\") on node \"ip-10-0-134-125.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:11:05.179459 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:11:05.176231 4405 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8842ad09-22ff-4492-ba5d-d4cacf4eac75-tmp-dir\") on node \"ip-10-0-134-125.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:11:05.245774 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:11:05.243755 4405 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w0-fe4e0c-7dcb6cb4dd-vdb4p" podUID="7d5dc4d7-ea24-4845-8087-1e999e739f22" containerName="payload-processing" containerID="cri-o://3af3e50749525b71bf5ef60f63d83cb4e87edec65450942627b6648b148a7c6e" gracePeriod=25 Nov 04 03:11:05.799008 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:11:05.798976 4405 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w0-fe4e0c-7dcb6cb4dd-vdb4p" Nov 04 03:11:05.833106 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:11:05.833059 4405 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7d5dc4d7-ea24-4845-8087-1e999e739f22-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7d5dc4d7-ea24-4845-8087-1e999e739f22-plugins-config-volume\") pod \"7d5dc4d7-ea24-4845-8087-1e999e739f22\" (UID: \"7d5dc4d7-ea24-4845-8087-1e999e739f22\") " Nov 04 03:11:05.833311 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:11:05.833131 4405 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7d5dc4d7-ea24-4845-8087-1e999e739f22-kube-api-access-z7hvd\" (UniqueName: \"kubernetes.io/projected/7d5dc4d7-ea24-4845-8087-1e999e739f22-kube-api-access-z7hvd\") pod \"7d5dc4d7-ea24-4845-8087-1e999e739f22\" (UID: \"7d5dc4d7-ea24-4845-8087-1e999e739f22\") " Nov 04 03:11:05.833803 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:11:05.833772 4405 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7d5dc4d7-ea24-4845-8087-1e999e739f22-plugins-config-volume" pod "7d5dc4d7-ea24-4845-8087-1e999e739f22" (UID: "7d5dc4d7-ea24-4845-8087-1e999e739f22"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 03:11:05.836289 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:11:05.836258 4405 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7d5dc4d7-ea24-4845-8087-1e999e739f22-kube-api-access-z7hvd" pod "7d5dc4d7-ea24-4845-8087-1e999e739f22" (UID: "7d5dc4d7-ea24-4845-8087-1e999e739f22"). InnerVolumeSpecName "kube-api-access-z7hvd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 03:11:05.850380 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:11:05.850343 4405 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-e516ff/distinct-w1-e516ff-kserve-f4474cb9c-glpzw" podUID="8842ad09-22ff-4492-ba5d-d4cacf4eac75" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/ready\": context deadline exceeded" Nov 04 03:11:05.893689 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:11:05.890863 4405 generic.go:356] "Generic (PLEG): container finished" podID="7d5dc4d7-ea24-4845-8087-1e999e739f22" containerID="3af3e50749525b71bf5ef60f63d83cb4e87edec65450942627b6648b148a7c6e" exitCode=0 Nov 04 03:11:05.893689 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:11:05.891058 4405 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w0-fe4e0c-7dcb6cb4dd-vdb4p" event={"ID":"7d5dc4d7-ea24-4845-8087-1e999e739f22","Type":"ContainerDied","Data":"3af3e50749525b71bf5ef60f63d83cb4e87edec65450942627b6648b148a7c6e"} Nov 04 03:11:05.893689 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:11:05.891147 4405 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w0-fe4e0c-7dcb6cb4dd-vdb4p" event={"ID":"7d5dc4d7-ea24-4845-8087-1e999e739f22","Type":"ContainerDied","Data":"54978bcace432a3a1b05f4957dd709d76e5e460732528339ac7eb53fb75a2c54"} Nov 04 03:11:05.893689 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:11:05.891185 4405 scope.go:122] "RemoveContainer" containerID="3af3e50749525b71bf5ef60f63d83cb4e87edec65450942627b6648b148a7c6e" Nov 04 03:11:05.893689 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:11:05.891385 4405 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w0-fe4e0c-7dcb6cb4dd-vdb4p" Nov 04 03:11:05.896949 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:11:05.896919 4405 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-e516ff/distinct-w1-e516ff-kserve-f4474cb9c-glpzw" event={"ID":"8842ad09-22ff-4492-ba5d-d4cacf4eac75","Type":"ContainerDied","Data":"54d72f22a7ddba3dcd0cd9418c8b969d1256e9da25486b1a06437413a075c6a4"} Nov 04 03:11:05.897945 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:11:05.896944 4405 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-e516ff/distinct-w1-e516ff-kserve-f4474cb9c-glpzw" Nov 04 03:11:05.912728 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:11:05.912698 4405 scope.go:122] "RemoveContainer" containerID="3af3e50749525b71bf5ef60f63d83cb4e87edec65450942627b6648b148a7c6e" Nov 04 03:11:05.913124 ip-10-0-134-125 kubenswrapper[4405]: E1104 03:11:05.913097 4405 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3af3e50749525b71bf5ef60f63d83cb4e87edec65450942627b6648b148a7c6e\": container with ID starting with 3af3e50749525b71bf5ef60f63d83cb4e87edec65450942627b6648b148a7c6e not found: ID does not exist" containerID="3af3e50749525b71bf5ef60f63d83cb4e87edec65450942627b6648b148a7c6e" Nov 04 03:11:05.913246 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:11:05.913134 4405 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3af3e50749525b71bf5ef60f63d83cb4e87edec65450942627b6648b148a7c6e"} err="failed to get container status \"3af3e50749525b71bf5ef60f63d83cb4e87edec65450942627b6648b148a7c6e\": rpc error: code = NotFound desc = could not find container \"3af3e50749525b71bf5ef60f63d83cb4e87edec65450942627b6648b148a7c6e\": container with ID starting with 3af3e50749525b71bf5ef60f63d83cb4e87edec65450942627b6648b148a7c6e not found: ID does not exist" Nov 04 03:11:05.913246 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:11:05.913159 4405 scope.go:122] "RemoveContainer" containerID="8466c7b2a5aee2d8317eed1fead0e2d0f8a7976eb7f7a3bb8242a77ff6d01666" Nov 04 03:11:05.940891 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:11:05.940181 4405 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7d5dc4d7-ea24-4845-8087-1e999e739f22-plugins-config-volume\") on node \"ip-10-0-134-125.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:11:05.940891 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:11:05.940215 4405 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z7hvd\" (UniqueName: \"kubernetes.io/projected/7d5dc4d7-ea24-4845-8087-1e999e739f22-kube-api-access-z7hvd\") on node \"ip-10-0-134-125.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:11:06.032719 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:11:06.032685 4405 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-e516ff/distinct-w1-e516ff-kserve-f4474cb9c-glpzw"] Nov 04 03:11:06.039049 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:11:06.039008 4405 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w1-e516ff/distinct-w1-e516ff-kserve-f4474cb9c-glpzw"] Nov 04 03:11:06.218999 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:11:06.218860 4405 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-fe4e0c-7dcb6cb4dd-vdb4p"] Nov 04 03:11:06.228619 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:11:06.228579 4405 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-fe4e0c-7dcb6cb4dd-vdb4p"] Nov 04 03:11:06.887872 ip-10-0-134-125 kubenswrapper[4405]: E1104 03:11:06.887828 4405 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5f2vb" containerName="extractor" Nov 04 03:11:07.578029 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:11:07.577997 4405 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-22e72e-openshift-default-64656d9fb9-26n5c"] Nov 04 03:11:07.578528 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:11:07.578230 4405 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-22e72e-openshift-default-64656d9fb9-26n5c" podUID="fd747189-5360-4360-83cd-07a4e414fda4" containerName="istio-proxy" containerID="cri-o://9038e37f5c39ccd27ebcaedfcbfff9feb82d7fa234e173e625fe1ebdc0d3f11e" gracePeriod=30 Nov 04 03:11:07.724912 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:11:07.724869 4405 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7d5dc4d7-ea24-4845-8087-1e999e739f22" path="/var/lib/kubelet/pods/7d5dc4d7-ea24-4845-8087-1e999e739f22/volumes" Nov 04 03:11:07.725378 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:11:07.725362 4405 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8842ad09-22ff-4492-ba5d-d4cacf4eac75" path="/var/lib/kubelet/pods/8842ad09-22ff-4492-ba5d-d4cacf4eac75/volumes" Nov 04 03:11:11.550400 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:11:11.550361 4405 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w6-ebb7fd/distinct-2-w6-ebb7fd-kserve-6f68c5db86-zrpjs"] Nov 04 03:11:11.601112 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:11:11.601077 4405 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w6-ebb7fd/distinct-w6-ebb7fd-kserve-55c47d8865-7z9pz"] Nov 04 03:11:12.845042 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:11:12.844977 4405 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-22e72e-openshift-default-64656d9fb9-26n5c" Nov 04 03:11:12.949389 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:11:12.949352 4405 generic.go:356] "Generic (PLEG): container finished" podID="fd747189-5360-4360-83cd-07a4e414fda4" containerID="9038e37f5c39ccd27ebcaedfcbfff9feb82d7fa234e173e625fe1ebdc0d3f11e" exitCode=0 Nov 04 03:11:12.949609 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:11:12.949416 4405 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-22e72e-openshift-default-64656d9fb9-26n5c" Nov 04 03:11:12.949609 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:11:12.949434 4405 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-22e72e-openshift-default-64656d9fb9-26n5c" event={"ID":"fd747189-5360-4360-83cd-07a4e414fda4","Type":"ContainerDied","Data":"9038e37f5c39ccd27ebcaedfcbfff9feb82d7fa234e173e625fe1ebdc0d3f11e"} Nov 04 03:11:12.949609 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:11:12.949488 4405 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-22e72e-openshift-default-64656d9fb9-26n5c" event={"ID":"fd747189-5360-4360-83cd-07a4e414fda4","Type":"ContainerDied","Data":"22f652c5ab1a3f26e4618c6db785aa4bee2684e7a3403afd44cb2947bf7a97ce"} Nov 04 03:11:12.949609 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:11:12.949511 4405 scope.go:122] "RemoveContainer" containerID="9038e37f5c39ccd27ebcaedfcbfff9feb82d7fa234e173e625fe1ebdc0d3f11e" Nov 04 03:11:12.960637 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:11:12.960616 4405 scope.go:122] "RemoveContainer" containerID="27b80f527345011ee23f01f21487a67b9800509257b6b45c3ec35831fdfe7633" Nov 04 03:11:12.971172 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:11:12.971148 4405 scope.go:122] "RemoveContainer" containerID="9038e37f5c39ccd27ebcaedfcbfff9feb82d7fa234e173e625fe1ebdc0d3f11e" Nov 04 03:11:12.971510 ip-10-0-134-125 kubenswrapper[4405]: E1104 03:11:12.971479 4405 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9038e37f5c39ccd27ebcaedfcbfff9feb82d7fa234e173e625fe1ebdc0d3f11e\": container with ID starting with 9038e37f5c39ccd27ebcaedfcbfff9feb82d7fa234e173e625fe1ebdc0d3f11e not found: ID does not exist" containerID="9038e37f5c39ccd27ebcaedfcbfff9feb82d7fa234e173e625fe1ebdc0d3f11e" Nov 04 03:11:12.971585 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:11:12.971522 4405 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9038e37f5c39ccd27ebcaedfcbfff9feb82d7fa234e173e625fe1ebdc0d3f11e"} err="failed to get container status \"9038e37f5c39ccd27ebcaedfcbfff9feb82d7fa234e173e625fe1ebdc0d3f11e\": rpc error: code = NotFound desc = could not find container \"9038e37f5c39ccd27ebcaedfcbfff9feb82d7fa234e173e625fe1ebdc0d3f11e\": container with ID starting with 9038e37f5c39ccd27ebcaedfcbfff9feb82d7fa234e173e625fe1ebdc0d3f11e not found: ID does not exist" Nov 04 03:11:12.971585 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:11:12.971547 4405 scope.go:122] "RemoveContainer" containerID="27b80f527345011ee23f01f21487a67b9800509257b6b45c3ec35831fdfe7633" Nov 04 03:11:12.971880 ip-10-0-134-125 kubenswrapper[4405]: E1104 03:11:12.971862 4405 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"27b80f527345011ee23f01f21487a67b9800509257b6b45c3ec35831fdfe7633\": container with ID starting with 27b80f527345011ee23f01f21487a67b9800509257b6b45c3ec35831fdfe7633 not found: ID does not exist" containerID="27b80f527345011ee23f01f21487a67b9800509257b6b45c3ec35831fdfe7633" Nov 04 03:11:12.971943 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:11:12.971887 4405 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"27b80f527345011ee23f01f21487a67b9800509257b6b45c3ec35831fdfe7633"} err="failed to get container status \"27b80f527345011ee23f01f21487a67b9800509257b6b45c3ec35831fdfe7633\": rpc error: code = NotFound desc = could not find container \"27b80f527345011ee23f01f21487a67b9800509257b6b45c3ec35831fdfe7633\": container with ID starting with 27b80f527345011ee23f01f21487a67b9800509257b6b45c3ec35831fdfe7633 not found: ID does not exist" Nov 04 03:11:12.977352 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:11:12.977332 4405 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fd747189-5360-4360-83cd-07a4e414fda4-istio-token\" (UniqueName: \"kubernetes.io/projected/fd747189-5360-4360-83cd-07a4e414fda4-istio-token\") pod \"fd747189-5360-4360-83cd-07a4e414fda4\" (UID: \"fd747189-5360-4360-83cd-07a4e414fda4\") " Nov 04 03:11:12.977416 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:11:12.977378 4405 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fd747189-5360-4360-83cd-07a4e414fda4-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fd747189-5360-4360-83cd-07a4e414fda4-istio-data\") pod \"fd747189-5360-4360-83cd-07a4e414fda4\" (UID: \"fd747189-5360-4360-83cd-07a4e414fda4\") " Nov 04 03:11:12.977452 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:11:12.977421 4405 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fd747189-5360-4360-83cd-07a4e414fda4-kube-api-access-dwk9j\" (UniqueName: \"kubernetes.io/projected/fd747189-5360-4360-83cd-07a4e414fda4-kube-api-access-dwk9j\") pod \"fd747189-5360-4360-83cd-07a4e414fda4\" (UID: \"fd747189-5360-4360-83cd-07a4e414fda4\") " Nov 04 03:11:12.977521 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:11:12.977505 4405 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fd747189-5360-4360-83cd-07a4e414fda4-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fd747189-5360-4360-83cd-07a4e414fda4-istiod-ca-cert\") pod \"fd747189-5360-4360-83cd-07a4e414fda4\" (UID: \"fd747189-5360-4360-83cd-07a4e414fda4\") " Nov 04 03:11:12.977577 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:11:12.977544 4405 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/fd747189-5360-4360-83cd-07a4e414fda4-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fd747189-5360-4360-83cd-07a4e414fda4-istio-podinfo\") pod \"fd747189-5360-4360-83cd-07a4e414fda4\" (UID: \"fd747189-5360-4360-83cd-07a4e414fda4\") " Nov 04 03:11:12.977628 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:11:12.977604 4405 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fd747189-5360-4360-83cd-07a4e414fda4-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fd747189-5360-4360-83cd-07a4e414fda4-istio-envoy\") pod \"fd747189-5360-4360-83cd-07a4e414fda4\" (UID: \"fd747189-5360-4360-83cd-07a4e414fda4\") " Nov 04 03:11:12.977697 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:11:12.977651 4405 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fd747189-5360-4360-83cd-07a4e414fda4-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fd747189-5360-4360-83cd-07a4e414fda4-workload-certs\") pod \"fd747189-5360-4360-83cd-07a4e414fda4\" (UID: \"fd747189-5360-4360-83cd-07a4e414fda4\") " Nov 04 03:11:12.977754 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:11:12.977708 4405 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fd747189-5360-4360-83cd-07a4e414fda4-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fd747189-5360-4360-83cd-07a4e414fda4-workload-socket\") pod \"fd747189-5360-4360-83cd-07a4e414fda4\" (UID: \"fd747189-5360-4360-83cd-07a4e414fda4\") " Nov 04 03:11:12.977754 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:11:12.977740 4405 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fd747189-5360-4360-83cd-07a4e414fda4-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fd747189-5360-4360-83cd-07a4e414fda4-credential-socket\") pod \"fd747189-5360-4360-83cd-07a4e414fda4\" (UID: \"fd747189-5360-4360-83cd-07a4e414fda4\") " Nov 04 03:11:12.977862 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:11:12.977757 4405 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fd747189-5360-4360-83cd-07a4e414fda4-istio-data" pod "fd747189-5360-4360-83cd-07a4e414fda4" (UID: "fd747189-5360-4360-83cd-07a4e414fda4"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 03:11:12.978089 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:11:12.977980 4405 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fd747189-5360-4360-83cd-07a4e414fda4-workload-certs" pod "fd747189-5360-4360-83cd-07a4e414fda4" (UID: "fd747189-5360-4360-83cd-07a4e414fda4"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 03:11:12.978089 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:11:12.977996 4405 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fd747189-5360-4360-83cd-07a4e414fda4-workload-socket" pod "fd747189-5360-4360-83cd-07a4e414fda4" (UID: "fd747189-5360-4360-83cd-07a4e414fda4"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 03:11:12.978089 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:11:12.978031 4405 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fd747189-5360-4360-83cd-07a4e414fda4-istiod-ca-cert" pod "fd747189-5360-4360-83cd-07a4e414fda4" (UID: "fd747189-5360-4360-83cd-07a4e414fda4"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 03:11:12.978285 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:11:12.978148 4405 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fd747189-5360-4360-83cd-07a4e414fda4-credential-socket" pod "fd747189-5360-4360-83cd-07a4e414fda4" (UID: "fd747189-5360-4360-83cd-07a4e414fda4"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 03:11:12.978421 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:11:12.978399 4405 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fd747189-5360-4360-83cd-07a4e414fda4-istio-data\") on node \"ip-10-0-134-125.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:11:12.978421 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:11:12.978420 4405 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fd747189-5360-4360-83cd-07a4e414fda4-istiod-ca-cert\") on node \"ip-10-0-134-125.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:11:12.978591 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:11:12.978430 4405 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fd747189-5360-4360-83cd-07a4e414fda4-workload-certs\") on node \"ip-10-0-134-125.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:11:12.978591 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:11:12.978439 4405 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fd747189-5360-4360-83cd-07a4e414fda4-workload-socket\") on node \"ip-10-0-134-125.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:11:12.978591 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:11:12.978447 4405 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fd747189-5360-4360-83cd-07a4e414fda4-credential-socket\") on node \"ip-10-0-134-125.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:11:12.980128 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:11:12.980093 4405 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fd747189-5360-4360-83cd-07a4e414fda4-kube-api-access-dwk9j" pod "fd747189-5360-4360-83cd-07a4e414fda4" (UID: "fd747189-5360-4360-83cd-07a4e414fda4"). InnerVolumeSpecName "kube-api-access-dwk9j". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 03:11:12.980128 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:11:12.980094 4405 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fd747189-5360-4360-83cd-07a4e414fda4-istio-envoy" pod "fd747189-5360-4360-83cd-07a4e414fda4" (UID: "fd747189-5360-4360-83cd-07a4e414fda4"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 03:11:12.980128 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:11:12.980117 4405 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fd747189-5360-4360-83cd-07a4e414fda4-istio-token" pod "fd747189-5360-4360-83cd-07a4e414fda4" (UID: "fd747189-5360-4360-83cd-07a4e414fda4"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 03:11:12.980629 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:11:12.980601 4405 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/fd747189-5360-4360-83cd-07a4e414fda4-istio-podinfo" pod "fd747189-5360-4360-83cd-07a4e414fda4" (UID: "fd747189-5360-4360-83cd-07a4e414fda4"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 04 03:11:13.079699 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:11:13.079637 4405 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fd747189-5360-4360-83cd-07a4e414fda4-istio-podinfo\") on node \"ip-10-0-134-125.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:11:13.079699 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:11:13.079698 4405 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fd747189-5360-4360-83cd-07a4e414fda4-istio-envoy\") on node \"ip-10-0-134-125.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:11:13.079699 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:11:13.079708 4405 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fd747189-5360-4360-83cd-07a4e414fda4-istio-token\") on node \"ip-10-0-134-125.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:11:13.080063 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:11:13.079721 4405 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dwk9j\" (UniqueName: \"kubernetes.io/projected/fd747189-5360-4360-83cd-07a4e414fda4-kube-api-access-dwk9j\") on node \"ip-10-0-134-125.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:11:13.372880 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:11:13.372839 4405 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-22e72e-openshift-default-64656d9fb9-26n5c"] Nov 04 03:11:13.379322 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:11:13.379296 4405 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-22e72e-openshift-default-64656d9fb9-26n5c"] Nov 04 03:11:13.702301 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:11:13.702266 4405 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fd747189-5360-4360-83cd-07a4e414fda4" path="/var/lib/kubelet/pods/fd747189-5360-4360-83cd-07a4e414fda4/volumes" Nov 04 03:11:14.588480 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:11:14.588442 4405 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-cdcf67/distinct-w3-cdcf67-kserve-667cc7f7d7-5m2lq"] Nov 04 03:11:21.631564 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:11:21.631533 4405 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-095081-bd58cc787-xz5kc"] Nov 04 03:11:21.632072 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:11:21.632058 4405 cpu_manager.go:414] "containerMap: removing container" podUID="8842ad09-22ff-4492-ba5d-d4cacf4eac75" containerName="main" Nov 04 03:11:21.632121 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:11:21.632073 4405 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8842ad09-22ff-4492-ba5d-d4cacf4eac75" containerName="main" Nov 04 03:11:21.632121 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:11:21.632091 4405 cpu_manager.go:414] "containerMap: removing container" podUID="8a0008f3-4ab9-4ceb-8feb-0f536776610a" containerName="istio-proxy" Nov 04 03:11:21.632121 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:11:21.632097 4405 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8a0008f3-4ab9-4ceb-8feb-0f536776610a" containerName="istio-proxy" Nov 04 03:11:21.632121 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:11:21.632108 4405 cpu_manager.go:414] "containerMap: removing container" podUID="43f2feb2-2005-4eca-a400-0d66bd1a4aac" containerName="maas-api" Nov 04 03:11:21.632121 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:11:21.632113 4405 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="43f2feb2-2005-4eca-a400-0d66bd1a4aac" containerName="maas-api" Nov 04 03:11:21.632121 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:11:21.632120 4405 cpu_manager.go:414] "containerMap: removing container" podUID="fd747189-5360-4360-83cd-07a4e414fda4" containerName="istio-proxy" Nov 04 03:11:21.632307 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:11:21.632125 4405 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fd747189-5360-4360-83cd-07a4e414fda4" containerName="istio-proxy" Nov 04 03:11:21.632307 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:11:21.632132 4405 cpu_manager.go:414] "containerMap: removing container" podUID="e6cb6771-a9e6-452f-a0b6-30ffb955feb8" containerName="istio-proxy" Nov 04 03:11:21.632307 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:11:21.632137 4405 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e6cb6771-a9e6-452f-a0b6-30ffb955feb8" containerName="istio-proxy" Nov 04 03:11:21.632307 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:11:21.632147 4405 cpu_manager.go:414] "containerMap: removing container" podUID="fd747189-5360-4360-83cd-07a4e414fda4" containerName="istio-proxy" Nov 04 03:11:21.632307 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:11:21.632152 4405 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fd747189-5360-4360-83cd-07a4e414fda4" containerName="istio-proxy" Nov 04 03:11:21.632307 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:11:21.632162 4405 cpu_manager.go:414] "containerMap: removing container" podUID="8a0008f3-4ab9-4ceb-8feb-0f536776610a" containerName="istio-proxy" Nov 04 03:11:21.632307 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:11:21.632167 4405 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8a0008f3-4ab9-4ceb-8feb-0f536776610a" containerName="istio-proxy" Nov 04 03:11:21.632307 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:11:21.632183 4405 cpu_manager.go:414] "containerMap: removing container" podUID="2b2ab722-d1bb-4376-bede-f5f4ee4562aa" containerName="main" Nov 04 03:11:21.632307 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:11:21.632188 4405 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2b2ab722-d1bb-4376-bede-f5f4ee4562aa" containerName="main" Nov 04 03:11:21.632307 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:11:21.632201 4405 cpu_manager.go:414] "containerMap: removing container" podUID="97c488a6-ee10-46bf-bc00-ef8578040f88" containerName="maas-api" Nov 04 03:11:21.632307 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:11:21.632205 4405 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="97c488a6-ee10-46bf-bc00-ef8578040f88" containerName="maas-api" Nov 04 03:11:21.632307 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:11:21.632214 4405 cpu_manager.go:414] "containerMap: removing container" podUID="7d5dc4d7-ea24-4845-8087-1e999e739f22" containerName="payload-processing" Nov 04 03:11:21.632307 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:11:21.632219 4405 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7d5dc4d7-ea24-4845-8087-1e999e739f22" containerName="payload-processing" Nov 04 03:11:21.632307 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:11:21.632298 4405 memory_manager.go:365] "RemoveStaleState removing state" podUID="8842ad09-22ff-4492-ba5d-d4cacf4eac75" containerName="main" Nov 04 03:11:21.632307 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:11:21.632308 4405 memory_manager.go:365] "RemoveStaleState removing state" podUID="8a0008f3-4ab9-4ceb-8feb-0f536776610a" containerName="istio-proxy" Nov 04 03:11:21.632801 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:11:21.632314 4405 memory_manager.go:365] "RemoveStaleState removing state" podUID="43f2feb2-2005-4eca-a400-0d66bd1a4aac" containerName="maas-api" Nov 04 03:11:21.632801 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:11:21.632320 4405 memory_manager.go:365] "RemoveStaleState removing state" podUID="fd747189-5360-4360-83cd-07a4e414fda4" containerName="istio-proxy" Nov 04 03:11:21.632801 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:11:21.632325 4405 memory_manager.go:365] "RemoveStaleState removing state" podUID="e6cb6771-a9e6-452f-a0b6-30ffb955feb8" containerName="istio-proxy" Nov 04 03:11:21.632801 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:11:21.632331 4405 memory_manager.go:365] "RemoveStaleState removing state" podUID="fd747189-5360-4360-83cd-07a4e414fda4" containerName="istio-proxy" Nov 04 03:11:21.632801 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:11:21.632337 4405 memory_manager.go:365] "RemoveStaleState removing state" podUID="8a0008f3-4ab9-4ceb-8feb-0f536776610a" containerName="istio-proxy" Nov 04 03:11:21.632801 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:11:21.632344 4405 memory_manager.go:365] "RemoveStaleState removing state" podUID="2b2ab722-d1bb-4376-bede-f5f4ee4562aa" containerName="main" Nov 04 03:11:21.632801 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:11:21.632349 4405 memory_manager.go:365] "RemoveStaleState removing state" podUID="97c488a6-ee10-46bf-bc00-ef8578040f88" containerName="maas-api" Nov 04 03:11:21.632801 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:11:21.632354 4405 memory_manager.go:365] "RemoveStaleState removing state" podUID="7d5dc4d7-ea24-4845-8087-1e999e739f22" containerName="payload-processing" Nov 04 03:11:21.636071 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:11:21.636052 4405 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-095081-bd58cc787-xz5kc" Nov 04 03:11:21.692644 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:11:21.692601 4405 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-095081-bd58cc787-xz5kc"] Nov 04 03:11:21.697936 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:11:21.697898 4405 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3eb56d77-29c0-4a72-863d-354d2cb764a0-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-095081-bd58cc787-xz5kc\" (UID: \"3eb56d77-29c0-4a72-863d-354d2cb764a0\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-095081-bd58cc787-xz5kc" Nov 04 03:11:21.698119 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:11:21.697995 4405 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vqvjz\" (UniqueName: \"kubernetes.io/projected/3eb56d77-29c0-4a72-863d-354d2cb764a0-kube-api-access-vqvjz\") pod \"payload-pre-processing-e2e-inf-b-095081-bd58cc787-xz5kc\" (UID: \"3eb56d77-29c0-4a72-863d-354d2cb764a0\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-095081-bd58cc787-xz5kc" Nov 04 03:11:21.738507 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:11:21.738470 4405 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-b-095081-dockercfg-h2np2\"" Nov 04 03:11:21.738994 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:11:21.738973 4405 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-b-095081\"" Nov 04 03:11:21.784565 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:11:21.784525 4405 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-095081-6c6bf65b7b-rdq44"] Nov 04 03:11:21.816633 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:11:21.816597 4405 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-095081-6c6bf65b7b-rdq44" Nov 04 03:11:21.821225 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:11:21.818386 4405 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3eb56d77-29c0-4a72-863d-354d2cb764a0-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-095081-bd58cc787-xz5kc\" (UID: \"3eb56d77-29c0-4a72-863d-354d2cb764a0\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-095081-bd58cc787-xz5kc" Nov 04 03:11:21.821360 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:11:21.818808 4405 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3eb56d77-29c0-4a72-863d-354d2cb764a0-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-095081-bd58cc787-xz5kc\" (UID: \"3eb56d77-29c0-4a72-863d-354d2cb764a0\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-095081-bd58cc787-xz5kc" Nov 04 03:11:21.821360 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:11:21.821284 4405 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qp4qw\" (UniqueName: \"kubernetes.io/projected/a6f009b4-0c0b-4652-812f-a7d3333f3880-kube-api-access-qp4qw\") pod \"payload-processing-e2e-inf-b-095081-6c6bf65b7b-rdq44\" (UID: \"a6f009b4-0c0b-4652-812f-a7d3333f3880\") " pod="openshift-ingress/payload-processing-e2e-inf-b-095081-6c6bf65b7b-rdq44" Nov 04 03:11:21.821360 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:11:21.821350 4405 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vqvjz\" (UniqueName: \"kubernetes.io/projected/3eb56d77-29c0-4a72-863d-354d2cb764a0-kube-api-access-vqvjz\") pod \"payload-pre-processing-e2e-inf-b-095081-bd58cc787-xz5kc\" (UID: \"3eb56d77-29c0-4a72-863d-354d2cb764a0\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-095081-bd58cc787-xz5kc" Nov 04 03:11:21.821550 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:11:21.821528 4405 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a6f009b4-0c0b-4652-812f-a7d3333f3880-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-095081-6c6bf65b7b-rdq44\" (UID: \"a6f009b4-0c0b-4652-812f-a7d3333f3880\") " pod="openshift-ingress/payload-processing-e2e-inf-b-095081-6c6bf65b7b-rdq44" Nov 04 03:11:21.843784 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:11:21.843740 4405 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-095081-6c6bf65b7b-rdq44"] Nov 04 03:11:21.882371 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:11:21.882290 4405 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vqvjz\" (UniqueName: \"kubernetes.io/projected/3eb56d77-29c0-4a72-863d-354d2cb764a0-kube-api-access-vqvjz\") pod \"payload-pre-processing-e2e-inf-b-095081-bd58cc787-xz5kc\" (UID: \"3eb56d77-29c0-4a72-863d-354d2cb764a0\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-095081-bd58cc787-xz5kc" Nov 04 03:11:21.922420 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:11:21.922382 4405 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a6f009b4-0c0b-4652-812f-a7d3333f3880-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-095081-6c6bf65b7b-rdq44\" (UID: \"a6f009b4-0c0b-4652-812f-a7d3333f3880\") " pod="openshift-ingress/payload-processing-e2e-inf-b-095081-6c6bf65b7b-rdq44" Nov 04 03:11:21.922619 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:11:21.922500 4405 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qp4qw\" (UniqueName: \"kubernetes.io/projected/a6f009b4-0c0b-4652-812f-a7d3333f3880-kube-api-access-qp4qw\") pod \"payload-processing-e2e-inf-b-095081-6c6bf65b7b-rdq44\" (UID: \"a6f009b4-0c0b-4652-812f-a7d3333f3880\") " pod="openshift-ingress/payload-processing-e2e-inf-b-095081-6c6bf65b7b-rdq44" Nov 04 03:11:21.923295 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:11:21.923266 4405 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a6f009b4-0c0b-4652-812f-a7d3333f3880-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-095081-6c6bf65b7b-rdq44\" (UID: \"a6f009b4-0c0b-4652-812f-a7d3333f3880\") " pod="openshift-ingress/payload-processing-e2e-inf-b-095081-6c6bf65b7b-rdq44" Nov 04 03:11:21.949174 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:11:21.949139 4405 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-095081-bd58cc787-xz5kc" Nov 04 03:11:21.983724 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:11:21.983694 4405 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qp4qw\" (UniqueName: \"kubernetes.io/projected/a6f009b4-0c0b-4652-812f-a7d3333f3880-kube-api-access-qp4qw\") pod \"payload-processing-e2e-inf-b-095081-6c6bf65b7b-rdq44\" (UID: \"a6f009b4-0c0b-4652-812f-a7d3333f3880\") " pod="openshift-ingress/payload-processing-e2e-inf-b-095081-6c6bf65b7b-rdq44" Nov 04 03:11:22.135974 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:11:22.135864 4405 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-095081-6c6bf65b7b-rdq44" Nov 04 03:11:22.332155 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:11:22.332125 4405 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-095081-bd58cc787-xz5kc"] Nov 04 03:11:22.333554 ip-10-0-134-125 kubenswrapper[4405]: W1104 03:11:22.333521 4405 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3eb56d77_29c0_4a72_863d_354d2cb764a0.slice/crio-c3b82e72aefd1a76a6ce8a5c671fefd798211e8dafabac17809a5d6b94211499 WatchSource:0}: Error finding container c3b82e72aefd1a76a6ce8a5c671fefd798211e8dafabac17809a5d6b94211499: Status 404 returned error can't find the container with id c3b82e72aefd1a76a6ce8a5c671fefd798211e8dafabac17809a5d6b94211499 Nov 04 03:11:22.526050 ip-10-0-134-125 kubenswrapper[4405]: W1104 03:11:22.526014 4405 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda6f009b4_0c0b_4652_812f_a7d3333f3880.slice/crio-4065c260e2f0a317e6667660d1bb84218fcf338289171feeb8d79b1a47b9ef91 WatchSource:0}: Error finding container 4065c260e2f0a317e6667660d1bb84218fcf338289171feeb8d79b1a47b9ef91: Status 404 returned error can't find the container with id 4065c260e2f0a317e6667660d1bb84218fcf338289171feeb8d79b1a47b9ef91 Nov 04 03:11:22.532480 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:11:22.532248 4405 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-095081-6c6bf65b7b-rdq44"] Nov 04 03:11:23.045691 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:11:23.042847 4405 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-095081-6c6bf65b7b-rdq44" event={"ID":"a6f009b4-0c0b-4652-812f-a7d3333f3880","Type":"ContainerStarted","Data":"a41597d148d56db195437cd58783bce0bf781c5694672f6f5389d6ac4bf1b751"} Nov 04 03:11:23.045691 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:11:23.042923 4405 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-095081-6c6bf65b7b-rdq44" event={"ID":"a6f009b4-0c0b-4652-812f-a7d3333f3880","Type":"ContainerStarted","Data":"4065c260e2f0a317e6667660d1bb84218fcf338289171feeb8d79b1a47b9ef91"} Nov 04 03:11:23.045691 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:11:23.043995 4405 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-b-095081-6c6bf65b7b-rdq44" Nov 04 03:11:23.045691 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:11:23.045284 4405 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-095081-bd58cc787-xz5kc" event={"ID":"3eb56d77-29c0-4a72-863d-354d2cb764a0","Type":"ContainerStarted","Data":"661b7e606486e657909844b5e698bca6cd782df0cfdfc06b6b55264c70e08797"} Nov 04 03:11:23.045691 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:11:23.045331 4405 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-095081-bd58cc787-xz5kc" event={"ID":"3eb56d77-29c0-4a72-863d-354d2cb764a0","Type":"ContainerStarted","Data":"c3b82e72aefd1a76a6ce8a5c671fefd798211e8dafabac17809a5d6b94211499"} Nov 04 03:11:23.045691 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:11:23.045364 4405 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-095081-bd58cc787-xz5kc" Nov 04 03:11:23.159418 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:11:23.159352 4405 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-b-095081-6c6bf65b7b-rdq44" podStartSLOduration=2.15933447 podStartE2EDuration="2.15933447s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 03:11:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 03:11:23.158375664 +0000 UTC m=+2329.589075785" watchObservedRunningTime="2026-11-04 03:11:23.15933447 +0000 UTC m=+2329.590034588" Nov 04 03:11:23.278512 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:11:23.278447 4405 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-095081-bd58cc787-xz5kc" podStartSLOduration=2.2784281760000002 podStartE2EDuration="2.278428176s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 03:11:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 03:11:23.275379386 +0000 UTC m=+2329.706079505" watchObservedRunningTime="2026-11-04 03:11:23.278428176 +0000 UTC m=+2329.709128297" Nov 04 03:11:26.572279 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:11:26.572236 4405 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w6-ebb7fd/distinct-2-w6-ebb7fd-kserve-6f68c5db86-zrpjs" podUID="b7b786e0-9c94-4126-9949-cd354ebbfb2a" containerName="main" containerID="cri-o://e2f3c926f88fc03a1997095f2ab36a48350313e4f58c734bfc2571d1daa8689d" gracePeriod=45 Nov 04 03:11:26.592547 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:11:26.586132 4405 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-ebb7fd/distinct-2-w6-ebb7fd-kserve-6f68c5db86-zrpjs" podUID="b7b786e0-9c94-4126-9949-cd354ebbfb2a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.65:8000/ready\": EOF" Nov 04 03:11:26.623489 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:11:26.623110 4405 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w6-ebb7fd/distinct-w6-ebb7fd-kserve-55c47d8865-7z9pz" podUID="ccb8e181-5610-46da-bb8b-e0b07fa02ff1" containerName="main" containerID="cri-o://33f579eb8d27949bb4ba12ebe84d9ec19d482af4a3509bf9dffe01fb714f22d3" gracePeriod=45 Nov 04 03:11:27.041928 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:11:27.041890 4405 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-ebb7fd/distinct-2-w6-ebb7fd-kserve-6f68c5db86-zrpjs" Nov 04 03:11:27.078709 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:11:27.077953 4405 generic.go:356] "Generic (PLEG): container finished" podID="b7b786e0-9c94-4126-9949-cd354ebbfb2a" containerID="e2f3c926f88fc03a1997095f2ab36a48350313e4f58c734bfc2571d1daa8689d" exitCode=0 Nov 04 03:11:27.078709 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:11:27.078082 4405 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-ebb7fd/distinct-2-w6-ebb7fd-kserve-6f68c5db86-zrpjs" event={"ID":"b7b786e0-9c94-4126-9949-cd354ebbfb2a","Type":"ContainerDied","Data":"e2f3c926f88fc03a1997095f2ab36a48350313e4f58c734bfc2571d1daa8689d"} Nov 04 03:11:27.078709 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:11:27.078122 4405 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-ebb7fd/distinct-2-w6-ebb7fd-kserve-6f68c5db86-zrpjs" event={"ID":"b7b786e0-9c94-4126-9949-cd354ebbfb2a","Type":"ContainerDied","Data":"5213d798ca18714e389301b7ab15565fa0279322501a0ee360aca5f464cde233"} Nov 04 03:11:27.078709 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:11:27.078145 4405 scope.go:122] "RemoveContainer" containerID="e2f3c926f88fc03a1997095f2ab36a48350313e4f58c734bfc2571d1daa8689d" Nov 04 03:11:27.078709 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:11:27.078288 4405 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-ebb7fd/distinct-2-w6-ebb7fd-kserve-6f68c5db86-zrpjs" Nov 04 03:11:27.130072 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:11:27.130042 4405 generic.go:356] "Generic (PLEG): container finished" podID="ccb8e181-5610-46da-bb8b-e0b07fa02ff1" containerID="33f579eb8d27949bb4ba12ebe84d9ec19d482af4a3509bf9dffe01fb714f22d3" exitCode=0 Nov 04 03:11:27.130333 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:11:27.130312 4405 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b7b786e0-9c94-4126-9949-cd354ebbfb2a-dshm\" (UniqueName: \"kubernetes.io/empty-dir/b7b786e0-9c94-4126-9949-cd354ebbfb2a-dshm\") pod \"b7b786e0-9c94-4126-9949-cd354ebbfb2a\" (UID: \"b7b786e0-9c94-4126-9949-cd354ebbfb2a\") " Nov 04 03:11:27.130697 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:11:27.130643 4405 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b7b786e0-9c94-4126-9949-cd354ebbfb2a-tls-certs\" (UniqueName: \"kubernetes.io/secret/b7b786e0-9c94-4126-9949-cd354ebbfb2a-tls-certs\") pod \"b7b786e0-9c94-4126-9949-cd354ebbfb2a\" (UID: \"b7b786e0-9c94-4126-9949-cd354ebbfb2a\") " Nov 04 03:11:27.131285 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:11:27.131264 4405 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b7b786e0-9c94-4126-9949-cd354ebbfb2a-kube-api-access-krcb7\" (UniqueName: \"kubernetes.io/projected/b7b786e0-9c94-4126-9949-cd354ebbfb2a-kube-api-access-krcb7\") pod \"b7b786e0-9c94-4126-9949-cd354ebbfb2a\" (UID: \"b7b786e0-9c94-4126-9949-cd354ebbfb2a\") " Nov 04 03:11:27.131463 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:11:27.131446 4405 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b7b786e0-9c94-4126-9949-cd354ebbfb2a-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b7b786e0-9c94-4126-9949-cd354ebbfb2a-model-cache\") pod \"b7b786e0-9c94-4126-9949-cd354ebbfb2a\" (UID: \"b7b786e0-9c94-4126-9949-cd354ebbfb2a\") " Nov 04 03:11:27.131637 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:11:27.131620 4405 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b7b786e0-9c94-4126-9949-cd354ebbfb2a-home\" (UniqueName: \"kubernetes.io/empty-dir/b7b786e0-9c94-4126-9949-cd354ebbfb2a-home\") pod \"b7b786e0-9c94-4126-9949-cd354ebbfb2a\" (UID: \"b7b786e0-9c94-4126-9949-cd354ebbfb2a\") " Nov 04 03:11:27.131797 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:11:27.130417 4405 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-ebb7fd/distinct-w6-ebb7fd-kserve-55c47d8865-7z9pz" event={"ID":"ccb8e181-5610-46da-bb8b-e0b07fa02ff1","Type":"ContainerDied","Data":"33f579eb8d27949bb4ba12ebe84d9ec19d482af4a3509bf9dffe01fb714f22d3"} Nov 04 03:11:27.131878 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:11:27.131829 4405 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-ebb7fd/distinct-w6-ebb7fd-kserve-55c47d8865-7z9pz" event={"ID":"ccb8e181-5610-46da-bb8b-e0b07fa02ff1","Type":"ContainerDied","Data":"61710beb52c66594a12b82ff262456f27fd09a6e2fa6ada0356aea5c0d572844"} Nov 04 03:11:27.131878 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:11:27.131854 4405 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="61710beb52c66594a12b82ff262456f27fd09a6e2fa6ada0356aea5c0d572844" Nov 04 03:11:27.132078 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:11:27.132061 4405 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b7b786e0-9c94-4126-9949-cd354ebbfb2a-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b7b786e0-9c94-4126-9949-cd354ebbfb2a-tmp-dir\") pod \"b7b786e0-9c94-4126-9949-cd354ebbfb2a\" (UID: \"b7b786e0-9c94-4126-9949-cd354ebbfb2a\") " Nov 04 03:11:27.132431 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:11:27.132391 4405 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b7b786e0-9c94-4126-9949-cd354ebbfb2a-model-cache" pod "b7b786e0-9c94-4126-9949-cd354ebbfb2a" (UID: "b7b786e0-9c94-4126-9949-cd354ebbfb2a"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 03:11:27.132852 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:11:27.132832 4405 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b7b786e0-9c94-4126-9949-cd354ebbfb2a-model-cache\") on node \"ip-10-0-134-125.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:11:27.133022 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:11:27.133002 4405 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b7b786e0-9c94-4126-9949-cd354ebbfb2a-home" pod "b7b786e0-9c94-4126-9949-cd354ebbfb2a" (UID: "b7b786e0-9c94-4126-9949-cd354ebbfb2a"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 03:11:27.133156 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:11:27.133130 4405 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b7b786e0-9c94-4126-9949-cd354ebbfb2a-tmp-dir" pod "b7b786e0-9c94-4126-9949-cd354ebbfb2a" (UID: "b7b786e0-9c94-4126-9949-cd354ebbfb2a"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 03:11:27.139704 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:11:27.138270 4405 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b7b786e0-9c94-4126-9949-cd354ebbfb2a-tls-certs" pod "b7b786e0-9c94-4126-9949-cd354ebbfb2a" (UID: "b7b786e0-9c94-4126-9949-cd354ebbfb2a"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 04 03:11:27.139704 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:11:27.138377 4405 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b7b786e0-9c94-4126-9949-cd354ebbfb2a-dshm" pod "b7b786e0-9c94-4126-9949-cd354ebbfb2a" (UID: "b7b786e0-9c94-4126-9949-cd354ebbfb2a"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 03:11:27.139704 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:11:27.138892 4405 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b7b786e0-9c94-4126-9949-cd354ebbfb2a-kube-api-access-krcb7" pod "b7b786e0-9c94-4126-9949-cd354ebbfb2a" (UID: "b7b786e0-9c94-4126-9949-cd354ebbfb2a"). InnerVolumeSpecName "kube-api-access-krcb7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 03:11:27.141910 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:11:27.141891 4405 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-ebb7fd/distinct-w6-ebb7fd-kserve-55c47d8865-7z9pz" Nov 04 03:11:27.147057 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:11:27.147035 4405 scope.go:122] "RemoveContainer" containerID="e2f3c926f88fc03a1997095f2ab36a48350313e4f58c734bfc2571d1daa8689d" Nov 04 03:11:27.147552 ip-10-0-134-125 kubenswrapper[4405]: E1104 03:11:27.147526 4405 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e2f3c926f88fc03a1997095f2ab36a48350313e4f58c734bfc2571d1daa8689d\": container with ID starting with e2f3c926f88fc03a1997095f2ab36a48350313e4f58c734bfc2571d1daa8689d not found: ID does not exist" containerID="e2f3c926f88fc03a1997095f2ab36a48350313e4f58c734bfc2571d1daa8689d" Nov 04 03:11:27.147745 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:11:27.147716 4405 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e2f3c926f88fc03a1997095f2ab36a48350313e4f58c734bfc2571d1daa8689d"} err="failed to get container status \"e2f3c926f88fc03a1997095f2ab36a48350313e4f58c734bfc2571d1daa8689d\": rpc error: code = NotFound desc = could not find container \"e2f3c926f88fc03a1997095f2ab36a48350313e4f58c734bfc2571d1daa8689d\": container with ID starting with e2f3c926f88fc03a1997095f2ab36a48350313e4f58c734bfc2571d1daa8689d not found: ID does not exist" Nov 04 03:11:27.234239 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:11:27.234197 4405 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ccb8e181-5610-46da-bb8b-e0b07fa02ff1-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ccb8e181-5610-46da-bb8b-e0b07fa02ff1-model-cache\") pod \"ccb8e181-5610-46da-bb8b-e0b07fa02ff1\" (UID: \"ccb8e181-5610-46da-bb8b-e0b07fa02ff1\") " Nov 04 03:11:27.240919 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:11:27.240877 4405 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ccb8e181-5610-46da-bb8b-e0b07fa02ff1-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ccb8e181-5610-46da-bb8b-e0b07fa02ff1-tmp-dir\") pod \"ccb8e181-5610-46da-bb8b-e0b07fa02ff1\" (UID: \"ccb8e181-5610-46da-bb8b-e0b07fa02ff1\") " Nov 04 03:11:27.241562 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:11:27.240754 4405 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ccb8e181-5610-46da-bb8b-e0b07fa02ff1-model-cache" pod "ccb8e181-5610-46da-bb8b-e0b07fa02ff1" (UID: "ccb8e181-5610-46da-bb8b-e0b07fa02ff1"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 03:11:27.243397 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:11:27.241357 4405 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ccb8e181-5610-46da-bb8b-e0b07fa02ff1-tmp-dir" pod "ccb8e181-5610-46da-bb8b-e0b07fa02ff1" (UID: "ccb8e181-5610-46da-bb8b-e0b07fa02ff1"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 03:11:27.243630 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:11:27.243610 4405 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ccb8e181-5610-46da-bb8b-e0b07fa02ff1-home\" (UniqueName: \"kubernetes.io/empty-dir/ccb8e181-5610-46da-bb8b-e0b07fa02ff1-home\") pod \"ccb8e181-5610-46da-bb8b-e0b07fa02ff1\" (UID: \"ccb8e181-5610-46da-bb8b-e0b07fa02ff1\") " Nov 04 03:11:27.244263 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:11:27.244124 4405 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ccb8e181-5610-46da-bb8b-e0b07fa02ff1-home" pod "ccb8e181-5610-46da-bb8b-e0b07fa02ff1" (UID: "ccb8e181-5610-46da-bb8b-e0b07fa02ff1"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 03:11:27.244441 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:11:27.244422 4405 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ccb8e181-5610-46da-bb8b-e0b07fa02ff1-kube-api-access-shknm\" (UniqueName: \"kubernetes.io/projected/ccb8e181-5610-46da-bb8b-e0b07fa02ff1-kube-api-access-shknm\") pod \"ccb8e181-5610-46da-bb8b-e0b07fa02ff1\" (UID: \"ccb8e181-5610-46da-bb8b-e0b07fa02ff1\") " Nov 04 03:11:27.262200 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:11:27.262168 4405 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ccb8e181-5610-46da-bb8b-e0b07fa02ff1-tls-certs\" (UniqueName: \"kubernetes.io/secret/ccb8e181-5610-46da-bb8b-e0b07fa02ff1-tls-certs\") pod \"ccb8e181-5610-46da-bb8b-e0b07fa02ff1\" (UID: \"ccb8e181-5610-46da-bb8b-e0b07fa02ff1\") " Nov 04 03:11:27.262506 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:11:27.262484 4405 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ccb8e181-5610-46da-bb8b-e0b07fa02ff1-dshm\" (UniqueName: \"kubernetes.io/empty-dir/ccb8e181-5610-46da-bb8b-e0b07fa02ff1-dshm\") pod \"ccb8e181-5610-46da-bb8b-e0b07fa02ff1\" (UID: \"ccb8e181-5610-46da-bb8b-e0b07fa02ff1\") " Nov 04 03:11:27.263849 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:11:27.263809 4405 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b7b786e0-9c94-4126-9949-cd354ebbfb2a-tmp-dir\") on node \"ip-10-0-134-125.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:11:27.264020 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:11:27.264005 4405 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b7b786e0-9c94-4126-9949-cd354ebbfb2a-dshm\") on node \"ip-10-0-134-125.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:11:27.264150 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:11:27.264136 4405 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ccb8e181-5610-46da-bb8b-e0b07fa02ff1-model-cache\") on node \"ip-10-0-134-125.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:11:27.264273 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:11:27.264258 4405 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ccb8e181-5610-46da-bb8b-e0b07fa02ff1-tmp-dir\") on node \"ip-10-0-134-125.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:11:27.264389 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:11:27.264377 4405 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ccb8e181-5610-46da-bb8b-e0b07fa02ff1-home\") on node \"ip-10-0-134-125.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:11:27.264594 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:11:27.264571 4405 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b7b786e0-9c94-4126-9949-cd354ebbfb2a-tls-certs\") on node \"ip-10-0-134-125.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:11:27.264825 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:11:27.264798 4405 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-krcb7\" (UniqueName: \"kubernetes.io/projected/b7b786e0-9c94-4126-9949-cd354ebbfb2a-kube-api-access-krcb7\") on node \"ip-10-0-134-125.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:11:27.264968 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:11:27.264953 4405 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b7b786e0-9c94-4126-9949-cd354ebbfb2a-home\") on node \"ip-10-0-134-125.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:11:27.266019 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:11:27.265997 4405 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ccb8e181-5610-46da-bb8b-e0b07fa02ff1-dshm" pod "ccb8e181-5610-46da-bb8b-e0b07fa02ff1" (UID: "ccb8e181-5610-46da-bb8b-e0b07fa02ff1"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 03:11:27.266452 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:11:27.266413 4405 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ccb8e181-5610-46da-bb8b-e0b07fa02ff1-kube-api-access-shknm" pod "ccb8e181-5610-46da-bb8b-e0b07fa02ff1" (UID: "ccb8e181-5610-46da-bb8b-e0b07fa02ff1"). InnerVolumeSpecName "kube-api-access-shknm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 03:11:27.268390 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:11:27.268230 4405 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ccb8e181-5610-46da-bb8b-e0b07fa02ff1-tls-certs" pod "ccb8e181-5610-46da-bb8b-e0b07fa02ff1" (UID: "ccb8e181-5610-46da-bb8b-e0b07fa02ff1"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 04 03:11:27.368163 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:11:27.366361 4405 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ccb8e181-5610-46da-bb8b-e0b07fa02ff1-tls-certs\") on node \"ip-10-0-134-125.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:11:27.368163 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:11:27.366393 4405 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ccb8e181-5610-46da-bb8b-e0b07fa02ff1-dshm\") on node \"ip-10-0-134-125.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:11:27.368163 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:11:27.366409 4405 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-shknm\" (UniqueName: \"kubernetes.io/projected/ccb8e181-5610-46da-bb8b-e0b07fa02ff1-kube-api-access-shknm\") on node \"ip-10-0-134-125.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:11:27.558694 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:11:27.555919 4405 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w6-ebb7fd/distinct-2-w6-ebb7fd-kserve-6f68c5db86-zrpjs"] Nov 04 03:11:27.565696 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:11:27.560881 4405 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w6-ebb7fd/distinct-2-w6-ebb7fd-kserve-6f68c5db86-zrpjs"] Nov 04 03:11:27.756268 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:11:27.756181 4405 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b7b786e0-9c94-4126-9949-cd354ebbfb2a" path="/var/lib/kubelet/pods/b7b786e0-9c94-4126-9949-cd354ebbfb2a/volumes" Nov 04 03:11:28.137167 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:11:28.137126 4405 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-ebb7fd/distinct-w6-ebb7fd-kserve-55c47d8865-7z9pz" Nov 04 03:11:28.301826 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:11:28.301790 4405 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w6-ebb7fd/distinct-w6-ebb7fd-kserve-55c47d8865-7z9pz"] Nov 04 03:11:28.351535 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:11:28.351506 4405 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w6-ebb7fd/distinct-w6-ebb7fd-kserve-55c47d8865-7z9pz"] Nov 04 03:11:29.354340 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:11:29.354305 4405 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w6-ebb7fd-b8fd547bb-7jkvh"] Nov 04 03:11:29.609804 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:11:29.609725 4405 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w3-cdcf67/distinct-w3-cdcf67-kserve-667cc7f7d7-5m2lq" podUID="52ed22cc-96e4-4ef0-b34d-ed58c89a9183" containerName="main" containerID="cri-o://bf8c65e09b476c923b798d5224fc5e9e0b4a06a3dbd06904a953f30cddbc6571" gracePeriod=45 Nov 04 03:11:29.701118 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:11:29.701085 4405 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ccb8e181-5610-46da-bb8b-e0b07fa02ff1" path="/var/lib/kubelet/pods/ccb8e181-5610-46da-bb8b-e0b07fa02ff1/volumes" Nov 04 03:11:29.891440 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:11:29.891414 4405 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-cdcf67/distinct-w3-cdcf67-kserve-667cc7f7d7-5m2lq" Nov 04 03:11:29.915732 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:11:29.915706 4405 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/52ed22cc-96e4-4ef0-b34d-ed58c89a9183-tls-certs\" (UniqueName: \"kubernetes.io/secret/52ed22cc-96e4-4ef0-b34d-ed58c89a9183-tls-certs\") pod \"52ed22cc-96e4-4ef0-b34d-ed58c89a9183\" (UID: \"52ed22cc-96e4-4ef0-b34d-ed58c89a9183\") " Nov 04 03:11:29.915840 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:11:29.915745 4405 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/52ed22cc-96e4-4ef0-b34d-ed58c89a9183-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/52ed22cc-96e4-4ef0-b34d-ed58c89a9183-tmp-dir\") pod \"52ed22cc-96e4-4ef0-b34d-ed58c89a9183\" (UID: \"52ed22cc-96e4-4ef0-b34d-ed58c89a9183\") " Nov 04 03:11:29.915840 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:11:29.915774 4405 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/52ed22cc-96e4-4ef0-b34d-ed58c89a9183-dshm\" (UniqueName: \"kubernetes.io/empty-dir/52ed22cc-96e4-4ef0-b34d-ed58c89a9183-dshm\") pod \"52ed22cc-96e4-4ef0-b34d-ed58c89a9183\" (UID: \"52ed22cc-96e4-4ef0-b34d-ed58c89a9183\") " Nov 04 03:11:29.915962 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:11:29.915863 4405 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/52ed22cc-96e4-4ef0-b34d-ed58c89a9183-home\" (UniqueName: \"kubernetes.io/empty-dir/52ed22cc-96e4-4ef0-b34d-ed58c89a9183-home\") pod \"52ed22cc-96e4-4ef0-b34d-ed58c89a9183\" (UID: \"52ed22cc-96e4-4ef0-b34d-ed58c89a9183\") " Nov 04 03:11:29.915962 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:11:29.915892 4405 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/52ed22cc-96e4-4ef0-b34d-ed58c89a9183-kube-api-access-pb5bk\" (UniqueName: \"kubernetes.io/projected/52ed22cc-96e4-4ef0-b34d-ed58c89a9183-kube-api-access-pb5bk\") pod \"52ed22cc-96e4-4ef0-b34d-ed58c89a9183\" (UID: \"52ed22cc-96e4-4ef0-b34d-ed58c89a9183\") " Nov 04 03:11:29.915962 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:11:29.915929 4405 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/52ed22cc-96e4-4ef0-b34d-ed58c89a9183-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/52ed22cc-96e4-4ef0-b34d-ed58c89a9183-model-cache\") pod \"52ed22cc-96e4-4ef0-b34d-ed58c89a9183\" (UID: \"52ed22cc-96e4-4ef0-b34d-ed58c89a9183\") " Nov 04 03:11:29.916127 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:11:29.915992 4405 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/52ed22cc-96e4-4ef0-b34d-ed58c89a9183-tmp-dir" pod "52ed22cc-96e4-4ef0-b34d-ed58c89a9183" (UID: "52ed22cc-96e4-4ef0-b34d-ed58c89a9183"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 03:11:29.916127 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:11:29.916100 4405 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/52ed22cc-96e4-4ef0-b34d-ed58c89a9183-home" pod "52ed22cc-96e4-4ef0-b34d-ed58c89a9183" (UID: "52ed22cc-96e4-4ef0-b34d-ed58c89a9183"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 03:11:29.916220 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:11:29.916198 4405 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/52ed22cc-96e4-4ef0-b34d-ed58c89a9183-model-cache" pod "52ed22cc-96e4-4ef0-b34d-ed58c89a9183" (UID: "52ed22cc-96e4-4ef0-b34d-ed58c89a9183"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 03:11:29.916497 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:11:29.916468 4405 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/52ed22cc-96e4-4ef0-b34d-ed58c89a9183-tmp-dir\") on node \"ip-10-0-134-125.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:11:29.916497 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:11:29.916492 4405 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/52ed22cc-96e4-4ef0-b34d-ed58c89a9183-home\") on node \"ip-10-0-134-125.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:11:29.916707 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:11:29.916504 4405 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/52ed22cc-96e4-4ef0-b34d-ed58c89a9183-model-cache\") on node \"ip-10-0-134-125.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:11:29.918208 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:11:29.918181 4405 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/52ed22cc-96e4-4ef0-b34d-ed58c89a9183-dshm" pod "52ed22cc-96e4-4ef0-b34d-ed58c89a9183" (UID: "52ed22cc-96e4-4ef0-b34d-ed58c89a9183"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 03:11:29.918298 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:11:29.918262 4405 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/52ed22cc-96e4-4ef0-b34d-ed58c89a9183-tls-certs" pod "52ed22cc-96e4-4ef0-b34d-ed58c89a9183" (UID: "52ed22cc-96e4-4ef0-b34d-ed58c89a9183"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 04 03:11:29.918337 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:11:29.918300 4405 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/52ed22cc-96e4-4ef0-b34d-ed58c89a9183-kube-api-access-pb5bk" pod "52ed22cc-96e4-4ef0-b34d-ed58c89a9183" (UID: "52ed22cc-96e4-4ef0-b34d-ed58c89a9183"). InnerVolumeSpecName "kube-api-access-pb5bk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 03:11:30.018002 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:11:30.017960 4405 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/52ed22cc-96e4-4ef0-b34d-ed58c89a9183-tls-certs\") on node \"ip-10-0-134-125.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:11:30.018002 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:11:30.017990 4405 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/52ed22cc-96e4-4ef0-b34d-ed58c89a9183-dshm\") on node \"ip-10-0-134-125.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:11:30.018002 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:11:30.018006 4405 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pb5bk\" (UniqueName: \"kubernetes.io/projected/52ed22cc-96e4-4ef0-b34d-ed58c89a9183-kube-api-access-pb5bk\") on node \"ip-10-0-134-125.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:11:30.151588 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:11:30.151496 4405 generic.go:356] "Generic (PLEG): container finished" podID="52ed22cc-96e4-4ef0-b34d-ed58c89a9183" containerID="bf8c65e09b476c923b798d5224fc5e9e0b4a06a3dbd06904a953f30cddbc6571" exitCode=0 Nov 04 03:11:30.151588 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:11:30.151547 4405 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-cdcf67/distinct-w3-cdcf67-kserve-667cc7f7d7-5m2lq" Nov 04 03:11:30.151588 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:11:30.151554 4405 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-cdcf67/distinct-w3-cdcf67-kserve-667cc7f7d7-5m2lq" event={"ID":"52ed22cc-96e4-4ef0-b34d-ed58c89a9183","Type":"ContainerDied","Data":"bf8c65e09b476c923b798d5224fc5e9e0b4a06a3dbd06904a953f30cddbc6571"} Nov 04 03:11:30.151916 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:11:30.151594 4405 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-cdcf67/distinct-w3-cdcf67-kserve-667cc7f7d7-5m2lq" event={"ID":"52ed22cc-96e4-4ef0-b34d-ed58c89a9183","Type":"ContainerDied","Data":"18102e2aa5eea61d2882013dfcb2d162e34b669e9d004cab6c90b3c3c8225a2f"} Nov 04 03:11:30.151916 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:11:30.151619 4405 scope.go:122] "RemoveContainer" containerID="bf8c65e09b476c923b798d5224fc5e9e0b4a06a3dbd06904a953f30cddbc6571" Nov 04 03:11:30.163953 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:11:30.163929 4405 scope.go:122] "RemoveContainer" containerID="bf8c65e09b476c923b798d5224fc5e9e0b4a06a3dbd06904a953f30cddbc6571" Nov 04 03:11:30.164274 ip-10-0-134-125 kubenswrapper[4405]: E1104 03:11:30.164251 4405 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bf8c65e09b476c923b798d5224fc5e9e0b4a06a3dbd06904a953f30cddbc6571\": container with ID starting with bf8c65e09b476c923b798d5224fc5e9e0b4a06a3dbd06904a953f30cddbc6571 not found: ID does not exist" containerID="bf8c65e09b476c923b798d5224fc5e9e0b4a06a3dbd06904a953f30cddbc6571" Nov 04 03:11:30.164353 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:11:30.164290 4405 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bf8c65e09b476c923b798d5224fc5e9e0b4a06a3dbd06904a953f30cddbc6571"} err="failed to get container status \"bf8c65e09b476c923b798d5224fc5e9e0b4a06a3dbd06904a953f30cddbc6571\": rpc error: code = NotFound desc = could not find container \"bf8c65e09b476c923b798d5224fc5e9e0b4a06a3dbd06904a953f30cddbc6571\": container with ID starting with bf8c65e09b476c923b798d5224fc5e9e0b4a06a3dbd06904a953f30cddbc6571 not found: ID does not exist" Nov 04 03:11:30.319505 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:11:30.319465 4405 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-cdcf67/distinct-w3-cdcf67-kserve-667cc7f7d7-5m2lq"] Nov 04 03:11:30.323538 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:11:30.323504 4405 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w3-cdcf67/distinct-w3-cdcf67-kserve-667cc7f7d7-5m2lq"] Nov 04 03:11:30.889693 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:11:30.881203 4405 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-cdcf67/distinct-w3-cdcf67-kserve-667cc7f7d7-5m2lq" podUID="52ed22cc-96e4-4ef0-b34d-ed58c89a9183" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 03:11:31.701943 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:11:31.701899 4405 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="52ed22cc-96e4-4ef0-b34d-ed58c89a9183" path="/var/lib/kubelet/pods/52ed22cc-96e4-4ef0-b34d-ed58c89a9183/volumes" Nov 04 03:11:31.763618 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:11:31.763584 4405 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-fe4e0c-openshift-default-c7f4fc84c-6hhrb"] Nov 04 03:11:31.763820 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:11:31.763797 4405 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-fe4e0c-openshift-default-c7f4fc84c-6hhrb" podUID="40a233af-2ad7-4c33-85b2-a2e251d7ea3e" containerName="istio-proxy" containerID="cri-o://31d61c1873e1df2d29db741027dcf73fcb3dd3c14329a064a7f68b47cd9b0557" gracePeriod=30 Nov 04 03:11:33.244148 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:11:33.244112 4405 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-e516ff-openshift-default-6f4b957485-xg2tv"] Nov 04 03:11:33.244659 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:11:33.244412 4405 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-e516ff-openshift-default-6f4b957485-xg2tv" podUID="a4bcafa2-ec97-486c-819f-7be10e3110a9" containerName="istio-proxy" containerID="cri-o://5ffa0fcee1b9de3f3ef3bb807b85ed515c57d72a9041a34d52c80bf8ce43b404" gracePeriod=30 Nov 04 03:11:34.055042 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:11:34.055014 4405 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-095081-bd58cc787-xz5kc" Nov 04 03:11:34.314250 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:11:34.314155 4405 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w3-cdcf67-76579b7df4-vqbkj"] Nov 04 03:11:34.354633 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:11:34.354558 4405 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w6-ebb7fd-b8fd547bb-7jkvh" podUID="af244918-878e-430e-a893-aac69d2498ef" containerName="payload-processing" containerID="cri-o://ac88d64eb459108f07130e51fce895960a73ba7d8556f333dd1abf4501429d23" gracePeriod=25 Nov 04 03:11:34.627383 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:11:34.627352 4405 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w6-ebb7fd-b8fd547bb-7jkvh" Nov 04 03:11:34.673871 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:11:34.673835 4405 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/af244918-878e-430e-a893-aac69d2498ef-kube-api-access-dcgv6\" (UniqueName: \"kubernetes.io/projected/af244918-878e-430e-a893-aac69d2498ef-kube-api-access-dcgv6\") pod \"af244918-878e-430e-a893-aac69d2498ef\" (UID: \"af244918-878e-430e-a893-aac69d2498ef\") " Nov 04 03:11:34.674037 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:11:34.673896 4405 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/af244918-878e-430e-a893-aac69d2498ef-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/af244918-878e-430e-a893-aac69d2498ef-plugins-config-volume\") pod \"af244918-878e-430e-a893-aac69d2498ef\" (UID: \"af244918-878e-430e-a893-aac69d2498ef\") " Nov 04 03:11:34.674261 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:11:34.674238 4405 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/af244918-878e-430e-a893-aac69d2498ef-plugins-config-volume" pod "af244918-878e-430e-a893-aac69d2498ef" (UID: "af244918-878e-430e-a893-aac69d2498ef"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 03:11:34.674384 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:11:34.674370 4405 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/af244918-878e-430e-a893-aac69d2498ef-plugins-config-volume\") on node \"ip-10-0-134-125.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:11:34.676119 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:11:34.676097 4405 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/af244918-878e-430e-a893-aac69d2498ef-kube-api-access-dcgv6" pod "af244918-878e-430e-a893-aac69d2498ef" (UID: "af244918-878e-430e-a893-aac69d2498ef"). InnerVolumeSpecName "kube-api-access-dcgv6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 03:11:34.768942 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:11:34.768878 4405 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w6-ebb7fd-656b55b679-888kl"] Nov 04 03:11:34.769113 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:11:34.769074 4405 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-ebb7fd-656b55b679-888kl" podUID="52a0927a-e7c2-4c09-aa18-fab7751ca679" containerName="maas-api" containerID="cri-o://e22f477f23d5d43f7cca76976e417db7e18b13645a59ade7ef689a90b05e2782" gracePeriod=30 Nov 04 03:11:34.776090 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:11:34.776052 4405 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dcgv6\" (UniqueName: \"kubernetes.io/projected/af244918-878e-430e-a893-aac69d2498ef-kube-api-access-dcgv6\") on node \"ip-10-0-134-125.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:11:34.917776 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:11:34.917615 4405 scope.go:122] "RemoveContainer" containerID="ac88d64eb459108f07130e51fce895960a73ba7d8556f333dd1abf4501429d23" Nov 04 03:11:34.930486 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:11:34.930180 4405 scope.go:122] "RemoveContainer" containerID="33f579eb8d27949bb4ba12ebe84d9ec19d482af4a3509bf9dffe01fb714f22d3" Nov 04 03:11:35.057854 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:11:35.057829 4405 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-ebb7fd-656b55b679-888kl" Nov 04 03:11:35.060743 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:11:35.060723 4405 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-b-095081-6c6bf65b7b-rdq44" Nov 04 03:11:35.079464 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:11:35.079438 4405 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/52a0927a-e7c2-4c09-aa18-fab7751ca679-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/52a0927a-e7c2-4c09-aa18-fab7751ca679-openshift-service-ca\") pod \"52a0927a-e7c2-4c09-aa18-fab7751ca679\" (UID: \"52a0927a-e7c2-4c09-aa18-fab7751ca679\") " Nov 04 03:11:35.079587 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:11:35.079483 4405 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/52a0927a-e7c2-4c09-aa18-fab7751ca679-kube-api-access-hntrh\" (UniqueName: \"kubernetes.io/projected/52a0927a-e7c2-4c09-aa18-fab7751ca679-kube-api-access-hntrh\") pod \"52a0927a-e7c2-4c09-aa18-fab7751ca679\" (UID: \"52a0927a-e7c2-4c09-aa18-fab7751ca679\") " Nov 04 03:11:35.079635 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:11:35.079591 4405 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/52a0927a-e7c2-4c09-aa18-fab7751ca679-metrics-certs\" (UniqueName: \"kubernetes.io/secret/52a0927a-e7c2-4c09-aa18-fab7751ca679-metrics-certs\") pod \"52a0927a-e7c2-4c09-aa18-fab7751ca679\" (UID: \"52a0927a-e7c2-4c09-aa18-fab7751ca679\") " Nov 04 03:11:35.079635 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:11:35.079625 4405 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/52a0927a-e7c2-4c09-aa18-fab7751ca679-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/52a0927a-e7c2-4c09-aa18-fab7751ca679-maas-api-tls\") pod \"52a0927a-e7c2-4c09-aa18-fab7751ca679\" (UID: \"52a0927a-e7c2-4c09-aa18-fab7751ca679\") " Nov 04 03:11:35.079879 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:11:35.079851 4405 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/52a0927a-e7c2-4c09-aa18-fab7751ca679-openshift-service-ca" pod "52a0927a-e7c2-4c09-aa18-fab7751ca679" (UID: "52a0927a-e7c2-4c09-aa18-fab7751ca679"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 03:11:35.080007 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:11:35.079992 4405 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/52a0927a-e7c2-4c09-aa18-fab7751ca679-openshift-service-ca\") on node \"ip-10-0-134-125.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:11:35.081874 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:11:35.081847 4405 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/52a0927a-e7c2-4c09-aa18-fab7751ca679-metrics-certs" pod "52a0927a-e7c2-4c09-aa18-fab7751ca679" (UID: "52a0927a-e7c2-4c09-aa18-fab7751ca679"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 04 03:11:35.081874 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:11:35.081864 4405 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/52a0927a-e7c2-4c09-aa18-fab7751ca679-maas-api-tls" pod "52a0927a-e7c2-4c09-aa18-fab7751ca679" (UID: "52a0927a-e7c2-4c09-aa18-fab7751ca679"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 04 03:11:35.081984 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:11:35.081933 4405 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/52a0927a-e7c2-4c09-aa18-fab7751ca679-kube-api-access-hntrh" pod "52a0927a-e7c2-4c09-aa18-fab7751ca679" (UID: "52a0927a-e7c2-4c09-aa18-fab7751ca679"). InnerVolumeSpecName "kube-api-access-hntrh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 03:11:35.199324 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:11:35.199215 4405 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/52a0927a-e7c2-4c09-aa18-fab7751ca679-metrics-certs\") on node \"ip-10-0-134-125.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:11:35.199594 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:11:35.199565 4405 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/52a0927a-e7c2-4c09-aa18-fab7751ca679-maas-api-tls\") on node \"ip-10-0-134-125.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:11:35.199772 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:11:35.199748 4405 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hntrh\" (UniqueName: \"kubernetes.io/projected/52a0927a-e7c2-4c09-aa18-fab7751ca679-kube-api-access-hntrh\") on node \"ip-10-0-134-125.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:11:35.227846 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:11:35.227800 4405 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w6-ebb7fd-b8fd547bb-7jkvh" event={"ID":"af244918-878e-430e-a893-aac69d2498ef","Type":"ContainerDied","Data":"ac88d64eb459108f07130e51fce895960a73ba7d8556f333dd1abf4501429d23"} Nov 04 03:11:35.228063 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:11:35.227902 4405 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w6-ebb7fd-b8fd547bb-7jkvh" event={"ID":"af244918-878e-430e-a893-aac69d2498ef","Type":"ContainerDied","Data":"dc26c9356ad761332dcdcfefe4001f0ac58707c719d4b5187b8e00b88dfcc214"} Nov 04 03:11:35.231074 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:11:35.231044 4405 generic.go:356] "Generic (PLEG): container finished" podID="52a0927a-e7c2-4c09-aa18-fab7751ca679" containerID="e22f477f23d5d43f7cca76976e417db7e18b13645a59ade7ef689a90b05e2782" exitCode=0 Nov 04 03:11:35.231204 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:11:35.231107 4405 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-ebb7fd-656b55b679-888kl" Nov 04 03:11:35.231204 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:11:35.231148 4405 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-ebb7fd-656b55b679-888kl" event={"ID":"52a0927a-e7c2-4c09-aa18-fab7751ca679","Type":"ContainerDied","Data":"e22f477f23d5d43f7cca76976e417db7e18b13645a59ade7ef689a90b05e2782"} Nov 04 03:11:35.231204 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:11:35.231194 4405 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-ebb7fd-656b55b679-888kl" event={"ID":"52a0927a-e7c2-4c09-aa18-fab7751ca679","Type":"ContainerDied","Data":"0f4dfa51d507242d38c3871eea04b847edbce30c884afde5483e479e745a689d"} Nov 04 03:11:35.231379 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:11:35.231217 4405 scope.go:122] "RemoveContainer" containerID="e22f477f23d5d43f7cca76976e417db7e18b13645a59ade7ef689a90b05e2782" Nov 04 03:11:35.231379 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:11:35.231318 4405 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w6-ebb7fd-b8fd547bb-7jkvh" Nov 04 03:11:35.243524 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:11:35.243508 4405 scope.go:122] "RemoveContainer" containerID="e22f477f23d5d43f7cca76976e417db7e18b13645a59ade7ef689a90b05e2782" Nov 04 03:11:35.243813 ip-10-0-134-125 kubenswrapper[4405]: E1104 03:11:35.243790 4405 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e22f477f23d5d43f7cca76976e417db7e18b13645a59ade7ef689a90b05e2782\": container with ID starting with e22f477f23d5d43f7cca76976e417db7e18b13645a59ade7ef689a90b05e2782 not found: ID does not exist" containerID="e22f477f23d5d43f7cca76976e417db7e18b13645a59ade7ef689a90b05e2782" Nov 04 03:11:35.243875 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:11:35.243821 4405 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e22f477f23d5d43f7cca76976e417db7e18b13645a59ade7ef689a90b05e2782"} err="failed to get container status \"e22f477f23d5d43f7cca76976e417db7e18b13645a59ade7ef689a90b05e2782\": rpc error: code = NotFound desc = could not find container \"e22f477f23d5d43f7cca76976e417db7e18b13645a59ade7ef689a90b05e2782\": container with ID starting with e22f477f23d5d43f7cca76976e417db7e18b13645a59ade7ef689a90b05e2782 not found: ID does not exist" Nov 04 03:11:35.495368 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:11:35.495282 4405 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w6-ebb7fd-b8fd547bb-7jkvh"] Nov 04 03:11:35.499747 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:11:35.499718 4405 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w6-ebb7fd-b8fd547bb-7jkvh"] Nov 04 03:11:35.607678 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:11:35.607639 4405 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w6-ebb7fd-656b55b679-888kl"] Nov 04 03:11:35.613891 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:11:35.613863 4405 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w6-ebb7fd-656b55b679-888kl"] Nov 04 03:11:35.701645 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:11:35.701612 4405 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="52a0927a-e7c2-4c09-aa18-fab7751ca679" path="/var/lib/kubelet/pods/52a0927a-e7c2-4c09-aa18-fab7751ca679/volumes" Nov 04 03:11:35.702102 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:11:35.702078 4405 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="af244918-878e-430e-a893-aac69d2498ef" path="/var/lib/kubelet/pods/af244918-878e-430e-a893-aac69d2498ef/volumes" Nov 04 03:11:36.888563 ip-10-0-134-125 kubenswrapper[4405]: E1104 03:11:36.888526 4405 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5f2vb" containerName="extractor" Nov 04 03:11:37.050305 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:11:37.050281 4405 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-fe4e0c-openshift-default-c7f4fc84c-6hhrb" Nov 04 03:11:37.125350 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:11:37.125308 4405 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/40a233af-2ad7-4c33-85b2-a2e251d7ea3e-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/40a233af-2ad7-4c33-85b2-a2e251d7ea3e-istio-podinfo\") pod \"40a233af-2ad7-4c33-85b2-a2e251d7ea3e\" (UID: \"40a233af-2ad7-4c33-85b2-a2e251d7ea3e\") " Nov 04 03:11:37.125555 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:11:37.125365 4405 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/40a233af-2ad7-4c33-85b2-a2e251d7ea3e-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/40a233af-2ad7-4c33-85b2-a2e251d7ea3e-istio-envoy\") pod \"40a233af-2ad7-4c33-85b2-a2e251d7ea3e\" (UID: \"40a233af-2ad7-4c33-85b2-a2e251d7ea3e\") " Nov 04 03:11:37.125555 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:11:37.125435 4405 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/40a233af-2ad7-4c33-85b2-a2e251d7ea3e-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/40a233af-2ad7-4c33-85b2-a2e251d7ea3e-istiod-ca-cert\") pod \"40a233af-2ad7-4c33-85b2-a2e251d7ea3e\" (UID: \"40a233af-2ad7-4c33-85b2-a2e251d7ea3e\") " Nov 04 03:11:37.125555 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:11:37.125501 4405 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/40a233af-2ad7-4c33-85b2-a2e251d7ea3e-kube-api-access-rbn5g\" (UniqueName: \"kubernetes.io/projected/40a233af-2ad7-4c33-85b2-a2e251d7ea3e-kube-api-access-rbn5g\") pod \"40a233af-2ad7-4c33-85b2-a2e251d7ea3e\" (UID: \"40a233af-2ad7-4c33-85b2-a2e251d7ea3e\") " Nov 04 03:11:37.125743 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:11:37.125555 4405 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/40a233af-2ad7-4c33-85b2-a2e251d7ea3e-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/40a233af-2ad7-4c33-85b2-a2e251d7ea3e-istio-data\") pod \"40a233af-2ad7-4c33-85b2-a2e251d7ea3e\" (UID: \"40a233af-2ad7-4c33-85b2-a2e251d7ea3e\") " Nov 04 03:11:37.125743 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:11:37.125653 4405 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/40a233af-2ad7-4c33-85b2-a2e251d7ea3e-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/40a233af-2ad7-4c33-85b2-a2e251d7ea3e-workload-socket\") pod \"40a233af-2ad7-4c33-85b2-a2e251d7ea3e\" (UID: \"40a233af-2ad7-4c33-85b2-a2e251d7ea3e\") " Nov 04 03:11:37.125852 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:11:37.125740 4405 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/40a233af-2ad7-4c33-85b2-a2e251d7ea3e-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/40a233af-2ad7-4c33-85b2-a2e251d7ea3e-workload-certs\") pod \"40a233af-2ad7-4c33-85b2-a2e251d7ea3e\" (UID: \"40a233af-2ad7-4c33-85b2-a2e251d7ea3e\") " Nov 04 03:11:37.125908 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:11:37.125869 4405 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/40a233af-2ad7-4c33-85b2-a2e251d7ea3e-istio-token\" (UniqueName: \"kubernetes.io/projected/40a233af-2ad7-4c33-85b2-a2e251d7ea3e-istio-token\") pod \"40a233af-2ad7-4c33-85b2-a2e251d7ea3e\" (UID: \"40a233af-2ad7-4c33-85b2-a2e251d7ea3e\") " Nov 04 03:11:37.125962 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:11:37.125899 4405 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/40a233af-2ad7-4c33-85b2-a2e251d7ea3e-istio-data" pod "40a233af-2ad7-4c33-85b2-a2e251d7ea3e" (UID: "40a233af-2ad7-4c33-85b2-a2e251d7ea3e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 03:11:37.125962 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:11:37.125925 4405 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/40a233af-2ad7-4c33-85b2-a2e251d7ea3e-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/40a233af-2ad7-4c33-85b2-a2e251d7ea3e-credential-socket\") pod \"40a233af-2ad7-4c33-85b2-a2e251d7ea3e\" (UID: \"40a233af-2ad7-4c33-85b2-a2e251d7ea3e\") " Nov 04 03:11:37.126069 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:11:37.125969 4405 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/40a233af-2ad7-4c33-85b2-a2e251d7ea3e-istiod-ca-cert" pod "40a233af-2ad7-4c33-85b2-a2e251d7ea3e" (UID: "40a233af-2ad7-4c33-85b2-a2e251d7ea3e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 03:11:37.126123 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:11:37.126059 4405 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/40a233af-2ad7-4c33-85b2-a2e251d7ea3e-workload-certs" pod "40a233af-2ad7-4c33-85b2-a2e251d7ea3e" (UID: "40a233af-2ad7-4c33-85b2-a2e251d7ea3e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 03:11:37.126181 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:11:37.126115 4405 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/40a233af-2ad7-4c33-85b2-a2e251d7ea3e-workload-socket" pod "40a233af-2ad7-4c33-85b2-a2e251d7ea3e" (UID: "40a233af-2ad7-4c33-85b2-a2e251d7ea3e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 03:11:37.126309 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:11:37.126273 4405 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/40a233af-2ad7-4c33-85b2-a2e251d7ea3e-credential-socket" pod "40a233af-2ad7-4c33-85b2-a2e251d7ea3e" (UID: "40a233af-2ad7-4c33-85b2-a2e251d7ea3e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 03:11:37.126533 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:11:37.126511 4405 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/40a233af-2ad7-4c33-85b2-a2e251d7ea3e-istio-data\") on node \"ip-10-0-134-125.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:11:37.126684 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:11:37.126588 4405 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/40a233af-2ad7-4c33-85b2-a2e251d7ea3e-workload-socket\") on node \"ip-10-0-134-125.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:11:37.126760 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:11:37.126733 4405 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/40a233af-2ad7-4c33-85b2-a2e251d7ea3e-workload-certs\") on node \"ip-10-0-134-125.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:11:37.126760 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:11:37.126748 4405 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/40a233af-2ad7-4c33-85b2-a2e251d7ea3e-credential-socket\") on node \"ip-10-0-134-125.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:11:37.126867 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:11:37.126761 4405 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/40a233af-2ad7-4c33-85b2-a2e251d7ea3e-istiod-ca-cert\") on node \"ip-10-0-134-125.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:11:37.128354 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:11:37.128328 4405 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/40a233af-2ad7-4c33-85b2-a2e251d7ea3e-istio-envoy" pod "40a233af-2ad7-4c33-85b2-a2e251d7ea3e" (UID: "40a233af-2ad7-4c33-85b2-a2e251d7ea3e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 03:11:37.128711 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:11:37.128658 4405 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/40a233af-2ad7-4c33-85b2-a2e251d7ea3e-kube-api-access-rbn5g" pod "40a233af-2ad7-4c33-85b2-a2e251d7ea3e" (UID: "40a233af-2ad7-4c33-85b2-a2e251d7ea3e"). InnerVolumeSpecName "kube-api-access-rbn5g". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 03:11:37.128825 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:11:37.128757 4405 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/40a233af-2ad7-4c33-85b2-a2e251d7ea3e-istio-token" pod "40a233af-2ad7-4c33-85b2-a2e251d7ea3e" (UID: "40a233af-2ad7-4c33-85b2-a2e251d7ea3e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 03:11:37.128896 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:11:37.128864 4405 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/40a233af-2ad7-4c33-85b2-a2e251d7ea3e-istio-podinfo" pod "40a233af-2ad7-4c33-85b2-a2e251d7ea3e" (UID: "40a233af-2ad7-4c33-85b2-a2e251d7ea3e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 04 03:11:37.227502 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:11:37.227460 4405 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/40a233af-2ad7-4c33-85b2-a2e251d7ea3e-istio-podinfo\") on node \"ip-10-0-134-125.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:11:37.227502 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:11:37.227493 4405 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/40a233af-2ad7-4c33-85b2-a2e251d7ea3e-istio-envoy\") on node \"ip-10-0-134-125.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:11:37.227502 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:11:37.227507 4405 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rbn5g\" (UniqueName: \"kubernetes.io/projected/40a233af-2ad7-4c33-85b2-a2e251d7ea3e-kube-api-access-rbn5g\") on node \"ip-10-0-134-125.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:11:37.227761 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:11:37.227522 4405 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/40a233af-2ad7-4c33-85b2-a2e251d7ea3e-istio-token\") on node \"ip-10-0-134-125.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:11:37.248267 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:11:37.248190 4405 generic.go:356] "Generic (PLEG): container finished" podID="40a233af-2ad7-4c33-85b2-a2e251d7ea3e" containerID="31d61c1873e1df2d29db741027dcf73fcb3dd3c14329a064a7f68b47cd9b0557" exitCode=0 Nov 04 03:11:37.248447 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:11:37.248269 4405 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-fe4e0c-openshift-default-c7f4fc84c-6hhrb" event={"ID":"40a233af-2ad7-4c33-85b2-a2e251d7ea3e","Type":"ContainerDied","Data":"31d61c1873e1df2d29db741027dcf73fcb3dd3c14329a064a7f68b47cd9b0557"} Nov 04 03:11:37.248447 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:11:37.248343 4405 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-fe4e0c-openshift-default-c7f4fc84c-6hhrb" event={"ID":"40a233af-2ad7-4c33-85b2-a2e251d7ea3e","Type":"ContainerDied","Data":"c7ee253eb0b150b935c215cd852522e6f9156e8a6c0d0ee547d8640a3f901b86"} Nov 04 03:11:37.248447 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:11:37.248374 4405 scope.go:122] "RemoveContainer" containerID="31d61c1873e1df2d29db741027dcf73fcb3dd3c14329a064a7f68b47cd9b0557" Nov 04 03:11:37.248627 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:11:37.248493 4405 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-fe4e0c-openshift-default-c7f4fc84c-6hhrb" Nov 04 03:11:37.259957 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:11:37.259908 4405 scope.go:122] "RemoveContainer" containerID="31d61c1873e1df2d29db741027dcf73fcb3dd3c14329a064a7f68b47cd9b0557" Nov 04 03:11:37.260319 ip-10-0-134-125 kubenswrapper[4405]: E1104 03:11:37.260286 4405 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"31d61c1873e1df2d29db741027dcf73fcb3dd3c14329a064a7f68b47cd9b0557\": container with ID starting with 31d61c1873e1df2d29db741027dcf73fcb3dd3c14329a064a7f68b47cd9b0557 not found: ID does not exist" containerID="31d61c1873e1df2d29db741027dcf73fcb3dd3c14329a064a7f68b47cd9b0557" Nov 04 03:11:37.260426 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:11:37.260325 4405 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"31d61c1873e1df2d29db741027dcf73fcb3dd3c14329a064a7f68b47cd9b0557"} err="failed to get container status \"31d61c1873e1df2d29db741027dcf73fcb3dd3c14329a064a7f68b47cd9b0557\": rpc error: code = NotFound desc = could not find container \"31d61c1873e1df2d29db741027dcf73fcb3dd3c14329a064a7f68b47cd9b0557\": container with ID starting with 31d61c1873e1df2d29db741027dcf73fcb3dd3c14329a064a7f68b47cd9b0557 not found: ID does not exist" Nov 04 03:11:37.414907 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:11:37.414846 4405 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-fe4e0c-openshift-default-c7f4fc84c-6hhrb"] Nov 04 03:11:37.418927 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:11:37.418895 4405 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-fe4e0c-openshift-default-c7f4fc84c-6hhrb"] Nov 04 03:11:37.703257 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:11:37.703176 4405 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="40a233af-2ad7-4c33-85b2-a2e251d7ea3e" path="/var/lib/kubelet/pods/40a233af-2ad7-4c33-85b2-a2e251d7ea3e/volumes" Nov 04 03:11:38.521414 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:11:38.521351 4405 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-e516ff-openshift-default-6f4b957485-xg2tv" Nov 04 03:11:38.644875 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:11:38.644836 4405 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a4bcafa2-ec97-486c-819f-7be10e3110a9-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a4bcafa2-ec97-486c-819f-7be10e3110a9-istio-data\") pod \"a4bcafa2-ec97-486c-819f-7be10e3110a9\" (UID: \"a4bcafa2-ec97-486c-819f-7be10e3110a9\") " Nov 04 03:11:38.645076 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:11:38.644927 4405 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a4bcafa2-ec97-486c-819f-7be10e3110a9-istio-token\" (UniqueName: \"kubernetes.io/projected/a4bcafa2-ec97-486c-819f-7be10e3110a9-istio-token\") pod \"a4bcafa2-ec97-486c-819f-7be10e3110a9\" (UID: \"a4bcafa2-ec97-486c-819f-7be10e3110a9\") " Nov 04 03:11:38.645076 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:11:38.644969 4405 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a4bcafa2-ec97-486c-819f-7be10e3110a9-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a4bcafa2-ec97-486c-819f-7be10e3110a9-credential-socket\") pod \"a4bcafa2-ec97-486c-819f-7be10e3110a9\" (UID: \"a4bcafa2-ec97-486c-819f-7be10e3110a9\") " Nov 04 03:11:38.645076 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:11:38.645034 4405 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a4bcafa2-ec97-486c-819f-7be10e3110a9-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a4bcafa2-ec97-486c-819f-7be10e3110a9-workload-socket\") pod \"a4bcafa2-ec97-486c-819f-7be10e3110a9\" (UID: \"a4bcafa2-ec97-486c-819f-7be10e3110a9\") " Nov 04 03:11:38.645271 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:11:38.645085 4405 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/a4bcafa2-ec97-486c-819f-7be10e3110a9-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a4bcafa2-ec97-486c-819f-7be10e3110a9-istio-podinfo\") pod \"a4bcafa2-ec97-486c-819f-7be10e3110a9\" (UID: \"a4bcafa2-ec97-486c-819f-7be10e3110a9\") " Nov 04 03:11:38.645271 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:11:38.645159 4405 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a4bcafa2-ec97-486c-819f-7be10e3110a9-kube-api-access-fcp55\" (UniqueName: \"kubernetes.io/projected/a4bcafa2-ec97-486c-819f-7be10e3110a9-kube-api-access-fcp55\") pod \"a4bcafa2-ec97-486c-819f-7be10e3110a9\" (UID: \"a4bcafa2-ec97-486c-819f-7be10e3110a9\") " Nov 04 03:11:38.645271 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:11:38.645161 4405 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a4bcafa2-ec97-486c-819f-7be10e3110a9-istio-data" pod "a4bcafa2-ec97-486c-819f-7be10e3110a9" (UID: "a4bcafa2-ec97-486c-819f-7be10e3110a9"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 03:11:38.645271 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:11:38.645223 4405 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a4bcafa2-ec97-486c-819f-7be10e3110a9-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a4bcafa2-ec97-486c-819f-7be10e3110a9-istiod-ca-cert\") pod \"a4bcafa2-ec97-486c-819f-7be10e3110a9\" (UID: \"a4bcafa2-ec97-486c-819f-7be10e3110a9\") " Nov 04 03:11:38.645271 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:11:38.645255 4405 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a4bcafa2-ec97-486c-819f-7be10e3110a9-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a4bcafa2-ec97-486c-819f-7be10e3110a9-istio-envoy\") pod \"a4bcafa2-ec97-486c-819f-7be10e3110a9\" (UID: \"a4bcafa2-ec97-486c-819f-7be10e3110a9\") " Nov 04 03:11:38.645564 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:11:38.645310 4405 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a4bcafa2-ec97-486c-819f-7be10e3110a9-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a4bcafa2-ec97-486c-819f-7be10e3110a9-workload-certs\") pod \"a4bcafa2-ec97-486c-819f-7be10e3110a9\" (UID: \"a4bcafa2-ec97-486c-819f-7be10e3110a9\") " Nov 04 03:11:38.645564 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:11:38.645370 4405 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a4bcafa2-ec97-486c-819f-7be10e3110a9-credential-socket" pod "a4bcafa2-ec97-486c-819f-7be10e3110a9" (UID: "a4bcafa2-ec97-486c-819f-7be10e3110a9"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 03:11:38.645837 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:11:38.645802 4405 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a4bcafa2-ec97-486c-819f-7be10e3110a9-istio-data\") on node \"ip-10-0-134-125.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:11:38.645837 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:11:38.645830 4405 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a4bcafa2-ec97-486c-819f-7be10e3110a9-credential-socket\") on node \"ip-10-0-134-125.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:11:38.646240 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:11:38.646070 4405 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a4bcafa2-ec97-486c-819f-7be10e3110a9-workload-certs" pod "a4bcafa2-ec97-486c-819f-7be10e3110a9" (UID: "a4bcafa2-ec97-486c-819f-7be10e3110a9"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 03:11:38.646501 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:11:38.646400 4405 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a4bcafa2-ec97-486c-819f-7be10e3110a9-istiod-ca-cert" pod "a4bcafa2-ec97-486c-819f-7be10e3110a9" (UID: "a4bcafa2-ec97-486c-819f-7be10e3110a9"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 03:11:38.647008 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:11:38.646981 4405 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a4bcafa2-ec97-486c-819f-7be10e3110a9-workload-socket" pod "a4bcafa2-ec97-486c-819f-7be10e3110a9" (UID: "a4bcafa2-ec97-486c-819f-7be10e3110a9"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 03:11:38.648517 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:11:38.648481 4405 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a4bcafa2-ec97-486c-819f-7be10e3110a9-istio-token" pod "a4bcafa2-ec97-486c-819f-7be10e3110a9" (UID: "a4bcafa2-ec97-486c-819f-7be10e3110a9"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 03:11:38.648836 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:11:38.648805 4405 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a4bcafa2-ec97-486c-819f-7be10e3110a9-istio-envoy" pod "a4bcafa2-ec97-486c-819f-7be10e3110a9" (UID: "a4bcafa2-ec97-486c-819f-7be10e3110a9"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 03:11:38.649102 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:11:38.649078 4405 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a4bcafa2-ec97-486c-819f-7be10e3110a9-kube-api-access-fcp55" pod "a4bcafa2-ec97-486c-819f-7be10e3110a9" (UID: "a4bcafa2-ec97-486c-819f-7be10e3110a9"). InnerVolumeSpecName "kube-api-access-fcp55". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 03:11:38.649778 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:11:38.649756 4405 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a4bcafa2-ec97-486c-819f-7be10e3110a9-istio-podinfo" pod "a4bcafa2-ec97-486c-819f-7be10e3110a9" (UID: "a4bcafa2-ec97-486c-819f-7be10e3110a9"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 04 03:11:38.746621 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:11:38.746584 4405 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fcp55\" (UniqueName: \"kubernetes.io/projected/a4bcafa2-ec97-486c-819f-7be10e3110a9-kube-api-access-fcp55\") on node \"ip-10-0-134-125.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:11:38.746621 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:11:38.746616 4405 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a4bcafa2-ec97-486c-819f-7be10e3110a9-istiod-ca-cert\") on node \"ip-10-0-134-125.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:11:38.746621 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:11:38.746625 4405 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a4bcafa2-ec97-486c-819f-7be10e3110a9-istio-envoy\") on node \"ip-10-0-134-125.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:11:38.746892 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:11:38.746634 4405 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a4bcafa2-ec97-486c-819f-7be10e3110a9-workload-certs\") on node \"ip-10-0-134-125.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:11:38.746892 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:11:38.746642 4405 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a4bcafa2-ec97-486c-819f-7be10e3110a9-istio-token\") on node \"ip-10-0-134-125.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:11:38.746892 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:11:38.746651 4405 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a4bcafa2-ec97-486c-819f-7be10e3110a9-workload-socket\") on node \"ip-10-0-134-125.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:11:38.746892 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:11:38.746659 4405 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a4bcafa2-ec97-486c-819f-7be10e3110a9-istio-podinfo\") on node \"ip-10-0-134-125.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:11:39.262240 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:11:39.262200 4405 generic.go:356] "Generic (PLEG): container finished" podID="a4bcafa2-ec97-486c-819f-7be10e3110a9" containerID="5ffa0fcee1b9de3f3ef3bb807b85ed515c57d72a9041a34d52c80bf8ce43b404" exitCode=0 Nov 04 03:11:39.262450 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:11:39.262253 4405 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-e516ff-openshift-default-6f4b957485-xg2tv" Nov 04 03:11:39.262450 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:11:39.262265 4405 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-e516ff-openshift-default-6f4b957485-xg2tv" event={"ID":"a4bcafa2-ec97-486c-819f-7be10e3110a9","Type":"ContainerDied","Data":"5ffa0fcee1b9de3f3ef3bb807b85ed515c57d72a9041a34d52c80bf8ce43b404"} Nov 04 03:11:39.262450 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:11:39.262309 4405 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-e516ff-openshift-default-6f4b957485-xg2tv" event={"ID":"a4bcafa2-ec97-486c-819f-7be10e3110a9","Type":"ContainerDied","Data":"7a3008f5730c1ff77eeecdc2011d555af6c751923a88f499e69b2bcfef0b9751"} Nov 04 03:11:39.262450 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:11:39.262330 4405 scope.go:122] "RemoveContainer" containerID="5ffa0fcee1b9de3f3ef3bb807b85ed515c57d72a9041a34d52c80bf8ce43b404" Nov 04 03:11:39.272495 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:11:39.272475 4405 scope.go:122] "RemoveContainer" containerID="5ffa0fcee1b9de3f3ef3bb807b85ed515c57d72a9041a34d52c80bf8ce43b404" Nov 04 03:11:39.272820 ip-10-0-134-125 kubenswrapper[4405]: E1104 03:11:39.272798 4405 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5ffa0fcee1b9de3f3ef3bb807b85ed515c57d72a9041a34d52c80bf8ce43b404\": container with ID starting with 5ffa0fcee1b9de3f3ef3bb807b85ed515c57d72a9041a34d52c80bf8ce43b404 not found: ID does not exist" containerID="5ffa0fcee1b9de3f3ef3bb807b85ed515c57d72a9041a34d52c80bf8ce43b404" Nov 04 03:11:39.272880 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:11:39.272829 4405 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5ffa0fcee1b9de3f3ef3bb807b85ed515c57d72a9041a34d52c80bf8ce43b404"} err="failed to get container status \"5ffa0fcee1b9de3f3ef3bb807b85ed515c57d72a9041a34d52c80bf8ce43b404\": rpc error: code = NotFound desc = could not find container \"5ffa0fcee1b9de3f3ef3bb807b85ed515c57d72a9041a34d52c80bf8ce43b404\": container with ID starting with 5ffa0fcee1b9de3f3ef3bb807b85ed515c57d72a9041a34d52c80bf8ce43b404 not found: ID does not exist" Nov 04 03:11:39.315055 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:11:39.315019 4405 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w3-cdcf67-76579b7df4-vqbkj" podUID="d280cbd2-035b-422b-85c2-79a39d4a06cc" containerName="payload-processing" containerID="cri-o://ba2c291bdac8dd46b4dbde415acaa0def7a9b987b27f52dcaff5dca346b06fc8" gracePeriod=25 Nov 04 03:11:39.429085 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:11:39.429024 4405 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-e516ff-openshift-default-6f4b957485-xg2tv"] Nov 04 03:11:39.477682 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:11:39.477627 4405 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-e516ff-openshift-default-6f4b957485-xg2tv"] Nov 04 03:11:39.704013 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:11:39.703974 4405 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a4bcafa2-ec97-486c-819f-7be10e3110a9" path="/var/lib/kubelet/pods/a4bcafa2-ec97-486c-819f-7be10e3110a9/volumes" Nov 04 03:11:45.636612 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:11:45.636563 4405 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w3-cdcf67-76579b7df4-vqbkj" podUID="d280cbd2-035b-422b-85c2-79a39d4a06cc" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 04 03:11:52.866961 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:11:52.866923 4405 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-ebb7fd-openshift-default-544f5768f-w6tt7"] Nov 04 03:11:52.867487 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:11:52.867170 4405 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w6-ebb7fd-openshift-default-544f5768f-w6tt7" podUID="2522c184-4d83-40b4-9a42-cbc55b7a5a37" containerName="istio-proxy" containerID="cri-o://83db20eac508cc7d5a8e7a6d62fdbd1c4ef770a08b559688768c7610becdf04f" gracePeriod=30 Nov 04 03:11:55.635956 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:11:55.635911 4405 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w3-cdcf67-76579b7df4-vqbkj" podUID="d280cbd2-035b-422b-85c2-79a39d4a06cc" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 04 03:11:56.435644 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:11:56.435603 4405 patch_prober.go:29] interesting pod/e2e-worker-w6-ebb7fd-openshift-default-544f5768f-w6tt7 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 04 03:11:56.435880 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:11:56.435698 4405 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w6-ebb7fd-openshift-default-544f5768f-w6tt7" podUID="2522c184-4d83-40b4-9a42-cbc55b7a5a37" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 04 03:11:57.438792 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:11:57.438752 4405 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-095081-85f7c6c868-qh2c7"] Nov 04 03:11:57.439519 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:11:57.439498 4405 cpu_manager.go:414] "containerMap: removing container" podUID="40a233af-2ad7-4c33-85b2-a2e251d7ea3e" containerName="istio-proxy" Nov 04 03:11:57.439569 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:11:57.439523 4405 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="40a233af-2ad7-4c33-85b2-a2e251d7ea3e" containerName="istio-proxy" Nov 04 03:11:57.439569 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:11:57.439545 4405 cpu_manager.go:414] "containerMap: removing container" podUID="a4bcafa2-ec97-486c-819f-7be10e3110a9" containerName="istio-proxy" Nov 04 03:11:57.439569 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:11:57.439554 4405 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a4bcafa2-ec97-486c-819f-7be10e3110a9" containerName="istio-proxy" Nov 04 03:11:57.439681 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:11:57.439568 4405 cpu_manager.go:414] "containerMap: removing container" podUID="af244918-878e-430e-a893-aac69d2498ef" containerName="payload-processing" Nov 04 03:11:57.439681 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:11:57.439577 4405 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="af244918-878e-430e-a893-aac69d2498ef" containerName="payload-processing" Nov 04 03:11:57.439681 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:11:57.439597 4405 cpu_manager.go:414] "containerMap: removing container" podUID="52ed22cc-96e4-4ef0-b34d-ed58c89a9183" containerName="main" Nov 04 03:11:57.439681 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:11:57.439605 4405 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="52ed22cc-96e4-4ef0-b34d-ed58c89a9183" containerName="main" Nov 04 03:11:57.439681 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:11:57.439625 4405 cpu_manager.go:414] "containerMap: removing container" podUID="52a0927a-e7c2-4c09-aa18-fab7751ca679" containerName="maas-api" Nov 04 03:11:57.439681 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:11:57.439632 4405 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="52a0927a-e7c2-4c09-aa18-fab7751ca679" containerName="maas-api" Nov 04 03:11:57.439681 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:11:57.439646 4405 cpu_manager.go:414] "containerMap: removing container" podUID="ccb8e181-5610-46da-bb8b-e0b07fa02ff1" containerName="main" Nov 04 03:11:57.439681 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:11:57.439652 4405 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ccb8e181-5610-46da-bb8b-e0b07fa02ff1" containerName="main" Nov 04 03:11:57.439920 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:11:57.439685 4405 cpu_manager.go:414] "containerMap: removing container" podUID="b7b786e0-9c94-4126-9949-cd354ebbfb2a" containerName="main" Nov 04 03:11:57.439920 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:11:57.439693 4405 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b7b786e0-9c94-4126-9949-cd354ebbfb2a" containerName="main" Nov 04 03:11:57.439920 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:11:57.439828 4405 memory_manager.go:365] "RemoveStaleState removing state" podUID="40a233af-2ad7-4c33-85b2-a2e251d7ea3e" containerName="istio-proxy" Nov 04 03:11:57.439920 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:11:57.439841 4405 memory_manager.go:365] "RemoveStaleState removing state" podUID="a4bcafa2-ec97-486c-819f-7be10e3110a9" containerName="istio-proxy" Nov 04 03:11:57.439920 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:11:57.439852 4405 memory_manager.go:365] "RemoveStaleState removing state" podUID="af244918-878e-430e-a893-aac69d2498ef" containerName="payload-processing" Nov 04 03:11:57.439920 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:11:57.439862 4405 memory_manager.go:365] "RemoveStaleState removing state" podUID="52ed22cc-96e4-4ef0-b34d-ed58c89a9183" containerName="main" Nov 04 03:11:57.439920 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:11:57.439871 4405 memory_manager.go:365] "RemoveStaleState removing state" podUID="52a0927a-e7c2-4c09-aa18-fab7751ca679" containerName="maas-api" Nov 04 03:11:57.439920 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:11:57.439880 4405 memory_manager.go:365] "RemoveStaleState removing state" podUID="ccb8e181-5610-46da-bb8b-e0b07fa02ff1" containerName="main" Nov 04 03:11:57.439920 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:11:57.439889 4405 memory_manager.go:365] "RemoveStaleState removing state" podUID="b7b786e0-9c94-4126-9949-cd354ebbfb2a" containerName="main" Nov 04 03:11:57.445123 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:11:57.445103 4405 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-095081-85f7c6c868-qh2c7" Nov 04 03:11:57.468967 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:11:57.468933 4405 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/239f941f-cf45-4550-90cd-e2be9f961e07-openshift-service-ca\") pod \"maas-api-e2e-inf-b-095081-85f7c6c868-qh2c7\" (UID: \"239f941f-cf45-4550-90cd-e2be9f961e07\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-095081-85f7c6c868-qh2c7" Nov 04 03:11:57.469145 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:11:57.468977 4405 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/239f941f-cf45-4550-90cd-e2be9f961e07-metrics-certs\") pod \"maas-api-e2e-inf-b-095081-85f7c6c868-qh2c7\" (UID: \"239f941f-cf45-4550-90cd-e2be9f961e07\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-095081-85f7c6c868-qh2c7" Nov 04 03:11:57.469221 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:11:57.469173 4405 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/239f941f-cf45-4550-90cd-e2be9f961e07-maas-api-tls\") pod \"maas-api-e2e-inf-b-095081-85f7c6c868-qh2c7\" (UID: \"239f941f-cf45-4550-90cd-e2be9f961e07\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-095081-85f7c6c868-qh2c7" Nov 04 03:11:57.469304 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:11:57.469284 4405 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jd9j2\" (UniqueName: \"kubernetes.io/projected/239f941f-cf45-4550-90cd-e2be9f961e07-kube-api-access-jd9j2\") pod \"maas-api-e2e-inf-b-095081-85f7c6c868-qh2c7\" (UID: \"239f941f-cf45-4550-90cd-e2be9f961e07\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-095081-85f7c6c868-qh2c7" Nov 04 03:11:57.546400 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:11:57.546369 4405 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-b-095081-serving-cert\"" Nov 04 03:11:57.549255 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:11:57.549228 4405 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-095081-85f7c6c868-qh2c7"] Nov 04 03:11:57.571082 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:11:57.571036 4405 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/239f941f-cf45-4550-90cd-e2be9f961e07-maas-api-tls\") pod \"maas-api-e2e-inf-b-095081-85f7c6c868-qh2c7\" (UID: \"239f941f-cf45-4550-90cd-e2be9f961e07\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-095081-85f7c6c868-qh2c7" Nov 04 03:11:57.571261 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:11:57.571134 4405 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jd9j2\" (UniqueName: \"kubernetes.io/projected/239f941f-cf45-4550-90cd-e2be9f961e07-kube-api-access-jd9j2\") pod \"maas-api-e2e-inf-b-095081-85f7c6c868-qh2c7\" (UID: \"239f941f-cf45-4550-90cd-e2be9f961e07\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-095081-85f7c6c868-qh2c7" Nov 04 03:11:57.571261 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:11:57.571205 4405 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/239f941f-cf45-4550-90cd-e2be9f961e07-openshift-service-ca\") pod \"maas-api-e2e-inf-b-095081-85f7c6c868-qh2c7\" (UID: \"239f941f-cf45-4550-90cd-e2be9f961e07\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-095081-85f7c6c868-qh2c7" Nov 04 03:11:57.571261 ip-10-0-134-125 kubenswrapper[4405]: E1104 03:11:57.571214 4405 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-inf-b-095081-serving-cert: secret "maas-api-e2e-inf-b-095081-serving-cert" not found Nov 04 03:11:57.571261 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:11:57.571235 4405 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/239f941f-cf45-4550-90cd-e2be9f961e07-metrics-certs\") pod \"maas-api-e2e-inf-b-095081-85f7c6c868-qh2c7\" (UID: \"239f941f-cf45-4550-90cd-e2be9f961e07\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-095081-85f7c6c868-qh2c7" Nov 04 03:11:57.571436 ip-10-0-134-125 kubenswrapper[4405]: E1104 03:11:57.571300 4405 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/239f941f-cf45-4550-90cd-e2be9f961e07-maas-api-tls podName:239f941f-cf45-4550-90cd-e2be9f961e07 nodeName:}" failed. No retries permitted until 2026-11-04 03:11:58.07127947 +0000 UTC m=+2364.501979588 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/239f941f-cf45-4550-90cd-e2be9f961e07-maas-api-tls") pod "maas-api-e2e-inf-b-095081-85f7c6c868-qh2c7" (UID: "239f941f-cf45-4550-90cd-e2be9f961e07") : secret "maas-api-e2e-inf-b-095081-serving-cert" not found Nov 04 03:11:57.572411 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:11:57.572383 4405 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/239f941f-cf45-4550-90cd-e2be9f961e07-openshift-service-ca\") pod \"maas-api-e2e-inf-b-095081-85f7c6c868-qh2c7\" (UID: \"239f941f-cf45-4550-90cd-e2be9f961e07\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-095081-85f7c6c868-qh2c7" Nov 04 03:11:57.575231 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:11:57.575204 4405 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/239f941f-cf45-4550-90cd-e2be9f961e07-metrics-certs\") pod \"maas-api-e2e-inf-b-095081-85f7c6c868-qh2c7\" (UID: \"239f941f-cf45-4550-90cd-e2be9f961e07\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-095081-85f7c6c868-qh2c7" Nov 04 03:11:57.632544 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:11:57.632515 4405 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jd9j2\" (UniqueName: \"kubernetes.io/projected/239f941f-cf45-4550-90cd-e2be9f961e07-kube-api-access-jd9j2\") pod \"maas-api-e2e-inf-b-095081-85f7c6c868-qh2c7\" (UID: \"239f941f-cf45-4550-90cd-e2be9f961e07\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-095081-85f7c6c868-qh2c7" Nov 04 03:11:58.079109 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:11:58.079065 4405 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/239f941f-cf45-4550-90cd-e2be9f961e07-maas-api-tls\") pod \"maas-api-e2e-inf-b-095081-85f7c6c868-qh2c7\" (UID: \"239f941f-cf45-4550-90cd-e2be9f961e07\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-095081-85f7c6c868-qh2c7" Nov 04 03:11:58.084558 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:11:58.084515 4405 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/239f941f-cf45-4550-90cd-e2be9f961e07-maas-api-tls\") pod \"maas-api-e2e-inf-b-095081-85f7c6c868-qh2c7\" (UID: \"239f941f-cf45-4550-90cd-e2be9f961e07\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-095081-85f7c6c868-qh2c7" Nov 04 03:11:58.122049 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:11:58.122021 4405 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-ebb7fd-openshift-default-544f5768f-w6tt7" Nov 04 03:11:58.180285 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:11:58.180185 4405 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2522c184-4d83-40b4-9a42-cbc55b7a5a37-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2522c184-4d83-40b4-9a42-cbc55b7a5a37-credential-socket\") pod \"2522c184-4d83-40b4-9a42-cbc55b7a5a37\" (UID: \"2522c184-4d83-40b4-9a42-cbc55b7a5a37\") " Nov 04 03:11:58.180285 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:11:58.180260 4405 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2522c184-4d83-40b4-9a42-cbc55b7a5a37-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2522c184-4d83-40b4-9a42-cbc55b7a5a37-istio-envoy\") pod \"2522c184-4d83-40b4-9a42-cbc55b7a5a37\" (UID: \"2522c184-4d83-40b4-9a42-cbc55b7a5a37\") " Nov 04 03:11:58.180285 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:11:58.180284 4405 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2522c184-4d83-40b4-9a42-cbc55b7a5a37-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2522c184-4d83-40b4-9a42-cbc55b7a5a37-workload-socket\") pod \"2522c184-4d83-40b4-9a42-cbc55b7a5a37\" (UID: \"2522c184-4d83-40b4-9a42-cbc55b7a5a37\") " Nov 04 03:11:58.180592 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:11:58.180316 4405 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2522c184-4d83-40b4-9a42-cbc55b7a5a37-kube-api-access-2tvxc\" (UniqueName: \"kubernetes.io/projected/2522c184-4d83-40b4-9a42-cbc55b7a5a37-kube-api-access-2tvxc\") pod \"2522c184-4d83-40b4-9a42-cbc55b7a5a37\" (UID: \"2522c184-4d83-40b4-9a42-cbc55b7a5a37\") " Nov 04 03:11:58.180592 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:11:58.180334 4405 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2522c184-4d83-40b4-9a42-cbc55b7a5a37-istio-token\" (UniqueName: \"kubernetes.io/projected/2522c184-4d83-40b4-9a42-cbc55b7a5a37-istio-token\") pod \"2522c184-4d83-40b4-9a42-cbc55b7a5a37\" (UID: \"2522c184-4d83-40b4-9a42-cbc55b7a5a37\") " Nov 04 03:11:58.180592 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:11:58.180365 4405 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/2522c184-4d83-40b4-9a42-cbc55b7a5a37-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2522c184-4d83-40b4-9a42-cbc55b7a5a37-istio-podinfo\") pod \"2522c184-4d83-40b4-9a42-cbc55b7a5a37\" (UID: \"2522c184-4d83-40b4-9a42-cbc55b7a5a37\") " Nov 04 03:11:58.180592 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:11:58.180428 4405 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2522c184-4d83-40b4-9a42-cbc55b7a5a37-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2522c184-4d83-40b4-9a42-cbc55b7a5a37-istiod-ca-cert\") pod \"2522c184-4d83-40b4-9a42-cbc55b7a5a37\" (UID: \"2522c184-4d83-40b4-9a42-cbc55b7a5a37\") " Nov 04 03:11:58.180592 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:11:58.180528 4405 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2522c184-4d83-40b4-9a42-cbc55b7a5a37-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2522c184-4d83-40b4-9a42-cbc55b7a5a37-istio-data\") pod \"2522c184-4d83-40b4-9a42-cbc55b7a5a37\" (UID: \"2522c184-4d83-40b4-9a42-cbc55b7a5a37\") " Nov 04 03:11:58.180592 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:11:58.180554 4405 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2522c184-4d83-40b4-9a42-cbc55b7a5a37-credential-socket" pod "2522c184-4d83-40b4-9a42-cbc55b7a5a37" (UID: "2522c184-4d83-40b4-9a42-cbc55b7a5a37"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 03:11:58.180592 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:11:58.180573 4405 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2522c184-4d83-40b4-9a42-cbc55b7a5a37-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2522c184-4d83-40b4-9a42-cbc55b7a5a37-workload-certs\") pod \"2522c184-4d83-40b4-9a42-cbc55b7a5a37\" (UID: \"2522c184-4d83-40b4-9a42-cbc55b7a5a37\") " Nov 04 03:11:58.181036 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:11:58.180789 4405 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2522c184-4d83-40b4-9a42-cbc55b7a5a37-workload-socket" pod "2522c184-4d83-40b4-9a42-cbc55b7a5a37" (UID: "2522c184-4d83-40b4-9a42-cbc55b7a5a37"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 03:11:58.181036 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:11:58.180925 4405 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2522c184-4d83-40b4-9a42-cbc55b7a5a37-istiod-ca-cert" pod "2522c184-4d83-40b4-9a42-cbc55b7a5a37" (UID: "2522c184-4d83-40b4-9a42-cbc55b7a5a37"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 03:11:58.181150 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:11:58.181059 4405 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2522c184-4d83-40b4-9a42-cbc55b7a5a37-workload-certs" pod "2522c184-4d83-40b4-9a42-cbc55b7a5a37" (UID: "2522c184-4d83-40b4-9a42-cbc55b7a5a37"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 03:11:58.181150 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:11:58.181141 4405 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2522c184-4d83-40b4-9a42-cbc55b7a5a37-credential-socket\") on node \"ip-10-0-134-125.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:11:58.181317 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:11:58.181160 4405 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2522c184-4d83-40b4-9a42-cbc55b7a5a37-istio-data" pod "2522c184-4d83-40b4-9a42-cbc55b7a5a37" (UID: "2522c184-4d83-40b4-9a42-cbc55b7a5a37"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 03:11:58.181317 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:11:58.181163 4405 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2522c184-4d83-40b4-9a42-cbc55b7a5a37-workload-socket\") on node \"ip-10-0-134-125.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:11:58.181398 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:11:58.181294 4405 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2522c184-4d83-40b4-9a42-cbc55b7a5a37-istiod-ca-cert\") on node \"ip-10-0-134-125.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:11:58.182814 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:11:58.182793 4405 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2522c184-4d83-40b4-9a42-cbc55b7a5a37-istio-token" pod "2522c184-4d83-40b4-9a42-cbc55b7a5a37" (UID: "2522c184-4d83-40b4-9a42-cbc55b7a5a37"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 03:11:58.182908 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:11:58.182892 4405 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2522c184-4d83-40b4-9a42-cbc55b7a5a37-kube-api-access-2tvxc" pod "2522c184-4d83-40b4-9a42-cbc55b7a5a37" (UID: "2522c184-4d83-40b4-9a42-cbc55b7a5a37"). InnerVolumeSpecName "kube-api-access-2tvxc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 03:11:58.183286 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:11:58.183268 4405 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2522c184-4d83-40b4-9a42-cbc55b7a5a37-istio-envoy" pod "2522c184-4d83-40b4-9a42-cbc55b7a5a37" (UID: "2522c184-4d83-40b4-9a42-cbc55b7a5a37"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 03:11:58.183353 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:11:58.183286 4405 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/2522c184-4d83-40b4-9a42-cbc55b7a5a37-istio-podinfo" pod "2522c184-4d83-40b4-9a42-cbc55b7a5a37" (UID: "2522c184-4d83-40b4-9a42-cbc55b7a5a37"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 04 03:11:58.282307 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:11:58.282270 4405 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2522c184-4d83-40b4-9a42-cbc55b7a5a37-istio-data\") on node \"ip-10-0-134-125.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:11:58.282307 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:11:58.282299 4405 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2522c184-4d83-40b4-9a42-cbc55b7a5a37-workload-certs\") on node \"ip-10-0-134-125.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:11:58.282307 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:11:58.282308 4405 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2522c184-4d83-40b4-9a42-cbc55b7a5a37-istio-envoy\") on node \"ip-10-0-134-125.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:11:58.282307 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:11:58.282317 4405 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2tvxc\" (UniqueName: \"kubernetes.io/projected/2522c184-4d83-40b4-9a42-cbc55b7a5a37-kube-api-access-2tvxc\") on node \"ip-10-0-134-125.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:11:58.282740 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:11:58.282326 4405 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2522c184-4d83-40b4-9a42-cbc55b7a5a37-istio-token\") on node \"ip-10-0-134-125.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:11:58.282740 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:11:58.282335 4405 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2522c184-4d83-40b4-9a42-cbc55b7a5a37-istio-podinfo\") on node \"ip-10-0-134-125.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:11:58.358929 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:11:58.358886 4405 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-095081-85f7c6c868-qh2c7" Nov 04 03:11:58.368703 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:11:58.368644 4405 generic.go:356] "Generic (PLEG): container finished" podID="2522c184-4d83-40b4-9a42-cbc55b7a5a37" containerID="83db20eac508cc7d5a8e7a6d62fdbd1c4ef770a08b559688768c7610becdf04f" exitCode=0 Nov 04 03:11:58.368838 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:11:58.368718 4405 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-ebb7fd-openshift-default-544f5768f-w6tt7" event={"ID":"2522c184-4d83-40b4-9a42-cbc55b7a5a37","Type":"ContainerDied","Data":"83db20eac508cc7d5a8e7a6d62fdbd1c4ef770a08b559688768c7610becdf04f"} Nov 04 03:11:58.368838 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:11:58.368740 4405 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-ebb7fd-openshift-default-544f5768f-w6tt7" Nov 04 03:11:58.368838 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:11:58.368776 4405 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-ebb7fd-openshift-default-544f5768f-w6tt7" event={"ID":"2522c184-4d83-40b4-9a42-cbc55b7a5a37","Type":"ContainerDied","Data":"a4c0ac9aed61eb6e9c510ed2ebbd24fa93f2119d68340e041e91047a1ab68cf4"} Nov 04 03:11:58.368838 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:11:58.368801 4405 scope.go:122] "RemoveContainer" containerID="83db20eac508cc7d5a8e7a6d62fdbd1c4ef770a08b559688768c7610becdf04f" Nov 04 03:11:58.380178 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:11:58.380153 4405 scope.go:122] "RemoveContainer" containerID="83db20eac508cc7d5a8e7a6d62fdbd1c4ef770a08b559688768c7610becdf04f" Nov 04 03:11:58.380546 ip-10-0-134-125 kubenswrapper[4405]: E1104 03:11:58.380519 4405 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"83db20eac508cc7d5a8e7a6d62fdbd1c4ef770a08b559688768c7610becdf04f\": container with ID starting with 83db20eac508cc7d5a8e7a6d62fdbd1c4ef770a08b559688768c7610becdf04f not found: ID does not exist" containerID="83db20eac508cc7d5a8e7a6d62fdbd1c4ef770a08b559688768c7610becdf04f" Nov 04 03:11:58.380634 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:11:58.380558 4405 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"83db20eac508cc7d5a8e7a6d62fdbd1c4ef770a08b559688768c7610becdf04f"} err="failed to get container status \"83db20eac508cc7d5a8e7a6d62fdbd1c4ef770a08b559688768c7610becdf04f\": rpc error: code = NotFound desc = could not find container \"83db20eac508cc7d5a8e7a6d62fdbd1c4ef770a08b559688768c7610becdf04f\": container with ID starting with 83db20eac508cc7d5a8e7a6d62fdbd1c4ef770a08b559688768c7610becdf04f not found: ID does not exist" Nov 04 03:11:58.539154 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:11:58.539062 4405 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-ebb7fd-openshift-default-544f5768f-w6tt7"] Nov 04 03:11:58.544793 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:11:58.544768 4405 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w6-ebb7fd-openshift-default-544f5768f-w6tt7"] Nov 04 03:11:58.738614 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:11:58.738582 4405 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-095081-85f7c6c868-qh2c7"] Nov 04 03:11:58.738794 ip-10-0-134-125 kubenswrapper[4405]: W1104 03:11:58.738767 4405 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod239f941f_cf45_4550_90cd_e2be9f961e07.slice/crio-93c80311be729d7708ea18f3a8b3a87e46cbc881265eae5eed9c5f08cfc10b2f WatchSource:0}: Error finding container 93c80311be729d7708ea18f3a8b3a87e46cbc881265eae5eed9c5f08cfc10b2f: Status 404 returned error can't find the container with id 93c80311be729d7708ea18f3a8b3a87e46cbc881265eae5eed9c5f08cfc10b2f Nov 04 03:11:59.376254 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:11:59.376219 4405 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-095081-85f7c6c868-qh2c7" event={"ID":"239f941f-cf45-4550-90cd-e2be9f961e07","Type":"ContainerStarted","Data":"93c80311be729d7708ea18f3a8b3a87e46cbc881265eae5eed9c5f08cfc10b2f"} Nov 04 03:11:59.588657 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:11:59.588625 4405 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w3-cdcf67-76579b7df4-vqbkj" Nov 04 03:11:59.697898 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:11:59.697871 4405 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d280cbd2-035b-422b-85c2-79a39d4a06cc-kube-api-access-7lk5m\" (UniqueName: \"kubernetes.io/projected/d280cbd2-035b-422b-85c2-79a39d4a06cc-kube-api-access-7lk5m\") pod \"d280cbd2-035b-422b-85c2-79a39d4a06cc\" (UID: \"d280cbd2-035b-422b-85c2-79a39d4a06cc\") " Nov 04 03:11:59.698157 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:11:59.698135 4405 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d280cbd2-035b-422b-85c2-79a39d4a06cc-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d280cbd2-035b-422b-85c2-79a39d4a06cc-plugins-config-volume\") pod \"d280cbd2-035b-422b-85c2-79a39d4a06cc\" (UID: \"d280cbd2-035b-422b-85c2-79a39d4a06cc\") " Nov 04 03:11:59.698598 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:11:59.698570 4405 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d280cbd2-035b-422b-85c2-79a39d4a06cc-plugins-config-volume" pod "d280cbd2-035b-422b-85c2-79a39d4a06cc" (UID: "d280cbd2-035b-422b-85c2-79a39d4a06cc"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 03:11:59.700802 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:11:59.700778 4405 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d280cbd2-035b-422b-85c2-79a39d4a06cc-kube-api-access-7lk5m" pod "d280cbd2-035b-422b-85c2-79a39d4a06cc" (UID: "d280cbd2-035b-422b-85c2-79a39d4a06cc"). InnerVolumeSpecName "kube-api-access-7lk5m". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 03:11:59.703714 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:11:59.703686 4405 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2522c184-4d83-40b4-9a42-cbc55b7a5a37" path="/var/lib/kubelet/pods/2522c184-4d83-40b4-9a42-cbc55b7a5a37/volumes" Nov 04 03:11:59.799546 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:11:59.799498 4405 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d280cbd2-035b-422b-85c2-79a39d4a06cc-plugins-config-volume\") on node \"ip-10-0-134-125.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:11:59.799546 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:11:59.799545 4405 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7lk5m\" (UniqueName: \"kubernetes.io/projected/d280cbd2-035b-422b-85c2-79a39d4a06cc-kube-api-access-7lk5m\") on node \"ip-10-0-134-125.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:12:00.385806 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:12:00.385758 4405 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-095081-85f7c6c868-qh2c7" event={"ID":"239f941f-cf45-4550-90cd-e2be9f961e07","Type":"ContainerStarted","Data":"6158d11aea76349ce7eeae299fa573bd28f033422a27089a724230063d24d76f"} Nov 04 03:12:00.386029 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:12:00.385901 4405 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-095081-85f7c6c868-qh2c7" Nov 04 03:12:00.387616 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:12:00.387550 4405 generic.go:356] "Generic (PLEG): container finished" podID="d280cbd2-035b-422b-85c2-79a39d4a06cc" containerID="ba2c291bdac8dd46b4dbde415acaa0def7a9b987b27f52dcaff5dca346b06fc8" exitCode=0 Nov 04 03:12:00.387772 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:12:00.387646 4405 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w3-cdcf67-76579b7df4-vqbkj" event={"ID":"d280cbd2-035b-422b-85c2-79a39d4a06cc","Type":"ContainerDied","Data":"ba2c291bdac8dd46b4dbde415acaa0def7a9b987b27f52dcaff5dca346b06fc8"} Nov 04 03:12:00.387772 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:12:00.387729 4405 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w3-cdcf67-76579b7df4-vqbkj" event={"ID":"d280cbd2-035b-422b-85c2-79a39d4a06cc","Type":"ContainerDied","Data":"552946d45d4c42ea9b8f51f4e9cfc6b2167d24be77b19e89ccab86330e0b2b16"} Nov 04 03:12:00.387772 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:12:00.387760 4405 scope.go:122] "RemoveContainer" containerID="ba2c291bdac8dd46b4dbde415acaa0def7a9b987b27f52dcaff5dca346b06fc8" Nov 04 03:12:00.387927 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:12:00.387914 4405 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w3-cdcf67-76579b7df4-vqbkj" Nov 04 03:12:00.403558 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:12:00.403370 4405 scope.go:122] "RemoveContainer" containerID="ba2c291bdac8dd46b4dbde415acaa0def7a9b987b27f52dcaff5dca346b06fc8" Nov 04 03:12:00.404111 ip-10-0-134-125 kubenswrapper[4405]: E1104 03:12:00.404082 4405 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ba2c291bdac8dd46b4dbde415acaa0def7a9b987b27f52dcaff5dca346b06fc8\": container with ID starting with ba2c291bdac8dd46b4dbde415acaa0def7a9b987b27f52dcaff5dca346b06fc8 not found: ID does not exist" containerID="ba2c291bdac8dd46b4dbde415acaa0def7a9b987b27f52dcaff5dca346b06fc8" Nov 04 03:12:00.404220 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:12:00.404125 4405 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ba2c291bdac8dd46b4dbde415acaa0def7a9b987b27f52dcaff5dca346b06fc8"} err="failed to get container status \"ba2c291bdac8dd46b4dbde415acaa0def7a9b987b27f52dcaff5dca346b06fc8\": rpc error: code = NotFound desc = could not find container \"ba2c291bdac8dd46b4dbde415acaa0def7a9b987b27f52dcaff5dca346b06fc8\": container with ID starting with ba2c291bdac8dd46b4dbde415acaa0def7a9b987b27f52dcaff5dca346b06fc8 not found: ID does not exist" Nov 04 03:12:00.511576 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:12:00.511493 4405 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-095081-85f7c6c868-qh2c7" podStartSLOduration=2.638960752 podStartE2EDuration="3.511473901s" totalImagesPullingTime="872.513149ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 03:11:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 03:12:00.510941972 +0000 UTC m=+2366.941642091" watchObservedRunningTime="2026-11-04 03:12:00.511473901 +0000 UTC m=+2366.942174021" Nov 04 03:12:00.626594 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:12:00.626548 4405 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w3-cdcf67-76579b7df4-vqbkj"] Nov 04 03:12:00.642252 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:12:00.642153 4405 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w3-cdcf67-76579b7df4-vqbkj"] Nov 04 03:12:01.703179 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:12:01.703142 4405 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d280cbd2-035b-422b-85c2-79a39d4a06cc" path="/var/lib/kubelet/pods/d280cbd2-035b-422b-85c2-79a39d4a06cc/volumes" Nov 04 03:12:01.982202 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:12:01.982121 4405 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w3-cdcf67-556f77498f-fltxv"] Nov 04 03:12:01.982407 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:12:01.982381 4405 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-cdcf67-556f77498f-fltxv" podUID="b1d26a63-824f-40dd-9379-603c6f9d77e1" containerName="maas-api" containerID="cri-o://2b5873d4155af77cf1362163aebf56cbcd8fc195fbeeb459466d0d40e482ac11" gracePeriod=30 Nov 04 03:12:02.218926 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:12:02.218880 4405 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-bl5mg"] Nov 04 03:12:02.257958 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:12:02.257934 4405 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-cdcf67-556f77498f-fltxv" Nov 04 03:12:02.351714 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:12:02.351650 4405 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b1d26a63-824f-40dd-9379-603c6f9d77e1-metrics-certs\" (UniqueName: \"kubernetes.io/secret/b1d26a63-824f-40dd-9379-603c6f9d77e1-metrics-certs\") pod \"b1d26a63-824f-40dd-9379-603c6f9d77e1\" (UID: \"b1d26a63-824f-40dd-9379-603c6f9d77e1\") " Nov 04 03:12:02.351971 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:12:02.351758 4405 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b1d26a63-824f-40dd-9379-603c6f9d77e1-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b1d26a63-824f-40dd-9379-603c6f9d77e1-maas-api-tls\") pod \"b1d26a63-824f-40dd-9379-603c6f9d77e1\" (UID: \"b1d26a63-824f-40dd-9379-603c6f9d77e1\") " Nov 04 03:12:02.351971 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:12:02.351874 4405 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b1d26a63-824f-40dd-9379-603c6f9d77e1-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b1d26a63-824f-40dd-9379-603c6f9d77e1-openshift-service-ca\") pod \"b1d26a63-824f-40dd-9379-603c6f9d77e1\" (UID: \"b1d26a63-824f-40dd-9379-603c6f9d77e1\") " Nov 04 03:12:02.351971 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:12:02.351963 4405 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b1d26a63-824f-40dd-9379-603c6f9d77e1-kube-api-access-r5l28\" (UniqueName: \"kubernetes.io/projected/b1d26a63-824f-40dd-9379-603c6f9d77e1-kube-api-access-r5l28\") pod \"b1d26a63-824f-40dd-9379-603c6f9d77e1\" (UID: \"b1d26a63-824f-40dd-9379-603c6f9d77e1\") " Nov 04 03:12:02.352296 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:12:02.352259 4405 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b1d26a63-824f-40dd-9379-603c6f9d77e1-openshift-service-ca" pod "b1d26a63-824f-40dd-9379-603c6f9d77e1" (UID: "b1d26a63-824f-40dd-9379-603c6f9d77e1"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 03:12:02.352653 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:12:02.352626 4405 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b1d26a63-824f-40dd-9379-603c6f9d77e1-openshift-service-ca\") on node \"ip-10-0-134-125.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:12:02.354202 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:12:02.354174 4405 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b1d26a63-824f-40dd-9379-603c6f9d77e1-maas-api-tls" pod "b1d26a63-824f-40dd-9379-603c6f9d77e1" (UID: "b1d26a63-824f-40dd-9379-603c6f9d77e1"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 04 03:12:02.354285 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:12:02.354209 4405 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b1d26a63-824f-40dd-9379-603c6f9d77e1-metrics-certs" pod "b1d26a63-824f-40dd-9379-603c6f9d77e1" (UID: "b1d26a63-824f-40dd-9379-603c6f9d77e1"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 04 03:12:02.354444 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:12:02.354426 4405 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b1d26a63-824f-40dd-9379-603c6f9d77e1-kube-api-access-r5l28" pod "b1d26a63-824f-40dd-9379-603c6f9d77e1" (UID: "b1d26a63-824f-40dd-9379-603c6f9d77e1"). InnerVolumeSpecName "kube-api-access-r5l28". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 03:12:02.407268 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:12:02.407227 4405 generic.go:356] "Generic (PLEG): container finished" podID="b1d26a63-824f-40dd-9379-603c6f9d77e1" containerID="2b5873d4155af77cf1362163aebf56cbcd8fc195fbeeb459466d0d40e482ac11" exitCode=0 Nov 04 03:12:02.407452 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:12:02.407300 4405 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-cdcf67-556f77498f-fltxv" event={"ID":"b1d26a63-824f-40dd-9379-603c6f9d77e1","Type":"ContainerDied","Data":"2b5873d4155af77cf1362163aebf56cbcd8fc195fbeeb459466d0d40e482ac11"} Nov 04 03:12:02.407452 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:12:02.407352 4405 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-cdcf67-556f77498f-fltxv" event={"ID":"b1d26a63-824f-40dd-9379-603c6f9d77e1","Type":"ContainerDied","Data":"d4096c1bde5c345cc89fe0a0d446fe60297b01e095afcd15bdd2f311a78cec7e"} Nov 04 03:12:02.407452 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:12:02.407386 4405 scope.go:122] "RemoveContainer" containerID="2b5873d4155af77cf1362163aebf56cbcd8fc195fbeeb459466d0d40e482ac11" Nov 04 03:12:02.407577 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:12:02.407512 4405 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-cdcf67-556f77498f-fltxv" Nov 04 03:12:02.422980 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:12:02.422942 4405 scope.go:122] "RemoveContainer" containerID="2b5873d4155af77cf1362163aebf56cbcd8fc195fbeeb459466d0d40e482ac11" Nov 04 03:12:02.423379 ip-10-0-134-125 kubenswrapper[4405]: E1104 03:12:02.423345 4405 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2b5873d4155af77cf1362163aebf56cbcd8fc195fbeeb459466d0d40e482ac11\": container with ID starting with 2b5873d4155af77cf1362163aebf56cbcd8fc195fbeeb459466d0d40e482ac11 not found: ID does not exist" containerID="2b5873d4155af77cf1362163aebf56cbcd8fc195fbeeb459466d0d40e482ac11" Nov 04 03:12:02.423471 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:12:02.423393 4405 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2b5873d4155af77cf1362163aebf56cbcd8fc195fbeeb459466d0d40e482ac11"} err="failed to get container status \"2b5873d4155af77cf1362163aebf56cbcd8fc195fbeeb459466d0d40e482ac11\": rpc error: code = NotFound desc = could not find container \"2b5873d4155af77cf1362163aebf56cbcd8fc195fbeeb459466d0d40e482ac11\": container with ID starting with 2b5873d4155af77cf1362163aebf56cbcd8fc195fbeeb459466d0d40e482ac11 not found: ID does not exist" Nov 04 03:12:02.453891 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:12:02.453863 4405 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b1d26a63-824f-40dd-9379-603c6f9d77e1-metrics-certs\") on node \"ip-10-0-134-125.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:12:02.453891 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:12:02.453889 4405 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b1d26a63-824f-40dd-9379-603c6f9d77e1-maas-api-tls\") on node \"ip-10-0-134-125.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:12:02.454101 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:12:02.453903 4405 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r5l28\" (UniqueName: \"kubernetes.io/projected/b1d26a63-824f-40dd-9379-603c6f9d77e1-kube-api-access-r5l28\") on node \"ip-10-0-134-125.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:12:02.575068 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:12:02.574984 4405 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w3-cdcf67-556f77498f-fltxv"] Nov 04 03:12:02.579283 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:12:02.579254 4405 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w3-cdcf67-556f77498f-fltxv"] Nov 04 03:12:03.725305 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:12:03.725267 4405 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b1d26a63-824f-40dd-9379-603c6f9d77e1" path="/var/lib/kubelet/pods/b1d26a63-824f-40dd-9379-603c6f9d77e1/volumes" Nov 04 03:12:06.418338 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:12:06.418307 4405 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-095081-85f7c6c868-qh2c7" Nov 04 03:12:06.889270 ip-10-0-134-125 kubenswrapper[4405]: E1104 03:12:06.889236 4405 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5f2vb" containerName="extractor" Nov 04 03:12:13.488915 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:12:13.488845 4405 generic.go:356] "Generic (PLEG): container finished" podID="a77157ba-0aed-46cf-90de-ffc15074a50f" containerID="81aa272b24db3e2aac8307e85ec1c538754b7333cddcdf89cdfbbdaaf0382d0e" exitCode=0 Nov 04 03:12:13.489547 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:12:13.489107 4405 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-4cbcf738-openshift-default-68fb497c9c-2ndqr" event={"ID":"a77157ba-0aed-46cf-90de-ffc15074a50f","Type":"ContainerDied","Data":"81aa272b24db3e2aac8307e85ec1c538754b7333cddcdf89cdfbbdaaf0382d0e"} Nov 04 03:12:13.489733 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:12:13.489709 4405 scope.go:122] "RemoveContainer" containerID="81aa272b24db3e2aac8307e85ec1c538754b7333cddcdf89cdfbbdaaf0382d0e" Nov 04 03:12:13.491841 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:12:13.491810 4405 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 04 03:12:13.491933 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:12:13.491882 4405 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 04 03:12:13.492050 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:12:13.492037 4405 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 04 03:12:14.520614 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:12:14.520560 4405 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-4cbcf738-openshift-default-68fb497c9c-2ndqr" event={"ID":"a77157ba-0aed-46cf-90de-ffc15074a50f","Type":"ContainerStarted","Data":"380cfa2a332239970422898ec218bee16d39f4fc2106c20638fa7e0982c35c0e"} Nov 04 03:12:15.163404 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:12:15.163309 4405 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-4cbcf738-openshift-default-68fb497c9c-2ndqr" Nov 04 03:12:15.163620 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:12:15.163596 4405 patch_prober.go:29] interesting pod/e2e-mt-4cbcf738-openshift-default-68fb497c9c-2ndqr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.127:15021/healthz/ready\": dial tcp 10.134.0.127:15021: connect: connection refused" start-of-body= Nov 04 03:12:15.163711 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:12:15.163658 4405 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-4cbcf738-openshift-default-68fb497c9c-2ndqr" podUID="a77157ba-0aed-46cf-90de-ffc15074a50f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.127:15021/healthz/ready\": dial tcp 10.134.0.127:15021: connect: connection refused" Nov 04 03:12:15.524368 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:12:15.524274 4405 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-cdcf67-openshift-default-6ff8f5cbc7-rwgvq"] Nov 04 03:12:15.524857 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:12:15.524539 4405 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-cdcf67-openshift-default-6ff8f5cbc7-rwgvq" podUID="dad304e5-de0a-44b2-a137-8139a2aa2f52" containerName="istio-proxy" containerID="cri-o://6a196d872afe8c00e1ad270f8064013e8a2638c5c7797c9e5c2a69524f602fff" gracePeriod=30 Nov 04 03:12:16.163728 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:12:16.163660 4405 patch_prober.go:29] interesting pod/e2e-mt-4cbcf738-openshift-default-68fb497c9c-2ndqr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.127:15021/healthz/ready\": dial tcp 10.134.0.127:15021: connect: connection refused" start-of-body= Nov 04 03:12:16.163960 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:12:16.163752 4405 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-4cbcf738-openshift-default-68fb497c9c-2ndqr" podUID="a77157ba-0aed-46cf-90de-ffc15074a50f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.127:15021/healthz/ready\": dial tcp 10.134.0.127:15021: connect: connection refused" Nov 04 03:12:17.163870 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:12:17.163828 4405 patch_prober.go:29] interesting pod/e2e-mt-4cbcf738-openshift-default-68fb497c9c-2ndqr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.127:15021/healthz/ready\": dial tcp 10.134.0.127:15021: connect: connection refused" start-of-body= Nov 04 03:12:17.164296 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:12:17.163894 4405 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-4cbcf738-openshift-default-68fb497c9c-2ndqr" podUID="a77157ba-0aed-46cf-90de-ffc15074a50f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.127:15021/healthz/ready\": dial tcp 10.134.0.127:15021: connect: connection refused" Nov 04 03:12:18.163475 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:12:18.163438 4405 patch_prober.go:29] interesting pod/e2e-mt-4cbcf738-openshift-default-68fb497c9c-2ndqr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.127:15021/healthz/ready\": dial tcp 10.134.0.127:15021: connect: connection refused" start-of-body= Nov 04 03:12:18.163662 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:12:18.163491 4405 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-4cbcf738-openshift-default-68fb497c9c-2ndqr" podUID="a77157ba-0aed-46cf-90de-ffc15074a50f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.127:15021/healthz/ready\": dial tcp 10.134.0.127:15021: connect: connection refused" Nov 04 03:12:19.163219 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:12:19.163175 4405 patch_prober.go:29] interesting pod/e2e-mt-4cbcf738-openshift-default-68fb497c9c-2ndqr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.127:15021/healthz/ready\": dial tcp 10.134.0.127:15021: connect: connection refused" start-of-body= Nov 04 03:12:19.163632 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:12:19.163236 4405 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-4cbcf738-openshift-default-68fb497c9c-2ndqr" podUID="a77157ba-0aed-46cf-90de-ffc15074a50f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.127:15021/healthz/ready\": dial tcp 10.134.0.127:15021: connect: connection refused" Nov 04 03:12:20.168604 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:12:20.168565 4405 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-4cbcf738-openshift-default-68fb497c9c-2ndqr" Nov 04 03:12:20.169065 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:12:20.168931 4405 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-4cbcf738-openshift-default-68fb497c9c-2ndqr" Nov 04 03:12:20.169785 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:12:20.169761 4405 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-4cbcf738-openshift-default-68fb497c9c-2ndqr" Nov 04 03:12:20.602417 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:12:20.602385 4405 patch_prober.go:29] interesting pod/e2e-worker-w3-cdcf67-openshift-default-6ff8f5cbc7-rwgvq container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.34:15021/healthz/ready\": dial tcp 10.134.0.34:15021: connect: connection refused" start-of-body= Nov 04 03:12:20.602588 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:12:20.602445 4405 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-cdcf67-openshift-default-6ff8f5cbc7-rwgvq" podUID="dad304e5-de0a-44b2-a137-8139a2aa2f52" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.34:15021/healthz/ready\": dial tcp 10.134.0.34:15021: connect: connection refused" Nov 04 03:12:20.796466 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:12:20.796441 4405 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-cdcf67-openshift-default-6ff8f5cbc7-rwgvq" Nov 04 03:12:20.888081 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:12:20.888049 4405 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dad304e5-de0a-44b2-a137-8139a2aa2f52-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dad304e5-de0a-44b2-a137-8139a2aa2f52-istio-data\") pod \"dad304e5-de0a-44b2-a137-8139a2aa2f52\" (UID: \"dad304e5-de0a-44b2-a137-8139a2aa2f52\") " Nov 04 03:12:20.888295 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:12:20.888089 4405 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dad304e5-de0a-44b2-a137-8139a2aa2f52-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dad304e5-de0a-44b2-a137-8139a2aa2f52-istiod-ca-cert\") pod \"dad304e5-de0a-44b2-a137-8139a2aa2f52\" (UID: \"dad304e5-de0a-44b2-a137-8139a2aa2f52\") " Nov 04 03:12:20.888295 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:12:20.888121 4405 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dad304e5-de0a-44b2-a137-8139a2aa2f52-kube-api-access-htpww\" (UniqueName: \"kubernetes.io/projected/dad304e5-de0a-44b2-a137-8139a2aa2f52-kube-api-access-htpww\") pod \"dad304e5-de0a-44b2-a137-8139a2aa2f52\" (UID: \"dad304e5-de0a-44b2-a137-8139a2aa2f52\") " Nov 04 03:12:20.888295 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:12:20.888152 4405 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dad304e5-de0a-44b2-a137-8139a2aa2f52-istio-token\" (UniqueName: \"kubernetes.io/projected/dad304e5-de0a-44b2-a137-8139a2aa2f52-istio-token\") pod \"dad304e5-de0a-44b2-a137-8139a2aa2f52\" (UID: \"dad304e5-de0a-44b2-a137-8139a2aa2f52\") " Nov 04 03:12:20.888295 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:12:20.888190 4405 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dad304e5-de0a-44b2-a137-8139a2aa2f52-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dad304e5-de0a-44b2-a137-8139a2aa2f52-credential-socket\") pod \"dad304e5-de0a-44b2-a137-8139a2aa2f52\" (UID: \"dad304e5-de0a-44b2-a137-8139a2aa2f52\") " Nov 04 03:12:20.888295 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:12:20.888242 4405 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dad304e5-de0a-44b2-a137-8139a2aa2f52-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dad304e5-de0a-44b2-a137-8139a2aa2f52-workload-certs\") pod \"dad304e5-de0a-44b2-a137-8139a2aa2f52\" (UID: \"dad304e5-de0a-44b2-a137-8139a2aa2f52\") " Nov 04 03:12:20.888595 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:12:20.888304 4405 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dad304e5-de0a-44b2-a137-8139a2aa2f52-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dad304e5-de0a-44b2-a137-8139a2aa2f52-istio-envoy\") pod \"dad304e5-de0a-44b2-a137-8139a2aa2f52\" (UID: \"dad304e5-de0a-44b2-a137-8139a2aa2f52\") " Nov 04 03:12:20.888595 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:12:20.888347 4405 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/dad304e5-de0a-44b2-a137-8139a2aa2f52-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dad304e5-de0a-44b2-a137-8139a2aa2f52-istio-podinfo\") pod \"dad304e5-de0a-44b2-a137-8139a2aa2f52\" (UID: \"dad304e5-de0a-44b2-a137-8139a2aa2f52\") " Nov 04 03:12:20.888595 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:12:20.888372 4405 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dad304e5-de0a-44b2-a137-8139a2aa2f52-istio-data" pod "dad304e5-de0a-44b2-a137-8139a2aa2f52" (UID: "dad304e5-de0a-44b2-a137-8139a2aa2f52"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 03:12:20.888595 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:12:20.888418 4405 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dad304e5-de0a-44b2-a137-8139a2aa2f52-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dad304e5-de0a-44b2-a137-8139a2aa2f52-workload-socket\") pod \"dad304e5-de0a-44b2-a137-8139a2aa2f52\" (UID: \"dad304e5-de0a-44b2-a137-8139a2aa2f52\") " Nov 04 03:12:20.888595 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:12:20.888557 4405 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dad304e5-de0a-44b2-a137-8139a2aa2f52-istiod-ca-cert" pod "dad304e5-de0a-44b2-a137-8139a2aa2f52" (UID: "dad304e5-de0a-44b2-a137-8139a2aa2f52"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 03:12:20.888882 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:12:20.888700 4405 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dad304e5-de0a-44b2-a137-8139a2aa2f52-credential-socket" pod "dad304e5-de0a-44b2-a137-8139a2aa2f52" (UID: "dad304e5-de0a-44b2-a137-8139a2aa2f52"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 03:12:20.888932 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:12:20.888882 4405 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dad304e5-de0a-44b2-a137-8139a2aa2f52-workload-socket" pod "dad304e5-de0a-44b2-a137-8139a2aa2f52" (UID: "dad304e5-de0a-44b2-a137-8139a2aa2f52"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 03:12:20.888932 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:12:20.888901 4405 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dad304e5-de0a-44b2-a137-8139a2aa2f52-workload-certs" pod "dad304e5-de0a-44b2-a137-8139a2aa2f52" (UID: "dad304e5-de0a-44b2-a137-8139a2aa2f52"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 03:12:20.889092 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:12:20.889061 4405 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dad304e5-de0a-44b2-a137-8139a2aa2f52-istiod-ca-cert\") on node \"ip-10-0-134-125.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:12:20.889149 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:12:20.889091 4405 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dad304e5-de0a-44b2-a137-8139a2aa2f52-credential-socket\") on node \"ip-10-0-134-125.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:12:20.889149 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:12:20.889107 4405 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dad304e5-de0a-44b2-a137-8139a2aa2f52-workload-certs\") on node \"ip-10-0-134-125.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:12:20.889149 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:12:20.889120 4405 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dad304e5-de0a-44b2-a137-8139a2aa2f52-workload-socket\") on node \"ip-10-0-134-125.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:12:20.889149 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:12:20.889132 4405 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dad304e5-de0a-44b2-a137-8139a2aa2f52-istio-data\") on node \"ip-10-0-134-125.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:12:20.890709 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:12:20.890680 4405 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dad304e5-de0a-44b2-a137-8139a2aa2f52-istio-token" pod "dad304e5-de0a-44b2-a137-8139a2aa2f52" (UID: "dad304e5-de0a-44b2-a137-8139a2aa2f52"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 03:12:20.890871 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:12:20.890849 4405 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dad304e5-de0a-44b2-a137-8139a2aa2f52-istio-envoy" pod "dad304e5-de0a-44b2-a137-8139a2aa2f52" (UID: "dad304e5-de0a-44b2-a137-8139a2aa2f52"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 03:12:20.891215 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:12:20.891190 4405 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/dad304e5-de0a-44b2-a137-8139a2aa2f52-istio-podinfo" pod "dad304e5-de0a-44b2-a137-8139a2aa2f52" (UID: "dad304e5-de0a-44b2-a137-8139a2aa2f52"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 04 03:12:20.891317 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:12:20.891227 4405 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dad304e5-de0a-44b2-a137-8139a2aa2f52-kube-api-access-htpww" pod "dad304e5-de0a-44b2-a137-8139a2aa2f52" (UID: "dad304e5-de0a-44b2-a137-8139a2aa2f52"). InnerVolumeSpecName "kube-api-access-htpww". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 03:12:20.990852 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:12:20.990807 4405 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dad304e5-de0a-44b2-a137-8139a2aa2f52-istio-envoy\") on node \"ip-10-0-134-125.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:12:20.990852 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:12:20.990844 4405 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dad304e5-de0a-44b2-a137-8139a2aa2f52-istio-podinfo\") on node \"ip-10-0-134-125.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:12:20.991118 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:12:20.990861 4405 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-htpww\" (UniqueName: \"kubernetes.io/projected/dad304e5-de0a-44b2-a137-8139a2aa2f52-kube-api-access-htpww\") on node \"ip-10-0-134-125.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:12:20.991118 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:12:20.990877 4405 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dad304e5-de0a-44b2-a137-8139a2aa2f52-istio-token\") on node \"ip-10-0-134-125.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:12:21.568019 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:12:21.567985 4405 generic.go:356] "Generic (PLEG): container finished" podID="dad304e5-de0a-44b2-a137-8139a2aa2f52" containerID="6a196d872afe8c00e1ad270f8064013e8a2638c5c7797c9e5c2a69524f602fff" exitCode=0 Nov 04 03:12:21.568473 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:12:21.568039 4405 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-cdcf67-openshift-default-6ff8f5cbc7-rwgvq" Nov 04 03:12:21.568473 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:12:21.568071 4405 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-cdcf67-openshift-default-6ff8f5cbc7-rwgvq" event={"ID":"dad304e5-de0a-44b2-a137-8139a2aa2f52","Type":"ContainerDied","Data":"6a196d872afe8c00e1ad270f8064013e8a2638c5c7797c9e5c2a69524f602fff"} Nov 04 03:12:21.568473 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:12:21.568127 4405 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-cdcf67-openshift-default-6ff8f5cbc7-rwgvq" event={"ID":"dad304e5-de0a-44b2-a137-8139a2aa2f52","Type":"ContainerDied","Data":"8c7750471f5b1c23493d386a91a2700251a5b0c68dc090adb4b7c64b96cd81d5"} Nov 04 03:12:21.568473 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:12:21.568153 4405 scope.go:122] "RemoveContainer" containerID="6a196d872afe8c00e1ad270f8064013e8a2638c5c7797c9e5c2a69524f602fff" Nov 04 03:12:21.580405 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:12:21.580386 4405 scope.go:122] "RemoveContainer" containerID="443defcf2b22d24c59168e7d581684bb84006228c2e217384d9131e127596c20" Nov 04 03:12:21.591382 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:12:21.591362 4405 scope.go:122] "RemoveContainer" containerID="6a196d872afe8c00e1ad270f8064013e8a2638c5c7797c9e5c2a69524f602fff" Nov 04 03:12:21.591711 ip-10-0-134-125 kubenswrapper[4405]: E1104 03:12:21.591662 4405 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6a196d872afe8c00e1ad270f8064013e8a2638c5c7797c9e5c2a69524f602fff\": container with ID starting with 6a196d872afe8c00e1ad270f8064013e8a2638c5c7797c9e5c2a69524f602fff not found: ID does not exist" containerID="6a196d872afe8c00e1ad270f8064013e8a2638c5c7797c9e5c2a69524f602fff" Nov 04 03:12:21.591809 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:12:21.591715 4405 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6a196d872afe8c00e1ad270f8064013e8a2638c5c7797c9e5c2a69524f602fff"} err="failed to get container status \"6a196d872afe8c00e1ad270f8064013e8a2638c5c7797c9e5c2a69524f602fff\": rpc error: code = NotFound desc = could not find container \"6a196d872afe8c00e1ad270f8064013e8a2638c5c7797c9e5c2a69524f602fff\": container with ID starting with 6a196d872afe8c00e1ad270f8064013e8a2638c5c7797c9e5c2a69524f602fff not found: ID does not exist" Nov 04 03:12:21.591809 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:12:21.591732 4405 scope.go:122] "RemoveContainer" containerID="443defcf2b22d24c59168e7d581684bb84006228c2e217384d9131e127596c20" Nov 04 03:12:21.591968 ip-10-0-134-125 kubenswrapper[4405]: E1104 03:12:21.591952 4405 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"443defcf2b22d24c59168e7d581684bb84006228c2e217384d9131e127596c20\": container with ID starting with 443defcf2b22d24c59168e7d581684bb84006228c2e217384d9131e127596c20 not found: ID does not exist" containerID="443defcf2b22d24c59168e7d581684bb84006228c2e217384d9131e127596c20" Nov 04 03:12:21.592011 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:12:21.591973 4405 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"443defcf2b22d24c59168e7d581684bb84006228c2e217384d9131e127596c20"} err="failed to get container status \"443defcf2b22d24c59168e7d581684bb84006228c2e217384d9131e127596c20\": rpc error: code = NotFound desc = could not find container \"443defcf2b22d24c59168e7d581684bb84006228c2e217384d9131e127596c20\": container with ID starting with 443defcf2b22d24c59168e7d581684bb84006228c2e217384d9131e127596c20 not found: ID does not exist" Nov 04 03:12:21.769076 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:12:21.769032 4405 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-cdcf67-openshift-default-6ff8f5cbc7-rwgvq"] Nov 04 03:12:21.818801 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:12:21.818728 4405 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-cdcf67-openshift-default-6ff8f5cbc7-rwgvq"] Nov 04 03:12:23.701431 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:12:23.701396 4405 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dad304e5-de0a-44b2-a137-8139a2aa2f52" path="/var/lib/kubelet/pods/dad304e5-de0a-44b2-a137-8139a2aa2f52/volumes" Nov 04 03:12:34.012045 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:12:34.012010 4405 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-vwcx8_ffbb2d83-5a93-4a5b-bce3-e40afb8f58ea/ovn-acl-logging/0.log" Nov 04 03:12:34.015147 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:12:34.015120 4405 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-vwcx8_ffbb2d83-5a93-4a5b-bce3-e40afb8f58ea/ovn-acl-logging/0.log" Nov 04 03:12:36.885086 ip-10-0-134-125 kubenswrapper[4405]: E1104 03:12:36.885044 4405 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5f2vb" containerName="extractor" Nov 04 03:12:50.211629 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:12:50.211589 4405 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-bl5mg"] Nov 04 03:12:53.984227 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:12:53.984184 4405 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-a-db5ef5/test-model-db5ef5-kserve-6d4b5bbd85-5jhrf"] Nov 04 03:12:53.984880 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:12:53.984864 4405 cpu_manager.go:414] "containerMap: removing container" podUID="d280cbd2-035b-422b-85c2-79a39d4a06cc" containerName="payload-processing" Nov 04 03:12:53.984934 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:12:53.984883 4405 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d280cbd2-035b-422b-85c2-79a39d4a06cc" containerName="payload-processing" Nov 04 03:12:53.984934 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:12:53.984901 4405 cpu_manager.go:414] "containerMap: removing container" podUID="dad304e5-de0a-44b2-a137-8139a2aa2f52" containerName="istio-proxy" Nov 04 03:12:53.984934 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:12:53.984908 4405 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dad304e5-de0a-44b2-a137-8139a2aa2f52" containerName="istio-proxy" Nov 04 03:12:53.984934 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:12:53.984932 4405 cpu_manager.go:414] "containerMap: removing container" podUID="b1d26a63-824f-40dd-9379-603c6f9d77e1" containerName="maas-api" Nov 04 03:12:53.985064 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:12:53.984940 4405 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b1d26a63-824f-40dd-9379-603c6f9d77e1" containerName="maas-api" Nov 04 03:12:53.985064 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:12:53.984960 4405 cpu_manager.go:414] "containerMap: removing container" podUID="2522c184-4d83-40b4-9a42-cbc55b7a5a37" containerName="istio-proxy" Nov 04 03:12:53.985064 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:12:53.984968 4405 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2522c184-4d83-40b4-9a42-cbc55b7a5a37" containerName="istio-proxy" Nov 04 03:12:53.985158 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:12:53.985108 4405 memory_manager.go:365] "RemoveStaleState removing state" podUID="dad304e5-de0a-44b2-a137-8139a2aa2f52" containerName="istio-proxy" Nov 04 03:12:53.985158 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:12:53.985121 4405 memory_manager.go:365] "RemoveStaleState removing state" podUID="d280cbd2-035b-422b-85c2-79a39d4a06cc" containerName="payload-processing" Nov 04 03:12:53.985158 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:12:53.985135 4405 memory_manager.go:365] "RemoveStaleState removing state" podUID="b1d26a63-824f-40dd-9379-603c6f9d77e1" containerName="maas-api" Nov 04 03:12:53.985158 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:12:53.985147 4405 memory_manager.go:365] "RemoveStaleState removing state" podUID="2522c184-4d83-40b4-9a42-cbc55b7a5a37" containerName="istio-proxy" Nov 04 03:12:53.990416 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:12:53.990397 4405 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-db5ef5/test-model-db5ef5-kserve-6d4b5bbd85-5jhrf" Nov 04 03:12:54.066554 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:12:54.066514 4405 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c1ed6380-62fc-4239-81a7-b1368b9a529b-tls-certs\") pod \"test-model-db5ef5-kserve-6d4b5bbd85-5jhrf\" (UID: \"c1ed6380-62fc-4239-81a7-b1368b9a529b\") " pod="ai-tenant-e2e-inf-a-db5ef5/test-model-db5ef5-kserve-6d4b5bbd85-5jhrf" Nov 04 03:12:54.066771 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:12:54.066638 4405 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c1ed6380-62fc-4239-81a7-b1368b9a529b-model-cache\") pod \"test-model-db5ef5-kserve-6d4b5bbd85-5jhrf\" (UID: \"c1ed6380-62fc-4239-81a7-b1368b9a529b\") " pod="ai-tenant-e2e-inf-a-db5ef5/test-model-db5ef5-kserve-6d4b5bbd85-5jhrf" Nov 04 03:12:54.066846 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:12:54.066821 4405 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7525p\" (UniqueName: \"kubernetes.io/projected/c1ed6380-62fc-4239-81a7-b1368b9a529b-kube-api-access-7525p\") pod \"test-model-db5ef5-kserve-6d4b5bbd85-5jhrf\" (UID: \"c1ed6380-62fc-4239-81a7-b1368b9a529b\") " pod="ai-tenant-e2e-inf-a-db5ef5/test-model-db5ef5-kserve-6d4b5bbd85-5jhrf" Nov 04 03:12:54.066894 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:12:54.066874 4405 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c1ed6380-62fc-4239-81a7-b1368b9a529b-dshm\") pod \"test-model-db5ef5-kserve-6d4b5bbd85-5jhrf\" (UID: \"c1ed6380-62fc-4239-81a7-b1368b9a529b\") " pod="ai-tenant-e2e-inf-a-db5ef5/test-model-db5ef5-kserve-6d4b5bbd85-5jhrf" Nov 04 03:12:54.066942 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:12:54.066898 4405 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c1ed6380-62fc-4239-81a7-b1368b9a529b-home\") pod \"test-model-db5ef5-kserve-6d4b5bbd85-5jhrf\" (UID: \"c1ed6380-62fc-4239-81a7-b1368b9a529b\") " pod="ai-tenant-e2e-inf-a-db5ef5/test-model-db5ef5-kserve-6d4b5bbd85-5jhrf" Nov 04 03:12:54.067036 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:12:54.067015 4405 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c1ed6380-62fc-4239-81a7-b1368b9a529b-tmp-dir\") pod \"test-model-db5ef5-kserve-6d4b5bbd85-5jhrf\" (UID: \"c1ed6380-62fc-4239-81a7-b1368b9a529b\") " pod="ai-tenant-e2e-inf-a-db5ef5/test-model-db5ef5-kserve-6d4b5bbd85-5jhrf" Nov 04 03:12:54.092529 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:12:54.092502 4405 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-db5ef5\"/\"default-dockercfg-mxx2h\"" Nov 04 03:12:54.092736 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:12:54.092558 4405 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-db5ef5\"/\"test-model-db5ef5-kserve-self-signed-certs\"" Nov 04 03:12:54.092736 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:12:54.092572 4405 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-db5ef5\"/\"kube-root-ca.crt\"" Nov 04 03:12:54.093131 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:12:54.093117 4405 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-db5ef5\"/\"openshift-service-ca.crt\"" Nov 04 03:12:54.095323 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:12:54.095287 4405 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-db5ef5/test-model-db5ef5-kserve-6d4b5bbd85-5jhrf"] Nov 04 03:12:54.168157 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:12:54.168114 4405 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7525p\" (UniqueName: \"kubernetes.io/projected/c1ed6380-62fc-4239-81a7-b1368b9a529b-kube-api-access-7525p\") pod \"test-model-db5ef5-kserve-6d4b5bbd85-5jhrf\" (UID: \"c1ed6380-62fc-4239-81a7-b1368b9a529b\") " pod="ai-tenant-e2e-inf-a-db5ef5/test-model-db5ef5-kserve-6d4b5bbd85-5jhrf" Nov 04 03:12:54.168376 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:12:54.168170 4405 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c1ed6380-62fc-4239-81a7-b1368b9a529b-dshm\") pod \"test-model-db5ef5-kserve-6d4b5bbd85-5jhrf\" (UID: \"c1ed6380-62fc-4239-81a7-b1368b9a529b\") " pod="ai-tenant-e2e-inf-a-db5ef5/test-model-db5ef5-kserve-6d4b5bbd85-5jhrf" Nov 04 03:12:54.168376 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:12:54.168191 4405 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c1ed6380-62fc-4239-81a7-b1368b9a529b-home\") pod \"test-model-db5ef5-kserve-6d4b5bbd85-5jhrf\" (UID: \"c1ed6380-62fc-4239-81a7-b1368b9a529b\") " pod="ai-tenant-e2e-inf-a-db5ef5/test-model-db5ef5-kserve-6d4b5bbd85-5jhrf" Nov 04 03:12:54.168376 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:12:54.168210 4405 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c1ed6380-62fc-4239-81a7-b1368b9a529b-tmp-dir\") pod \"test-model-db5ef5-kserve-6d4b5bbd85-5jhrf\" (UID: \"c1ed6380-62fc-4239-81a7-b1368b9a529b\") " pod="ai-tenant-e2e-inf-a-db5ef5/test-model-db5ef5-kserve-6d4b5bbd85-5jhrf" Nov 04 03:12:54.168376 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:12:54.168253 4405 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c1ed6380-62fc-4239-81a7-b1368b9a529b-tls-certs\") pod \"test-model-db5ef5-kserve-6d4b5bbd85-5jhrf\" (UID: \"c1ed6380-62fc-4239-81a7-b1368b9a529b\") " pod="ai-tenant-e2e-inf-a-db5ef5/test-model-db5ef5-kserve-6d4b5bbd85-5jhrf" Nov 04 03:12:54.168376 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:12:54.168297 4405 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c1ed6380-62fc-4239-81a7-b1368b9a529b-model-cache\") pod \"test-model-db5ef5-kserve-6d4b5bbd85-5jhrf\" (UID: \"c1ed6380-62fc-4239-81a7-b1368b9a529b\") " pod="ai-tenant-e2e-inf-a-db5ef5/test-model-db5ef5-kserve-6d4b5bbd85-5jhrf" Nov 04 03:12:54.168738 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:12:54.168716 4405 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c1ed6380-62fc-4239-81a7-b1368b9a529b-tmp-dir\") pod \"test-model-db5ef5-kserve-6d4b5bbd85-5jhrf\" (UID: \"c1ed6380-62fc-4239-81a7-b1368b9a529b\") " pod="ai-tenant-e2e-inf-a-db5ef5/test-model-db5ef5-kserve-6d4b5bbd85-5jhrf" Nov 04 03:12:54.168816 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:12:54.168756 4405 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c1ed6380-62fc-4239-81a7-b1368b9a529b-home\") pod \"test-model-db5ef5-kserve-6d4b5bbd85-5jhrf\" (UID: \"c1ed6380-62fc-4239-81a7-b1368b9a529b\") " pod="ai-tenant-e2e-inf-a-db5ef5/test-model-db5ef5-kserve-6d4b5bbd85-5jhrf" Nov 04 03:12:54.168816 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:12:54.168790 4405 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c1ed6380-62fc-4239-81a7-b1368b9a529b-model-cache\") pod \"test-model-db5ef5-kserve-6d4b5bbd85-5jhrf\" (UID: \"c1ed6380-62fc-4239-81a7-b1368b9a529b\") " pod="ai-tenant-e2e-inf-a-db5ef5/test-model-db5ef5-kserve-6d4b5bbd85-5jhrf" Nov 04 03:12:54.170764 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:12:54.170742 4405 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c1ed6380-62fc-4239-81a7-b1368b9a529b-dshm\") pod \"test-model-db5ef5-kserve-6d4b5bbd85-5jhrf\" (UID: \"c1ed6380-62fc-4239-81a7-b1368b9a529b\") " pod="ai-tenant-e2e-inf-a-db5ef5/test-model-db5ef5-kserve-6d4b5bbd85-5jhrf" Nov 04 03:12:54.170932 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:12:54.170910 4405 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c1ed6380-62fc-4239-81a7-b1368b9a529b-tls-certs\") pod \"test-model-db5ef5-kserve-6d4b5bbd85-5jhrf\" (UID: \"c1ed6380-62fc-4239-81a7-b1368b9a529b\") " pod="ai-tenant-e2e-inf-a-db5ef5/test-model-db5ef5-kserve-6d4b5bbd85-5jhrf" Nov 04 03:12:54.229349 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:12:54.229316 4405 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7525p\" (UniqueName: \"kubernetes.io/projected/c1ed6380-62fc-4239-81a7-b1368b9a529b-kube-api-access-7525p\") pod \"test-model-db5ef5-kserve-6d4b5bbd85-5jhrf\" (UID: \"c1ed6380-62fc-4239-81a7-b1368b9a529b\") " pod="ai-tenant-e2e-inf-a-db5ef5/test-model-db5ef5-kserve-6d4b5bbd85-5jhrf" Nov 04 03:12:54.304092 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:12:54.303997 4405 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-db5ef5/test-model-db5ef5-kserve-6d4b5bbd85-5jhrf" Nov 04 03:12:54.682591 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:12:54.682555 4405 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-db5ef5/test-model-db5ef5-kserve-6d4b5bbd85-5jhrf"] Nov 04 03:12:54.684307 ip-10-0-134-125 kubenswrapper[4405]: W1104 03:12:54.684275 4405 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podc1ed6380_62fc_4239_81a7_b1368b9a529b.slice/crio-6eb339a8a983a62fd6b6f25bf0f83b84d1b3139b718273e9f3fc369389f6029f WatchSource:0}: Error finding container 6eb339a8a983a62fd6b6f25bf0f83b84d1b3139b718273e9f3fc369389f6029f: Status 404 returned error can't find the container with id 6eb339a8a983a62fd6b6f25bf0f83b84d1b3139b718273e9f3fc369389f6029f Nov 04 03:12:54.755352 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:12:54.755315 4405 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-db5ef5/test-model-db5ef5-kserve-6d4b5bbd85-5jhrf" event={"ID":"c1ed6380-62fc-4239-81a7-b1368b9a529b","Type":"ContainerStarted","Data":"0520a36cdc8466525c94e01bf56438e375459208f93a5642950f7eba50fa8840"} Nov 04 03:12:54.755352 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:12:54.755363 4405 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-db5ef5/test-model-db5ef5-kserve-6d4b5bbd85-5jhrf" event={"ID":"c1ed6380-62fc-4239-81a7-b1368b9a529b","Type":"ContainerStarted","Data":"6eb339a8a983a62fd6b6f25bf0f83b84d1b3139b718273e9f3fc369389f6029f"} Nov 04 03:12:54.874332 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:12:54.874273 4405 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-inf-a-db5ef5/test-model-db5ef5-kserve-6d4b5bbd85-5jhrf" podStartSLOduration=1.8742586079999999 podStartE2EDuration="1.874258608s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 03:12:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 03:12:54.872546304 +0000 UTC m=+2421.303246423" watchObservedRunningTime="2026-11-04 03:12:54.874258608 +0000 UTC m=+2421.304958728" Nov 04 03:12:55.304605 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:12:55.304564 4405 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-a-db5ef5/test-model-db5ef5-kserve-6d4b5bbd85-5jhrf" Nov 04 03:13:04.305061 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:13:04.305025 4405 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-a-db5ef5/test-model-db5ef5-kserve-6d4b5bbd85-5jhrf" Nov 04 03:13:04.318110 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:13:04.318081 4405 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-a-db5ef5/test-model-db5ef5-kserve-6d4b5bbd85-5jhrf" Nov 04 03:13:04.825274 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:13:04.825239 4405 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-a-db5ef5/test-model-db5ef5-kserve-6d4b5bbd85-5jhrf" Nov 04 03:13:06.883738 ip-10-0-134-125 kubenswrapper[4405]: E1104 03:13:06.883701 4405 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5f2vb" containerName="extractor" Nov 04 03:13:13.542098 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:13:13.542064 4405 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-bl5mg"] Nov 04 03:13:33.809917 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:13:33.809876 4405 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-bl5mg"] Nov 04 03:13:36.883939 ip-10-0-134-125 kubenswrapper[4405]: E1104 03:13:36.883899 4405 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5f2vb" containerName="extractor" Nov 04 03:13:54.391798 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:13:54.391714 4405 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-bl5mg"] Nov 04 03:13:59.095510 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:13:59.095476 4405 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-4cbcf738-67bd6b6f59-mfssb"] Nov 04 03:14:04.097203 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:14:04.097132 4405 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-mt-4cbcf738-67bd6b6f59-mfssb" podUID="8b2f727a-24b4-4768-81c5-4ee22b3b5f26" containerName="payload-pre-processing" containerID="cri-o://81f9bc01d27cc3812c541659169ce05bc5d3299d4c26694bc621a3a2b120a8bf" gracePeriod=25 Nov 04 03:14:04.380398 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:14:04.380367 4405 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-4cbcf738-67bd6b6f59-mfssb" Nov 04 03:14:04.470181 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:14:04.470138 4405 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8b2f727a-24b4-4768-81c5-4ee22b3b5f26-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8b2f727a-24b4-4768-81c5-4ee22b3b5f26-plugins-config-volume\") pod \"8b2f727a-24b4-4768-81c5-4ee22b3b5f26\" (UID: \"8b2f727a-24b4-4768-81c5-4ee22b3b5f26\") " Nov 04 03:14:04.470364 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:14:04.470342 4405 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8b2f727a-24b4-4768-81c5-4ee22b3b5f26-kube-api-access-szbfw\" (UniqueName: \"kubernetes.io/projected/8b2f727a-24b4-4768-81c5-4ee22b3b5f26-kube-api-access-szbfw\") pod \"8b2f727a-24b4-4768-81c5-4ee22b3b5f26\" (UID: \"8b2f727a-24b4-4768-81c5-4ee22b3b5f26\") " Nov 04 03:14:04.470577 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:14:04.470553 4405 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8b2f727a-24b4-4768-81c5-4ee22b3b5f26-plugins-config-volume" pod "8b2f727a-24b4-4768-81c5-4ee22b3b5f26" (UID: "8b2f727a-24b4-4768-81c5-4ee22b3b5f26"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 03:14:04.470790 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:14:04.470774 4405 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8b2f727a-24b4-4768-81c5-4ee22b3b5f26-plugins-config-volume\") on node \"ip-10-0-134-125.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:14:04.472550 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:14:04.472520 4405 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8b2f727a-24b4-4768-81c5-4ee22b3b5f26-kube-api-access-szbfw" pod "8b2f727a-24b4-4768-81c5-4ee22b3b5f26" (UID: "8b2f727a-24b4-4768-81c5-4ee22b3b5f26"). InnerVolumeSpecName "kube-api-access-szbfw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 03:14:04.571903 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:14:04.571862 4405 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-szbfw\" (UniqueName: \"kubernetes.io/projected/8b2f727a-24b4-4768-81c5-4ee22b3b5f26-kube-api-access-szbfw\") on node \"ip-10-0-134-125.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:14:05.173401 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:14:05.173362 4405 generic.go:356] "Generic (PLEG): container finished" podID="8b2f727a-24b4-4768-81c5-4ee22b3b5f26" containerID="81f9bc01d27cc3812c541659169ce05bc5d3299d4c26694bc621a3a2b120a8bf" exitCode=0 Nov 04 03:14:05.173992 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:14:05.173412 4405 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-4cbcf738-67bd6b6f59-mfssb" Nov 04 03:14:05.173992 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:14:05.173425 4405 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-4cbcf738-67bd6b6f59-mfssb" event={"ID":"8b2f727a-24b4-4768-81c5-4ee22b3b5f26","Type":"ContainerDied","Data":"81f9bc01d27cc3812c541659169ce05bc5d3299d4c26694bc621a3a2b120a8bf"} Nov 04 03:14:05.173992 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:14:05.173481 4405 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-4cbcf738-67bd6b6f59-mfssb" event={"ID":"8b2f727a-24b4-4768-81c5-4ee22b3b5f26","Type":"ContainerDied","Data":"c183efe5ade9477d7d4812ce256d8cd28f0fe93bb1b34b9baf1b6d46fdca3efb"} Nov 04 03:14:05.173992 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:14:05.173511 4405 scope.go:122] "RemoveContainer" containerID="81f9bc01d27cc3812c541659169ce05bc5d3299d4c26694bc621a3a2b120a8bf" Nov 04 03:14:05.185097 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:14:05.185075 4405 scope.go:122] "RemoveContainer" containerID="81f9bc01d27cc3812c541659169ce05bc5d3299d4c26694bc621a3a2b120a8bf" Nov 04 03:14:05.185397 ip-10-0-134-125 kubenswrapper[4405]: E1104 03:14:05.185375 4405 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"81f9bc01d27cc3812c541659169ce05bc5d3299d4c26694bc621a3a2b120a8bf\": container with ID starting with 81f9bc01d27cc3812c541659169ce05bc5d3299d4c26694bc621a3a2b120a8bf not found: ID does not exist" containerID="81f9bc01d27cc3812c541659169ce05bc5d3299d4c26694bc621a3a2b120a8bf" Nov 04 03:14:05.185473 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:14:05.185407 4405 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"81f9bc01d27cc3812c541659169ce05bc5d3299d4c26694bc621a3a2b120a8bf"} err="failed to get container status \"81f9bc01d27cc3812c541659169ce05bc5d3299d4c26694bc621a3a2b120a8bf\": rpc error: code = NotFound desc = could not find container \"81f9bc01d27cc3812c541659169ce05bc5d3299d4c26694bc621a3a2b120a8bf\": container with ID starting with 81f9bc01d27cc3812c541659169ce05bc5d3299d4c26694bc621a3a2b120a8bf not found: ID does not exist" Nov 04 03:14:05.200366 ip-10-0-134-125 kubenswrapper[4405]: E1104 03:14:05.200337 4405 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8b2f727a_24b4_4768_81c5_4ee22b3b5f26.slice/crio-c183efe5ade9477d7d4812ce256d8cd28f0fe93bb1b34b9baf1b6d46fdca3efb\": RecentStats: unable to find data in memory cache]" Nov 04 03:14:05.336960 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:14:05.336913 4405 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-4cbcf738-67bd6b6f59-mfssb"] Nov 04 03:14:05.386466 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:14:05.386430 4405 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-4cbcf738-67bd6b6f59-mfssb"] Nov 04 03:14:05.700835 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:14:05.700801 4405 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8b2f727a-24b4-4768-81c5-4ee22b3b5f26" path="/var/lib/kubelet/pods/8b2f727a-24b4-4768-81c5-4ee22b3b5f26/volumes" Nov 04 03:14:06.884506 ip-10-0-134-125 kubenswrapper[4405]: E1104 03:14:06.884468 4405 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5f2vb" containerName="extractor" Nov 04 03:14:28.611972 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:14:28.611933 4405 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-bl5mg"] Nov 04 03:14:36.885533 ip-10-0-134-125 kubenswrapper[4405]: E1104 03:14:36.885494 4405 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5f2vb" containerName="extractor" Nov 04 03:14:55.330997 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:14:55.329884 4405 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-4cbcf738-97f795756-gk52w"] Nov 04 03:14:55.331447 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:14:55.331160 4405 cpu_manager.go:414] "containerMap: removing container" podUID="8b2f727a-24b4-4768-81c5-4ee22b3b5f26" containerName="payload-pre-processing" Nov 04 03:14:55.331447 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:14:55.331186 4405 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8b2f727a-24b4-4768-81c5-4ee22b3b5f26" containerName="payload-pre-processing" Nov 04 03:14:55.331447 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:14:55.331203 4405 cpu_manager.go:414] "containerMap: removing container" podUID="dad304e5-de0a-44b2-a137-8139a2aa2f52" containerName="istio-proxy" Nov 04 03:14:55.331447 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:14:55.331209 4405 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dad304e5-de0a-44b2-a137-8139a2aa2f52" containerName="istio-proxy" Nov 04 03:14:55.331649 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:14:55.331471 4405 memory_manager.go:365] "RemoveStaleState removing state" podUID="dad304e5-de0a-44b2-a137-8139a2aa2f52" containerName="istio-proxy" Nov 04 03:14:55.331649 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:14:55.331485 4405 memory_manager.go:365] "RemoveStaleState removing state" podUID="8b2f727a-24b4-4768-81c5-4ee22b3b5f26" containerName="payload-pre-processing" Nov 04 03:14:55.336096 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:14:55.336076 4405 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-4cbcf738-97f795756-gk52w" Nov 04 03:14:55.381450 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:14:55.381413 4405 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-4cbcf738-97f795756-gk52w"] Nov 04 03:14:55.427785 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:14:55.427751 4405 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-4cbcf738\"" Nov 04 03:14:55.427949 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:14:55.427843 4405 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-mt-4cbcf738-dockercfg-v9tdb\"" Nov 04 03:14:55.463378 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:14:55.463327 4405 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jdjgc\" (UniqueName: \"kubernetes.io/projected/a061347f-23bc-43c9-9b1a-594d184ebe7c-kube-api-access-jdjgc\") pod \"payload-processing-e2e-mt-4cbcf738-97f795756-gk52w\" (UID: \"a061347f-23bc-43c9-9b1a-594d184ebe7c\") " pod="openshift-ingress/payload-processing-e2e-mt-4cbcf738-97f795756-gk52w" Nov 04 03:14:55.463555 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:14:55.463439 4405 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a061347f-23bc-43c9-9b1a-594d184ebe7c-plugins-config-volume\") pod \"payload-processing-e2e-mt-4cbcf738-97f795756-gk52w\" (UID: \"a061347f-23bc-43c9-9b1a-594d184ebe7c\") " pod="openshift-ingress/payload-processing-e2e-mt-4cbcf738-97f795756-gk52w" Nov 04 03:14:55.487854 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:14:55.487821 4405 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-4cbcf738-67bd6b6f59-59zp6"] Nov 04 03:14:55.491782 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:14:55.491754 4405 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-4cbcf738-67bd6b6f59-59zp6" Nov 04 03:14:55.549511 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:14:55.549475 4405 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-4cbcf738-67bd6b6f59-59zp6"] Nov 04 03:14:55.565164 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:14:55.565125 4405 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9a57520d-dae3-45e9-8558-fa824bd746cd-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-4cbcf738-67bd6b6f59-59zp6\" (UID: \"9a57520d-dae3-45e9-8558-fa824bd746cd\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-4cbcf738-67bd6b6f59-59zp6" Nov 04 03:14:55.565342 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:14:55.565254 4405 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jdjgc\" (UniqueName: \"kubernetes.io/projected/a061347f-23bc-43c9-9b1a-594d184ebe7c-kube-api-access-jdjgc\") pod \"payload-processing-e2e-mt-4cbcf738-97f795756-gk52w\" (UID: \"a061347f-23bc-43c9-9b1a-594d184ebe7c\") " pod="openshift-ingress/payload-processing-e2e-mt-4cbcf738-97f795756-gk52w" Nov 04 03:14:55.565342 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:14:55.565316 4405 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a061347f-23bc-43c9-9b1a-594d184ebe7c-plugins-config-volume\") pod \"payload-processing-e2e-mt-4cbcf738-97f795756-gk52w\" (UID: \"a061347f-23bc-43c9-9b1a-594d184ebe7c\") " pod="openshift-ingress/payload-processing-e2e-mt-4cbcf738-97f795756-gk52w" Nov 04 03:14:55.565542 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:14:55.565374 4405 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-prhqc\" (UniqueName: \"kubernetes.io/projected/9a57520d-dae3-45e9-8558-fa824bd746cd-kube-api-access-prhqc\") pod \"payload-pre-processing-e2e-mt-4cbcf738-67bd6b6f59-59zp6\" (UID: \"9a57520d-dae3-45e9-8558-fa824bd746cd\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-4cbcf738-67bd6b6f59-59zp6" Nov 04 03:14:55.566291 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:14:55.566266 4405 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a061347f-23bc-43c9-9b1a-594d184ebe7c-plugins-config-volume\") pod \"payload-processing-e2e-mt-4cbcf738-97f795756-gk52w\" (UID: \"a061347f-23bc-43c9-9b1a-594d184ebe7c\") " pod="openshift-ingress/payload-processing-e2e-mt-4cbcf738-97f795756-gk52w" Nov 04 03:14:55.626526 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:14:55.626500 4405 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jdjgc\" (UniqueName: \"kubernetes.io/projected/a061347f-23bc-43c9-9b1a-594d184ebe7c-kube-api-access-jdjgc\") pod \"payload-processing-e2e-mt-4cbcf738-97f795756-gk52w\" (UID: \"a061347f-23bc-43c9-9b1a-594d184ebe7c\") " pod="openshift-ingress/payload-processing-e2e-mt-4cbcf738-97f795756-gk52w" Nov 04 03:14:55.652833 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:14:55.652799 4405 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-4cbcf738-97f795756-gk52w" Nov 04 03:14:55.666494 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:14:55.666467 4405 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-prhqc\" (UniqueName: \"kubernetes.io/projected/9a57520d-dae3-45e9-8558-fa824bd746cd-kube-api-access-prhqc\") pod \"payload-pre-processing-e2e-mt-4cbcf738-67bd6b6f59-59zp6\" (UID: \"9a57520d-dae3-45e9-8558-fa824bd746cd\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-4cbcf738-67bd6b6f59-59zp6" Nov 04 03:14:55.666596 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:14:55.666577 4405 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9a57520d-dae3-45e9-8558-fa824bd746cd-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-4cbcf738-67bd6b6f59-59zp6\" (UID: \"9a57520d-dae3-45e9-8558-fa824bd746cd\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-4cbcf738-67bd6b6f59-59zp6" Nov 04 03:14:55.667234 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:14:55.667210 4405 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9a57520d-dae3-45e9-8558-fa824bd746cd-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-4cbcf738-67bd6b6f59-59zp6\" (UID: \"9a57520d-dae3-45e9-8558-fa824bd746cd\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-4cbcf738-67bd6b6f59-59zp6" Nov 04 03:14:55.728790 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:14:55.728723 4405 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-prhqc\" (UniqueName: \"kubernetes.io/projected/9a57520d-dae3-45e9-8558-fa824bd746cd-kube-api-access-prhqc\") pod \"payload-pre-processing-e2e-mt-4cbcf738-67bd6b6f59-59zp6\" (UID: \"9a57520d-dae3-45e9-8558-fa824bd746cd\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-4cbcf738-67bd6b6f59-59zp6" Nov 04 03:14:55.805302 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:14:55.805268 4405 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-4cbcf738-67bd6b6f59-59zp6" Nov 04 03:14:56.031371 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:14:56.031350 4405 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-4cbcf738-97f795756-gk52w"] Nov 04 03:14:56.033539 ip-10-0-134-125 kubenswrapper[4405]: W1104 03:14:56.033512 4405 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda061347f_23bc_43c9_9b1a_594d184ebe7c.slice/crio-cab74d8be4c374064774c501f8f44f6de1be5e4da19b80eead1028b4fd4f59e9 WatchSource:0}: Error finding container cab74d8be4c374064774c501f8f44f6de1be5e4da19b80eead1028b4fd4f59e9: Status 404 returned error can't find the container with id cab74d8be4c374064774c501f8f44f6de1be5e4da19b80eead1028b4fd4f59e9 Nov 04 03:14:56.183439 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:14:56.183413 4405 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-4cbcf738-67bd6b6f59-59zp6"] Nov 04 03:14:56.185138 ip-10-0-134-125 kubenswrapper[4405]: W1104 03:14:56.185106 4405 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9a57520d_dae3_45e9_8558_fa824bd746cd.slice/crio-d7c4f85fe165bd93fcd5e6fa525a5a11c6abe5795eff54d3bd216ca8f6e1f9af WatchSource:0}: Error finding container d7c4f85fe165bd93fcd5e6fa525a5a11c6abe5795eff54d3bd216ca8f6e1f9af: Status 404 returned error can't find the container with id d7c4f85fe165bd93fcd5e6fa525a5a11c6abe5795eff54d3bd216ca8f6e1f9af Nov 04 03:14:56.467320 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:14:56.467279 4405 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-4cbcf738-67bd6b6f59-59zp6" event={"ID":"9a57520d-dae3-45e9-8558-fa824bd746cd","Type":"ContainerStarted","Data":"7d89adf2a9e729eb6c81da588e035da142f4c56ffd1b03d4d7662f7f945c615f"} Nov 04 03:14:56.467879 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:14:56.467334 4405 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-4cbcf738-67bd6b6f59-59zp6" event={"ID":"9a57520d-dae3-45e9-8558-fa824bd746cd","Type":"ContainerStarted","Data":"d7c4f85fe165bd93fcd5e6fa525a5a11c6abe5795eff54d3bd216ca8f6e1f9af"} Nov 04 03:14:56.467879 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:14:56.467362 4405 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-4cbcf738-67bd6b6f59-59zp6" Nov 04 03:14:56.468885 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:14:56.468859 4405 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-4cbcf738-97f795756-gk52w" event={"ID":"a061347f-23bc-43c9-9b1a-594d184ebe7c","Type":"ContainerStarted","Data":"fd96e616ff38ce0f28c88780b448bf1936b3b30cd669c21b733b281f5b0c4e57"} Nov 04 03:14:56.469033 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:14:56.468908 4405 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-mt-4cbcf738-97f795756-gk52w" Nov 04 03:14:56.469033 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:14:56.468924 4405 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-4cbcf738-97f795756-gk52w" event={"ID":"a061347f-23bc-43c9-9b1a-594d184ebe7c","Type":"ContainerStarted","Data":"cab74d8be4c374064774c501f8f44f6de1be5e4da19b80eead1028b4fd4f59e9"} Nov 04 03:14:56.583418 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:14:56.583370 4405 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-mt-4cbcf738-67bd6b6f59-59zp6" podStartSLOduration=1.58335549 podStartE2EDuration="1.58335549s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 03:14:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 03:14:56.581941363 +0000 UTC m=+2543.012641484" watchObservedRunningTime="2026-11-04 03:14:56.58335549 +0000 UTC m=+2543.014055629" Nov 04 03:14:56.700368 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:14:56.700307 4405 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-mt-4cbcf738-97f795756-gk52w" podStartSLOduration=1.700291338 podStartE2EDuration="1.700291338s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 03:14:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 03:14:56.69949915 +0000 UTC m=+2543.130199271" watchObservedRunningTime="2026-11-04 03:14:56.700291338 +0000 UTC m=+2543.130991480" Nov 04 03:15:03.355242 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:15:03.355195 4405 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-bl5mg"] Nov 04 03:15:06.886027 ip-10-0-134-125 kubenswrapper[4405]: E1104 03:15:06.885988 4405 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5f2vb" containerName="extractor" Nov 04 03:15:07.477977 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:15:07.477941 4405 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-4cbcf738-67bd6b6f59-59zp6" Nov 04 03:15:07.478163 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:15:07.478019 4405 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-mt-4cbcf738-97f795756-gk52w" Nov 04 03:15:19.362340 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:15:19.362305 4405 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-4cbcf738-7b5dfbb998-d5bb8"] Nov 04 03:15:19.366348 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:15:19.366326 4405 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-4cbcf738-7b5dfbb998-d5bb8" Nov 04 03:15:19.419059 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:15:19.419024 4405 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-mt-4cbcf738-serving-cert\"" Nov 04 03:15:19.443296 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:15:19.443256 4405 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/73c5ca67-bc51-492d-b055-0cbaf3a85b07-metrics-certs\") pod \"maas-api-e2e-mt-4cbcf738-7b5dfbb998-d5bb8\" (UID: \"73c5ca67-bc51-492d-b055-0cbaf3a85b07\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-4cbcf738-7b5dfbb998-d5bb8" Nov 04 03:15:19.443502 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:15:19.443350 4405 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/73c5ca67-bc51-492d-b055-0cbaf3a85b07-maas-api-tls\") pod \"maas-api-e2e-mt-4cbcf738-7b5dfbb998-d5bb8\" (UID: \"73c5ca67-bc51-492d-b055-0cbaf3a85b07\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-4cbcf738-7b5dfbb998-d5bb8" Nov 04 03:15:19.443575 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:15:19.443531 4405 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5vgz7\" (UniqueName: \"kubernetes.io/projected/73c5ca67-bc51-492d-b055-0cbaf3a85b07-kube-api-access-5vgz7\") pod \"maas-api-e2e-mt-4cbcf738-7b5dfbb998-d5bb8\" (UID: \"73c5ca67-bc51-492d-b055-0cbaf3a85b07\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-4cbcf738-7b5dfbb998-d5bb8" Nov 04 03:15:19.443634 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:15:19.443591 4405 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/73c5ca67-bc51-492d-b055-0cbaf3a85b07-openshift-service-ca\") pod \"maas-api-e2e-mt-4cbcf738-7b5dfbb998-d5bb8\" (UID: \"73c5ca67-bc51-492d-b055-0cbaf3a85b07\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-4cbcf738-7b5dfbb998-d5bb8" Nov 04 03:15:19.472007 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:15:19.471972 4405 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-4cbcf738-7b5dfbb998-d5bb8"] Nov 04 03:15:19.550691 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:15:19.550339 4405 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/73c5ca67-bc51-492d-b055-0cbaf3a85b07-openshift-service-ca\") pod \"maas-api-e2e-mt-4cbcf738-7b5dfbb998-d5bb8\" (UID: \"73c5ca67-bc51-492d-b055-0cbaf3a85b07\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-4cbcf738-7b5dfbb998-d5bb8" Nov 04 03:15:19.550691 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:15:19.550460 4405 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/73c5ca67-bc51-492d-b055-0cbaf3a85b07-metrics-certs\") pod \"maas-api-e2e-mt-4cbcf738-7b5dfbb998-d5bb8\" (UID: \"73c5ca67-bc51-492d-b055-0cbaf3a85b07\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-4cbcf738-7b5dfbb998-d5bb8" Nov 04 03:15:19.550691 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:15:19.550521 4405 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/73c5ca67-bc51-492d-b055-0cbaf3a85b07-maas-api-tls\") pod \"maas-api-e2e-mt-4cbcf738-7b5dfbb998-d5bb8\" (UID: \"73c5ca67-bc51-492d-b055-0cbaf3a85b07\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-4cbcf738-7b5dfbb998-d5bb8" Nov 04 03:15:19.550691 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:15:19.550631 4405 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5vgz7\" (UniqueName: \"kubernetes.io/projected/73c5ca67-bc51-492d-b055-0cbaf3a85b07-kube-api-access-5vgz7\") pod \"maas-api-e2e-mt-4cbcf738-7b5dfbb998-d5bb8\" (UID: \"73c5ca67-bc51-492d-b055-0cbaf3a85b07\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-4cbcf738-7b5dfbb998-d5bb8" Nov 04 03:15:19.551064 ip-10-0-134-125 kubenswrapper[4405]: E1104 03:15:19.550763 4405 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-mt-4cbcf738-serving-cert: secret "maas-api-e2e-mt-4cbcf738-serving-cert" not found Nov 04 03:15:19.551064 ip-10-0-134-125 kubenswrapper[4405]: E1104 03:15:19.550848 4405 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/73c5ca67-bc51-492d-b055-0cbaf3a85b07-maas-api-tls podName:73c5ca67-bc51-492d-b055-0cbaf3a85b07 nodeName:}" failed. No retries permitted until 2026-11-04 03:15:20.05082615 +0000 UTC m=+2566.481526248 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/73c5ca67-bc51-492d-b055-0cbaf3a85b07-maas-api-tls") pod "maas-api-e2e-mt-4cbcf738-7b5dfbb998-d5bb8" (UID: "73c5ca67-bc51-492d-b055-0cbaf3a85b07") : secret "maas-api-e2e-mt-4cbcf738-serving-cert" not found Nov 04 03:15:19.559936 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:15:19.551745 4405 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/73c5ca67-bc51-492d-b055-0cbaf3a85b07-openshift-service-ca\") pod \"maas-api-e2e-mt-4cbcf738-7b5dfbb998-d5bb8\" (UID: \"73c5ca67-bc51-492d-b055-0cbaf3a85b07\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-4cbcf738-7b5dfbb998-d5bb8" Nov 04 03:15:19.564387 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:15:19.564362 4405 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/73c5ca67-bc51-492d-b055-0cbaf3a85b07-metrics-certs\") pod \"maas-api-e2e-mt-4cbcf738-7b5dfbb998-d5bb8\" (UID: \"73c5ca67-bc51-492d-b055-0cbaf3a85b07\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-4cbcf738-7b5dfbb998-d5bb8" Nov 04 03:15:19.642815 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:15:19.641926 4405 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5vgz7\" (UniqueName: \"kubernetes.io/projected/73c5ca67-bc51-492d-b055-0cbaf3a85b07-kube-api-access-5vgz7\") pod \"maas-api-e2e-mt-4cbcf738-7b5dfbb998-d5bb8\" (UID: \"73c5ca67-bc51-492d-b055-0cbaf3a85b07\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-4cbcf738-7b5dfbb998-d5bb8" Nov 04 03:15:20.058156 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:15:20.058072 4405 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/73c5ca67-bc51-492d-b055-0cbaf3a85b07-maas-api-tls\") pod \"maas-api-e2e-mt-4cbcf738-7b5dfbb998-d5bb8\" (UID: \"73c5ca67-bc51-492d-b055-0cbaf3a85b07\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-4cbcf738-7b5dfbb998-d5bb8" Nov 04 03:15:20.060737 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:15:20.060707 4405 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/73c5ca67-bc51-492d-b055-0cbaf3a85b07-maas-api-tls\") pod \"maas-api-e2e-mt-4cbcf738-7b5dfbb998-d5bb8\" (UID: \"73c5ca67-bc51-492d-b055-0cbaf3a85b07\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-4cbcf738-7b5dfbb998-d5bb8" Nov 04 03:15:20.282099 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:15:20.282058 4405 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-4cbcf738-7b5dfbb998-d5bb8" Nov 04 03:15:20.662583 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:15:20.662551 4405 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-4cbcf738-7b5dfbb998-d5bb8"] Nov 04 03:15:20.665047 ip-10-0-134-125 kubenswrapper[4405]: W1104 03:15:20.665004 4405 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod73c5ca67_bc51_492d_b055_0cbaf3a85b07.slice/crio-350c684d614d271b303a4dedd03348238e88c0d56f047c2a82d39d38f97d5087 WatchSource:0}: Error finding container 350c684d614d271b303a4dedd03348238e88c0d56f047c2a82d39d38f97d5087: Status 404 returned error can't find the container with id 350c684d614d271b303a4dedd03348238e88c0d56f047c2a82d39d38f97d5087 Nov 04 03:15:20.666345 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:15:20.666324 4405 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 04 03:15:21.610431 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:15:21.610392 4405 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-4cbcf738-7b5dfbb998-d5bb8" event={"ID":"73c5ca67-bc51-492d-b055-0cbaf3a85b07","Type":"ContainerStarted","Data":"350c684d614d271b303a4dedd03348238e88c0d56f047c2a82d39d38f97d5087"} Nov 04 03:15:22.619636 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:15:22.619598 4405 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-4cbcf738-7b5dfbb998-d5bb8" event={"ID":"73c5ca67-bc51-492d-b055-0cbaf3a85b07","Type":"ContainerStarted","Data":"e8d1a5666eb7317340bf5fce328af705481496dfdeffbcb56490374b3b7e1181"} Nov 04 03:15:22.620048 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:15:22.619691 4405 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-4cbcf738-7b5dfbb998-d5bb8" Nov 04 03:15:22.738163 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:15:22.738096 4405 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-mt-4cbcf738-7b5dfbb998-d5bb8" podStartSLOduration=2.752543722 podStartE2EDuration="3.738075712s" totalImagesPullingTime="985.53199ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 03:15:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 03:15:22.736496505 +0000 UTC m=+2569.167196626" watchObservedRunningTime="2026-11-04 03:15:22.738075712 +0000 UTC m=+2569.168775834" Nov 04 03:15:23.228030 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:15:23.227987 4405 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-bl5mg"] Nov 04 03:15:28.630090 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:15:28.630060 4405 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-4cbcf738-7b5dfbb998-d5bb8" Nov 04 03:15:36.886785 ip-10-0-134-125 kubenswrapper[4405]: E1104 03:15:36.886748 4405 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5f2vb" containerName="extractor" Nov 04 03:15:49.484986 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:15:49.484952 4405 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-bl5mg"] Nov 04 03:16:01.465316 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:16:01.465282 4405 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-db5ef5/test-model-db5ef5-kserve-6d4b5bbd85-5jhrf"] Nov 04 03:16:06.883761 ip-10-0-134-125 kubenswrapper[4405]: E1104 03:16:06.883723 4405 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5f2vb" containerName="extractor" Nov 04 03:16:10.567836 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:16:10.567801 4405 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-bl5mg"] Nov 04 03:16:16.486825 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:16:16.486756 4405 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-inf-a-db5ef5/test-model-db5ef5-kserve-6d4b5bbd85-5jhrf" podUID="c1ed6380-62fc-4239-81a7-b1368b9a529b" containerName="main" containerID="cri-o://0520a36cdc8466525c94e01bf56438e375459208f93a5642950f7eba50fa8840" gracePeriod=45 Nov 04 03:16:16.763117 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:16:16.763093 4405 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-db5ef5/test-model-db5ef5-kserve-6d4b5bbd85-5jhrf" Nov 04 03:16:16.820815 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:16:16.820784 4405 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c1ed6380-62fc-4239-81a7-b1368b9a529b-home\" (UniqueName: \"kubernetes.io/empty-dir/c1ed6380-62fc-4239-81a7-b1368b9a529b-home\") pod \"c1ed6380-62fc-4239-81a7-b1368b9a529b\" (UID: \"c1ed6380-62fc-4239-81a7-b1368b9a529b\") " Nov 04 03:16:16.820980 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:16:16.820892 4405 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c1ed6380-62fc-4239-81a7-b1368b9a529b-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c1ed6380-62fc-4239-81a7-b1368b9a529b-model-cache\") pod \"c1ed6380-62fc-4239-81a7-b1368b9a529b\" (UID: \"c1ed6380-62fc-4239-81a7-b1368b9a529b\") " Nov 04 03:16:16.820980 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:16:16.820912 4405 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c1ed6380-62fc-4239-81a7-b1368b9a529b-dshm\" (UniqueName: \"kubernetes.io/empty-dir/c1ed6380-62fc-4239-81a7-b1368b9a529b-dshm\") pod \"c1ed6380-62fc-4239-81a7-b1368b9a529b\" (UID: \"c1ed6380-62fc-4239-81a7-b1368b9a529b\") " Nov 04 03:16:16.820980 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:16:16.820952 4405 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c1ed6380-62fc-4239-81a7-b1368b9a529b-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c1ed6380-62fc-4239-81a7-b1368b9a529b-tmp-dir\") pod \"c1ed6380-62fc-4239-81a7-b1368b9a529b\" (UID: \"c1ed6380-62fc-4239-81a7-b1368b9a529b\") " Nov 04 03:16:16.821108 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:16:16.821021 4405 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c1ed6380-62fc-4239-81a7-b1368b9a529b-tls-certs\" (UniqueName: \"kubernetes.io/secret/c1ed6380-62fc-4239-81a7-b1368b9a529b-tls-certs\") pod \"c1ed6380-62fc-4239-81a7-b1368b9a529b\" (UID: \"c1ed6380-62fc-4239-81a7-b1368b9a529b\") " Nov 04 03:16:16.821108 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:16:16.821068 4405 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c1ed6380-62fc-4239-81a7-b1368b9a529b-home" pod "c1ed6380-62fc-4239-81a7-b1368b9a529b" (UID: "c1ed6380-62fc-4239-81a7-b1368b9a529b"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 03:16:16.821216 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:16:16.821118 4405 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c1ed6380-62fc-4239-81a7-b1368b9a529b-kube-api-access-7525p\" (UniqueName: \"kubernetes.io/projected/c1ed6380-62fc-4239-81a7-b1368b9a529b-kube-api-access-7525p\") pod \"c1ed6380-62fc-4239-81a7-b1368b9a529b\" (UID: \"c1ed6380-62fc-4239-81a7-b1368b9a529b\") " Nov 04 03:16:16.821278 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:16:16.821192 4405 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c1ed6380-62fc-4239-81a7-b1368b9a529b-model-cache" pod "c1ed6380-62fc-4239-81a7-b1368b9a529b" (UID: "c1ed6380-62fc-4239-81a7-b1368b9a529b"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 03:16:16.821278 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:16:16.821253 4405 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c1ed6380-62fc-4239-81a7-b1368b9a529b-tmp-dir" pod "c1ed6380-62fc-4239-81a7-b1368b9a529b" (UID: "c1ed6380-62fc-4239-81a7-b1368b9a529b"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 03:16:16.821865 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:16:16.821571 4405 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c1ed6380-62fc-4239-81a7-b1368b9a529b-model-cache\") on node \"ip-10-0-134-125.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:16:16.821865 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:16:16.821596 4405 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c1ed6380-62fc-4239-81a7-b1368b9a529b-tmp-dir\") on node \"ip-10-0-134-125.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:16:16.821865 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:16:16.821610 4405 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c1ed6380-62fc-4239-81a7-b1368b9a529b-home\") on node \"ip-10-0-134-125.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:16:16.823313 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:16:16.823287 4405 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c1ed6380-62fc-4239-81a7-b1368b9a529b-dshm" pod "c1ed6380-62fc-4239-81a7-b1368b9a529b" (UID: "c1ed6380-62fc-4239-81a7-b1368b9a529b"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 03:16:16.823432 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:16:16.823419 4405 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c1ed6380-62fc-4239-81a7-b1368b9a529b-tls-certs" pod "c1ed6380-62fc-4239-81a7-b1368b9a529b" (UID: "c1ed6380-62fc-4239-81a7-b1368b9a529b"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 04 03:16:16.823500 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:16:16.823479 4405 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c1ed6380-62fc-4239-81a7-b1368b9a529b-kube-api-access-7525p" pod "c1ed6380-62fc-4239-81a7-b1368b9a529b" (UID: "c1ed6380-62fc-4239-81a7-b1368b9a529b"). InnerVolumeSpecName "kube-api-access-7525p". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 03:16:16.922854 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:16:16.922813 4405 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7525p\" (UniqueName: \"kubernetes.io/projected/c1ed6380-62fc-4239-81a7-b1368b9a529b-kube-api-access-7525p\") on node \"ip-10-0-134-125.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:16:16.922854 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:16:16.922842 4405 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c1ed6380-62fc-4239-81a7-b1368b9a529b-dshm\") on node \"ip-10-0-134-125.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:16:16.922854 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:16:16.922851 4405 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c1ed6380-62fc-4239-81a7-b1368b9a529b-tls-certs\") on node \"ip-10-0-134-125.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:16:16.937880 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:16:16.937848 4405 generic.go:356] "Generic (PLEG): container finished" podID="c1ed6380-62fc-4239-81a7-b1368b9a529b" containerID="0520a36cdc8466525c94e01bf56438e375459208f93a5642950f7eba50fa8840" exitCode=0 Nov 04 03:16:16.938082 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:16:16.937895 4405 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-db5ef5/test-model-db5ef5-kserve-6d4b5bbd85-5jhrf" Nov 04 03:16:16.938082 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:16:16.937931 4405 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-db5ef5/test-model-db5ef5-kserve-6d4b5bbd85-5jhrf" event={"ID":"c1ed6380-62fc-4239-81a7-b1368b9a529b","Type":"ContainerDied","Data":"0520a36cdc8466525c94e01bf56438e375459208f93a5642950f7eba50fa8840"} Nov 04 03:16:16.938082 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:16:16.937973 4405 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-db5ef5/test-model-db5ef5-kserve-6d4b5bbd85-5jhrf" event={"ID":"c1ed6380-62fc-4239-81a7-b1368b9a529b","Type":"ContainerDied","Data":"6eb339a8a983a62fd6b6f25bf0f83b84d1b3139b718273e9f3fc369389f6029f"} Nov 04 03:16:16.938082 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:16:16.937990 4405 scope.go:122] "RemoveContainer" containerID="0520a36cdc8466525c94e01bf56438e375459208f93a5642950f7eba50fa8840" Nov 04 03:16:16.949785 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:16:16.949765 4405 scope.go:122] "RemoveContainer" containerID="0520a36cdc8466525c94e01bf56438e375459208f93a5642950f7eba50fa8840" Nov 04 03:16:16.950076 ip-10-0-134-125 kubenswrapper[4405]: E1104 03:16:16.950054 4405 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0520a36cdc8466525c94e01bf56438e375459208f93a5642950f7eba50fa8840\": container with ID starting with 0520a36cdc8466525c94e01bf56438e375459208f93a5642950f7eba50fa8840 not found: ID does not exist" containerID="0520a36cdc8466525c94e01bf56438e375459208f93a5642950f7eba50fa8840" Nov 04 03:16:16.950180 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:16:16.950081 4405 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0520a36cdc8466525c94e01bf56438e375459208f93a5642950f7eba50fa8840"} err="failed to get container status \"0520a36cdc8466525c94e01bf56438e375459208f93a5642950f7eba50fa8840\": rpc error: code = NotFound desc = could not find container \"0520a36cdc8466525c94e01bf56438e375459208f93a5642950f7eba50fa8840\": container with ID starting with 0520a36cdc8466525c94e01bf56438e375459208f93a5642950f7eba50fa8840 not found: ID does not exist" Nov 04 03:16:17.102185 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:16:17.102157 4405 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-db5ef5/test-model-db5ef5-kserve-6d4b5bbd85-5jhrf"] Nov 04 03:16:17.105543 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:16:17.105518 4405 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-a-db5ef5/test-model-db5ef5-kserve-6d4b5bbd85-5jhrf"] Nov 04 03:16:17.701022 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:16:17.700986 4405 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c1ed6380-62fc-4239-81a7-b1368b9a529b" path="/var/lib/kubelet/pods/c1ed6380-62fc-4239-81a7-b1368b9a529b/volumes" Nov 04 03:16:31.854314 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:16:31.854275 4405 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-bl5mg"] Nov 04 03:16:36.885286 ip-10-0-134-125 kubenswrapper[4405]: E1104 03:16:36.885249 4405 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5f2vb" containerName="extractor" Nov 04 03:16:54.887901 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:16:54.887806 4405 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-bl5mg"] Nov 04 03:16:57.500323 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:16:57.500287 4405 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-4cbcf738-97f795756-gk52w"] Nov 04 03:16:57.611685 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:16:57.611641 4405 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-4cbcf738-67bd6b6f59-59zp6"] Nov 04 03:16:57.960624 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:16:57.960579 4405 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-db5ef5-5d5584b699-rjmpj"] Nov 04 03:16:57.960907 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:16:57.960860 4405 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-db5ef5-5d5584b699-rjmpj" podUID="3d80d037-3e20-4898-855b-86fbcf2bb9aa" containerName="maas-api" containerID="cri-o://e079012dbf9f0a256f342f2823ed63641442c2d25d24122f14a1ce243691052f" gracePeriod=30 Nov 04 03:16:58.202364 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:16:58.202324 4405 generic.go:356] "Generic (PLEG): container finished" podID="3d80d037-3e20-4898-855b-86fbcf2bb9aa" containerID="e079012dbf9f0a256f342f2823ed63641442c2d25d24122f14a1ce243691052f" exitCode=0 Nov 04 03:16:58.202549 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:16:58.202367 4405 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-db5ef5-5d5584b699-rjmpj" event={"ID":"3d80d037-3e20-4898-855b-86fbcf2bb9aa","Type":"ContainerDied","Data":"e079012dbf9f0a256f342f2823ed63641442c2d25d24122f14a1ce243691052f"} Nov 04 03:16:58.226600 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:16:58.226544 4405 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-db5ef5-5d5584b699-rjmpj" Nov 04 03:16:58.301254 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:16:58.301219 4405 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3d80d037-3e20-4898-855b-86fbcf2bb9aa-kube-api-access-5zqv5\" (UniqueName: \"kubernetes.io/projected/3d80d037-3e20-4898-855b-86fbcf2bb9aa-kube-api-access-5zqv5\") pod \"3d80d037-3e20-4898-855b-86fbcf2bb9aa\" (UID: \"3d80d037-3e20-4898-855b-86fbcf2bb9aa\") " Nov 04 03:16:58.301434 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:16:58.301285 4405 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3d80d037-3e20-4898-855b-86fbcf2bb9aa-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3d80d037-3e20-4898-855b-86fbcf2bb9aa-maas-api-tls\") pod \"3d80d037-3e20-4898-855b-86fbcf2bb9aa\" (UID: \"3d80d037-3e20-4898-855b-86fbcf2bb9aa\") " Nov 04 03:16:58.301434 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:16:58.301419 4405 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3d80d037-3e20-4898-855b-86fbcf2bb9aa-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3d80d037-3e20-4898-855b-86fbcf2bb9aa-openshift-service-ca\") pod \"3d80d037-3e20-4898-855b-86fbcf2bb9aa\" (UID: \"3d80d037-3e20-4898-855b-86fbcf2bb9aa\") " Nov 04 03:16:58.301524 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:16:58.301449 4405 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3d80d037-3e20-4898-855b-86fbcf2bb9aa-metrics-certs\" (UniqueName: \"kubernetes.io/secret/3d80d037-3e20-4898-855b-86fbcf2bb9aa-metrics-certs\") pod \"3d80d037-3e20-4898-855b-86fbcf2bb9aa\" (UID: \"3d80d037-3e20-4898-855b-86fbcf2bb9aa\") " Nov 04 03:16:58.301892 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:16:58.301862 4405 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3d80d037-3e20-4898-855b-86fbcf2bb9aa-openshift-service-ca" pod "3d80d037-3e20-4898-855b-86fbcf2bb9aa" (UID: "3d80d037-3e20-4898-855b-86fbcf2bb9aa"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 03:16:58.303788 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:16:58.303756 4405 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3d80d037-3e20-4898-855b-86fbcf2bb9aa-metrics-certs" pod "3d80d037-3e20-4898-855b-86fbcf2bb9aa" (UID: "3d80d037-3e20-4898-855b-86fbcf2bb9aa"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 04 03:16:58.303886 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:16:58.303762 4405 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3d80d037-3e20-4898-855b-86fbcf2bb9aa-kube-api-access-5zqv5" pod "3d80d037-3e20-4898-855b-86fbcf2bb9aa" (UID: "3d80d037-3e20-4898-855b-86fbcf2bb9aa"). InnerVolumeSpecName "kube-api-access-5zqv5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 03:16:58.303886 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:16:58.303817 4405 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3d80d037-3e20-4898-855b-86fbcf2bb9aa-maas-api-tls" pod "3d80d037-3e20-4898-855b-86fbcf2bb9aa" (UID: "3d80d037-3e20-4898-855b-86fbcf2bb9aa"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 04 03:16:58.403138 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:16:58.403094 4405 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3d80d037-3e20-4898-855b-86fbcf2bb9aa-openshift-service-ca\") on node \"ip-10-0-134-125.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:16:58.403138 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:16:58.403132 4405 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3d80d037-3e20-4898-855b-86fbcf2bb9aa-metrics-certs\") on node \"ip-10-0-134-125.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:16:58.403352 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:16:58.403147 4405 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5zqv5\" (UniqueName: \"kubernetes.io/projected/3d80d037-3e20-4898-855b-86fbcf2bb9aa-kube-api-access-5zqv5\") on node \"ip-10-0-134-125.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:16:58.403352 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:16:58.403160 4405 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3d80d037-3e20-4898-855b-86fbcf2bb9aa-maas-api-tls\") on node \"ip-10-0-134-125.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:16:59.211770 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:16:59.211741 4405 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-db5ef5-5d5584b699-rjmpj" Nov 04 03:16:59.211770 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:16:59.211748 4405 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-db5ef5-5d5584b699-rjmpj" event={"ID":"3d80d037-3e20-4898-855b-86fbcf2bb9aa","Type":"ContainerDied","Data":"45324060e1268053f6c60d52c17dc63e8035ae528abac61a297ca157efa255d9"} Nov 04 03:16:59.212346 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:16:59.211812 4405 scope.go:122] "RemoveContainer" containerID="e079012dbf9f0a256f342f2823ed63641442c2d25d24122f14a1ce243691052f" Nov 04 03:16:59.379730 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:16:59.379690 4405 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-db5ef5-5d5584b699-rjmpj"] Nov 04 03:16:59.384355 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:16:59.384329 4405 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-db5ef5-5d5584b699-rjmpj"] Nov 04 03:16:59.701597 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:16:59.701553 4405 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3d80d037-3e20-4898-855b-86fbcf2bb9aa" path="/var/lib/kubelet/pods/3d80d037-3e20-4898-855b-86fbcf2bb9aa/volumes" Nov 04 03:17:00.714128 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:17:00.714087 4405 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-76597d6855-vxzxx"] Nov 04 03:17:00.714684 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:17:00.714313 4405 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-vxzxx" podUID="eeec2ff5-38c0-4e67-aeed-9f0baa9fa1d1" containerName="istio-proxy" containerID="cri-o://1bdc0173c3101f7a1fe376035b7e0c6f0f0caceddaf8617db2c2b539c4019cfc" gracePeriod=30 Nov 04 03:17:02.501551 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:17:02.501485 4405 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-mt-4cbcf738-97f795756-gk52w" podUID="a061347f-23bc-43c9-9b1a-594d184ebe7c" containerName="payload-processing" containerID="cri-o://fd96e616ff38ce0f28c88780b448bf1936b3b30cd669c21b733b281f5b0c4e57" gracePeriod=25 Nov 04 03:17:02.612834 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:17:02.612794 4405 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-mt-4cbcf738-67bd6b6f59-59zp6" podUID="9a57520d-dae3-45e9-8558-fa824bd746cd" containerName="payload-pre-processing" containerID="cri-o://7d89adf2a9e729eb6c81da588e035da142f4c56ffd1b03d4d7662f7f945c615f" gracePeriod=25 Nov 04 03:17:02.775394 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:17:02.775367 4405 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-4cbcf738-97f795756-gk52w" Nov 04 03:17:02.858586 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:17:02.858555 4405 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a061347f-23bc-43c9-9b1a-594d184ebe7c-kube-api-access-jdjgc\" (UniqueName: \"kubernetes.io/projected/a061347f-23bc-43c9-9b1a-594d184ebe7c-kube-api-access-jdjgc\") pod \"a061347f-23bc-43c9-9b1a-594d184ebe7c\" (UID: \"a061347f-23bc-43c9-9b1a-594d184ebe7c\") " Nov 04 03:17:02.858848 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:17:02.858825 4405 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a061347f-23bc-43c9-9b1a-594d184ebe7c-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a061347f-23bc-43c9-9b1a-594d184ebe7c-plugins-config-volume\") pod \"a061347f-23bc-43c9-9b1a-594d184ebe7c\" (UID: \"a061347f-23bc-43c9-9b1a-594d184ebe7c\") " Nov 04 03:17:02.859311 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:17:02.859282 4405 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a061347f-23bc-43c9-9b1a-594d184ebe7c-plugins-config-volume" pod "a061347f-23bc-43c9-9b1a-594d184ebe7c" (UID: "a061347f-23bc-43c9-9b1a-594d184ebe7c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 03:17:02.862805 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:17:02.862784 4405 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a061347f-23bc-43c9-9b1a-594d184ebe7c-kube-api-access-jdjgc" pod "a061347f-23bc-43c9-9b1a-594d184ebe7c" (UID: "a061347f-23bc-43c9-9b1a-594d184ebe7c"). InnerVolumeSpecName "kube-api-access-jdjgc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 03:17:02.882406 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:17:02.882381 4405 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-4cbcf738-67bd6b6f59-59zp6" Nov 04 03:17:02.959750 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:17:02.959721 4405 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9a57520d-dae3-45e9-8558-fa824bd746cd-kube-api-access-prhqc\" (UniqueName: \"kubernetes.io/projected/9a57520d-dae3-45e9-8558-fa824bd746cd-kube-api-access-prhqc\") pod \"9a57520d-dae3-45e9-8558-fa824bd746cd\" (UID: \"9a57520d-dae3-45e9-8558-fa824bd746cd\") " Nov 04 03:17:02.959964 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:17:02.959950 4405 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9a57520d-dae3-45e9-8558-fa824bd746cd-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9a57520d-dae3-45e9-8558-fa824bd746cd-plugins-config-volume\") pod \"9a57520d-dae3-45e9-8558-fa824bd746cd\" (UID: \"9a57520d-dae3-45e9-8558-fa824bd746cd\") " Nov 04 03:17:02.960252 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:17:02.960236 4405 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jdjgc\" (UniqueName: \"kubernetes.io/projected/a061347f-23bc-43c9-9b1a-594d184ebe7c-kube-api-access-jdjgc\") on node \"ip-10-0-134-125.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:17:02.960294 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:17:02.960256 4405 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a061347f-23bc-43c9-9b1a-594d184ebe7c-plugins-config-volume\") on node \"ip-10-0-134-125.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:17:02.960294 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:17:02.960266 4405 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9a57520d-dae3-45e9-8558-fa824bd746cd-plugins-config-volume" pod "9a57520d-dae3-45e9-8558-fa824bd746cd" (UID: "9a57520d-dae3-45e9-8558-fa824bd746cd"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 03:17:02.962123 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:17:02.962101 4405 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9a57520d-dae3-45e9-8558-fa824bd746cd-kube-api-access-prhqc" pod "9a57520d-dae3-45e9-8558-fa824bd746cd" (UID: "9a57520d-dae3-45e9-8558-fa824bd746cd"). InnerVolumeSpecName "kube-api-access-prhqc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 03:17:03.061203 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:17:03.061121 4405 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9a57520d-dae3-45e9-8558-fa824bd746cd-plugins-config-volume\") on node \"ip-10-0-134-125.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:17:03.061203 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:17:03.061159 4405 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-prhqc\" (UniqueName: \"kubernetes.io/projected/9a57520d-dae3-45e9-8558-fa824bd746cd-kube-api-access-prhqc\") on node \"ip-10-0-134-125.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:17:03.264954 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:17:03.264920 4405 generic.go:356] "Generic (PLEG): container finished" podID="9a57520d-dae3-45e9-8558-fa824bd746cd" containerID="7d89adf2a9e729eb6c81da588e035da142f4c56ffd1b03d4d7662f7f945c615f" exitCode=0 Nov 04 03:17:03.265140 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:17:03.264974 4405 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-4cbcf738-67bd6b6f59-59zp6" Nov 04 03:17:03.265140 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:17:03.265006 4405 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-4cbcf738-67bd6b6f59-59zp6" event={"ID":"9a57520d-dae3-45e9-8558-fa824bd746cd","Type":"ContainerDied","Data":"7d89adf2a9e729eb6c81da588e035da142f4c56ffd1b03d4d7662f7f945c615f"} Nov 04 03:17:03.265140 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:17:03.265070 4405 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-4cbcf738-67bd6b6f59-59zp6" event={"ID":"9a57520d-dae3-45e9-8558-fa824bd746cd","Type":"ContainerDied","Data":"d7c4f85fe165bd93fcd5e6fa525a5a11c6abe5795eff54d3bd216ca8f6e1f9af"} Nov 04 03:17:03.265140 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:17:03.265099 4405 scope.go:122] "RemoveContainer" containerID="7d89adf2a9e729eb6c81da588e035da142f4c56ffd1b03d4d7662f7f945c615f" Nov 04 03:17:03.266525 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:17:03.266500 4405 generic.go:356] "Generic (PLEG): container finished" podID="a061347f-23bc-43c9-9b1a-594d184ebe7c" containerID="fd96e616ff38ce0f28c88780b448bf1936b3b30cd669c21b733b281f5b0c4e57" exitCode=0 Nov 04 03:17:03.266635 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:17:03.266559 4405 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-4cbcf738-97f795756-gk52w" Nov 04 03:17:03.266635 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:17:03.266581 4405 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-4cbcf738-97f795756-gk52w" event={"ID":"a061347f-23bc-43c9-9b1a-594d184ebe7c","Type":"ContainerDied","Data":"fd96e616ff38ce0f28c88780b448bf1936b3b30cd669c21b733b281f5b0c4e57"} Nov 04 03:17:03.266635 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:17:03.266628 4405 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-4cbcf738-97f795756-gk52w" event={"ID":"a061347f-23bc-43c9-9b1a-594d184ebe7c","Type":"ContainerDied","Data":"cab74d8be4c374064774c501f8f44f6de1be5e4da19b80eead1028b4fd4f59e9"} Nov 04 03:17:03.275245 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:17:03.275223 4405 scope.go:122] "RemoveContainer" containerID="7d89adf2a9e729eb6c81da588e035da142f4c56ffd1b03d4d7662f7f945c615f" Nov 04 03:17:03.275599 ip-10-0-134-125 kubenswrapper[4405]: E1104 03:17:03.275573 4405 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7d89adf2a9e729eb6c81da588e035da142f4c56ffd1b03d4d7662f7f945c615f\": container with ID starting with 7d89adf2a9e729eb6c81da588e035da142f4c56ffd1b03d4d7662f7f945c615f not found: ID does not exist" containerID="7d89adf2a9e729eb6c81da588e035da142f4c56ffd1b03d4d7662f7f945c615f" Nov 04 03:17:03.275843 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:17:03.275611 4405 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7d89adf2a9e729eb6c81da588e035da142f4c56ffd1b03d4d7662f7f945c615f"} err="failed to get container status \"7d89adf2a9e729eb6c81da588e035da142f4c56ffd1b03d4d7662f7f945c615f\": rpc error: code = NotFound desc = could not find container \"7d89adf2a9e729eb6c81da588e035da142f4c56ffd1b03d4d7662f7f945c615f\": container with ID starting with 7d89adf2a9e729eb6c81da588e035da142f4c56ffd1b03d4d7662f7f945c615f not found: ID does not exist" Nov 04 03:17:03.275843 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:17:03.275635 4405 scope.go:122] "RemoveContainer" containerID="fd96e616ff38ce0f28c88780b448bf1936b3b30cd669c21b733b281f5b0c4e57" Nov 04 03:17:03.285121 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:17:03.285104 4405 scope.go:122] "RemoveContainer" containerID="fd96e616ff38ce0f28c88780b448bf1936b3b30cd669c21b733b281f5b0c4e57" Nov 04 03:17:03.285391 ip-10-0-134-125 kubenswrapper[4405]: E1104 03:17:03.285373 4405 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fd96e616ff38ce0f28c88780b448bf1936b3b30cd669c21b733b281f5b0c4e57\": container with ID starting with fd96e616ff38ce0f28c88780b448bf1936b3b30cd669c21b733b281f5b0c4e57 not found: ID does not exist" containerID="fd96e616ff38ce0f28c88780b448bf1936b3b30cd669c21b733b281f5b0c4e57" Nov 04 03:17:03.285441 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:17:03.285402 4405 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fd96e616ff38ce0f28c88780b448bf1936b3b30cd669c21b733b281f5b0c4e57"} err="failed to get container status \"fd96e616ff38ce0f28c88780b448bf1936b3b30cd669c21b733b281f5b0c4e57\": rpc error: code = NotFound desc = could not find container \"fd96e616ff38ce0f28c88780b448bf1936b3b30cd669c21b733b281f5b0c4e57\": container with ID starting with fd96e616ff38ce0f28c88780b448bf1936b3b30cd669c21b733b281f5b0c4e57 not found: ID does not exist" Nov 04 03:17:03.428384 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:17:03.428353 4405 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-4cbcf738-97f795756-gk52w"] Nov 04 03:17:03.434708 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:17:03.434654 4405 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-4cbcf738-97f795756-gk52w"] Nov 04 03:17:03.543411 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:17:03.543373 4405 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-4cbcf738-67bd6b6f59-59zp6"] Nov 04 03:17:03.548050 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:17:03.548027 4405 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-4cbcf738-67bd6b6f59-59zp6"] Nov 04 03:17:03.701644 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:17:03.701564 4405 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9a57520d-dae3-45e9-8558-fa824bd746cd" path="/var/lib/kubelet/pods/9a57520d-dae3-45e9-8558-fa824bd746cd/volumes" Nov 04 03:17:03.701987 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:17:03.701972 4405 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a061347f-23bc-43c9-9b1a-594d184ebe7c" path="/var/lib/kubelet/pods/a061347f-23bc-43c9-9b1a-594d184ebe7c/volumes" Nov 04 03:17:05.993091 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:17:05.993027 4405 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-vxzxx" Nov 04 03:17:06.095741 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:17:06.095705 4405 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/eeec2ff5-38c0-4e67-aeed-9f0baa9fa1d1-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/eeec2ff5-38c0-4e67-aeed-9f0baa9fa1d1-credential-socket\") pod \"eeec2ff5-38c0-4e67-aeed-9f0baa9fa1d1\" (UID: \"eeec2ff5-38c0-4e67-aeed-9f0baa9fa1d1\") " Nov 04 03:17:06.096046 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:17:06.095787 4405 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/eeec2ff5-38c0-4e67-aeed-9f0baa9fa1d1-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/eeec2ff5-38c0-4e67-aeed-9f0baa9fa1d1-istio-podinfo\") pod \"eeec2ff5-38c0-4e67-aeed-9f0baa9fa1d1\" (UID: \"eeec2ff5-38c0-4e67-aeed-9f0baa9fa1d1\") " Nov 04 03:17:06.096046 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:17:06.095821 4405 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/eeec2ff5-38c0-4e67-aeed-9f0baa9fa1d1-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/eeec2ff5-38c0-4e67-aeed-9f0baa9fa1d1-istio-envoy\") pod \"eeec2ff5-38c0-4e67-aeed-9f0baa9fa1d1\" (UID: \"eeec2ff5-38c0-4e67-aeed-9f0baa9fa1d1\") " Nov 04 03:17:06.096046 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:17:06.095850 4405 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/eeec2ff5-38c0-4e67-aeed-9f0baa9fa1d1-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/eeec2ff5-38c0-4e67-aeed-9f0baa9fa1d1-istiod-ca-cert\") pod \"eeec2ff5-38c0-4e67-aeed-9f0baa9fa1d1\" (UID: \"eeec2ff5-38c0-4e67-aeed-9f0baa9fa1d1\") " Nov 04 03:17:06.096046 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:17:06.095969 4405 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/eeec2ff5-38c0-4e67-aeed-9f0baa9fa1d1-istio-token\" (UniqueName: \"kubernetes.io/projected/eeec2ff5-38c0-4e67-aeed-9f0baa9fa1d1-istio-token\") pod \"eeec2ff5-38c0-4e67-aeed-9f0baa9fa1d1\" (UID: \"eeec2ff5-38c0-4e67-aeed-9f0baa9fa1d1\") " Nov 04 03:17:06.096046 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:17:06.096021 4405 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eeec2ff5-38c0-4e67-aeed-9f0baa9fa1d1-credential-socket" pod "eeec2ff5-38c0-4e67-aeed-9f0baa9fa1d1" (UID: "eeec2ff5-38c0-4e67-aeed-9f0baa9fa1d1"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 03:17:06.096046 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:17:06.096036 4405 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/eeec2ff5-38c0-4e67-aeed-9f0baa9fa1d1-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/eeec2ff5-38c0-4e67-aeed-9f0baa9fa1d1-workload-certs\") pod \"eeec2ff5-38c0-4e67-aeed-9f0baa9fa1d1\" (UID: \"eeec2ff5-38c0-4e67-aeed-9f0baa9fa1d1\") " Nov 04 03:17:06.096413 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:17:06.096082 4405 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/eeec2ff5-38c0-4e67-aeed-9f0baa9fa1d1-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/eeec2ff5-38c0-4e67-aeed-9f0baa9fa1d1-istio-data\") pod \"eeec2ff5-38c0-4e67-aeed-9f0baa9fa1d1\" (UID: \"eeec2ff5-38c0-4e67-aeed-9f0baa9fa1d1\") " Nov 04 03:17:06.096413 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:17:06.096145 4405 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/eeec2ff5-38c0-4e67-aeed-9f0baa9fa1d1-kube-api-access-cjnvx\" (UniqueName: \"kubernetes.io/projected/eeec2ff5-38c0-4e67-aeed-9f0baa9fa1d1-kube-api-access-cjnvx\") pod \"eeec2ff5-38c0-4e67-aeed-9f0baa9fa1d1\" (UID: \"eeec2ff5-38c0-4e67-aeed-9f0baa9fa1d1\") " Nov 04 03:17:06.096413 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:17:06.096205 4405 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/eeec2ff5-38c0-4e67-aeed-9f0baa9fa1d1-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/eeec2ff5-38c0-4e67-aeed-9f0baa9fa1d1-workload-socket\") pod \"eeec2ff5-38c0-4e67-aeed-9f0baa9fa1d1\" (UID: \"eeec2ff5-38c0-4e67-aeed-9f0baa9fa1d1\") " Nov 04 03:17:06.096413 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:17:06.096305 4405 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/eeec2ff5-38c0-4e67-aeed-9f0baa9fa1d1-istiod-ca-cert" pod "eeec2ff5-38c0-4e67-aeed-9f0baa9fa1d1" (UID: "eeec2ff5-38c0-4e67-aeed-9f0baa9fa1d1"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 03:17:06.096413 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:17:06.096311 4405 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eeec2ff5-38c0-4e67-aeed-9f0baa9fa1d1-workload-certs" pod "eeec2ff5-38c0-4e67-aeed-9f0baa9fa1d1" (UID: "eeec2ff5-38c0-4e67-aeed-9f0baa9fa1d1"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 03:17:06.096733 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:17:06.096570 4405 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eeec2ff5-38c0-4e67-aeed-9f0baa9fa1d1-workload-socket" pod "eeec2ff5-38c0-4e67-aeed-9f0baa9fa1d1" (UID: "eeec2ff5-38c0-4e67-aeed-9f0baa9fa1d1"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 03:17:06.096795 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:17:06.096702 4405 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/eeec2ff5-38c0-4e67-aeed-9f0baa9fa1d1-workload-socket\") on node \"ip-10-0-134-125.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:17:06.096947 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:17:06.096915 4405 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/eeec2ff5-38c0-4e67-aeed-9f0baa9fa1d1-credential-socket\") on node \"ip-10-0-134-125.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:17:06.096947 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:17:06.096935 4405 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/eeec2ff5-38c0-4e67-aeed-9f0baa9fa1d1-istiod-ca-cert\") on node \"ip-10-0-134-125.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:17:06.096947 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:17:06.096950 4405 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/eeec2ff5-38c0-4e67-aeed-9f0baa9fa1d1-workload-certs\") on node \"ip-10-0-134-125.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:17:06.097183 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:17:06.096826 4405 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eeec2ff5-38c0-4e67-aeed-9f0baa9fa1d1-istio-data" pod "eeec2ff5-38c0-4e67-aeed-9f0baa9fa1d1" (UID: "eeec2ff5-38c0-4e67-aeed-9f0baa9fa1d1"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 03:17:06.098387 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:17:06.098365 4405 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/eeec2ff5-38c0-4e67-aeed-9f0baa9fa1d1-istio-podinfo" pod "eeec2ff5-38c0-4e67-aeed-9f0baa9fa1d1" (UID: "eeec2ff5-38c0-4e67-aeed-9f0baa9fa1d1"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 04 03:17:06.098584 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:17:06.098560 4405 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eeec2ff5-38c0-4e67-aeed-9f0baa9fa1d1-kube-api-access-cjnvx" pod "eeec2ff5-38c0-4e67-aeed-9f0baa9fa1d1" (UID: "eeec2ff5-38c0-4e67-aeed-9f0baa9fa1d1"). InnerVolumeSpecName "kube-api-access-cjnvx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 03:17:06.098727 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:17:06.098687 4405 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eeec2ff5-38c0-4e67-aeed-9f0baa9fa1d1-istio-envoy" pod "eeec2ff5-38c0-4e67-aeed-9f0baa9fa1d1" (UID: "eeec2ff5-38c0-4e67-aeed-9f0baa9fa1d1"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 03:17:06.098783 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:17:06.098738 4405 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eeec2ff5-38c0-4e67-aeed-9f0baa9fa1d1-istio-token" pod "eeec2ff5-38c0-4e67-aeed-9f0baa9fa1d1" (UID: "eeec2ff5-38c0-4e67-aeed-9f0baa9fa1d1"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 03:17:06.198278 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:17:06.198229 4405 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/eeec2ff5-38c0-4e67-aeed-9f0baa9fa1d1-istio-podinfo\") on node \"ip-10-0-134-125.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:17:06.198278 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:17:06.198265 4405 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/eeec2ff5-38c0-4e67-aeed-9f0baa9fa1d1-istio-envoy\") on node \"ip-10-0-134-125.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:17:06.198278 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:17:06.198275 4405 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/eeec2ff5-38c0-4e67-aeed-9f0baa9fa1d1-istio-token\") on node \"ip-10-0-134-125.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:17:06.198278 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:17:06.198284 4405 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/eeec2ff5-38c0-4e67-aeed-9f0baa9fa1d1-istio-data\") on node \"ip-10-0-134-125.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:17:06.198711 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:17:06.198293 4405 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cjnvx\" (UniqueName: \"kubernetes.io/projected/eeec2ff5-38c0-4e67-aeed-9f0baa9fa1d1-kube-api-access-cjnvx\") on node \"ip-10-0-134-125.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:17:06.286596 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:17:06.286501 4405 generic.go:356] "Generic (PLEG): container finished" podID="eeec2ff5-38c0-4e67-aeed-9f0baa9fa1d1" containerID="1bdc0173c3101f7a1fe376035b7e0c6f0f0caceddaf8617db2c2b539c4019cfc" exitCode=0 Nov 04 03:17:06.286596 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:17:06.286551 4405 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-vxzxx" event={"ID":"eeec2ff5-38c0-4e67-aeed-9f0baa9fa1d1","Type":"ContainerDied","Data":"1bdc0173c3101f7a1fe376035b7e0c6f0f0caceddaf8617db2c2b539c4019cfc"} Nov 04 03:17:06.286596 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:17:06.286570 4405 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-vxzxx" Nov 04 03:17:06.286892 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:17:06.286612 4405 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-vxzxx" event={"ID":"eeec2ff5-38c0-4e67-aeed-9f0baa9fa1d1","Type":"ContainerDied","Data":"8075b1c999bf29c05d87e2d09c8ec401fc9bc6518cc5a89a82f2cc9bd5d31eed"} Nov 04 03:17:06.286892 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:17:06.286639 4405 scope.go:122] "RemoveContainer" containerID="1bdc0173c3101f7a1fe376035b7e0c6f0f0caceddaf8617db2c2b539c4019cfc" Nov 04 03:17:06.297925 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:17:06.297908 4405 scope.go:122] "RemoveContainer" containerID="1bdc0173c3101f7a1fe376035b7e0c6f0f0caceddaf8617db2c2b539c4019cfc" Nov 04 03:17:06.298222 ip-10-0-134-125 kubenswrapper[4405]: E1104 03:17:06.298202 4405 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1bdc0173c3101f7a1fe376035b7e0c6f0f0caceddaf8617db2c2b539c4019cfc\": container with ID starting with 1bdc0173c3101f7a1fe376035b7e0c6f0f0caceddaf8617db2c2b539c4019cfc not found: ID does not exist" containerID="1bdc0173c3101f7a1fe376035b7e0c6f0f0caceddaf8617db2c2b539c4019cfc" Nov 04 03:17:06.298263 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:17:06.298233 4405 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1bdc0173c3101f7a1fe376035b7e0c6f0f0caceddaf8617db2c2b539c4019cfc"} err="failed to get container status \"1bdc0173c3101f7a1fe376035b7e0c6f0f0caceddaf8617db2c2b539c4019cfc\": rpc error: code = NotFound desc = could not find container \"1bdc0173c3101f7a1fe376035b7e0c6f0f0caceddaf8617db2c2b539c4019cfc\": container with ID starting with 1bdc0173c3101f7a1fe376035b7e0c6f0f0caceddaf8617db2c2b539c4019cfc not found: ID does not exist" Nov 04 03:17:06.451833 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:17:06.451797 4405 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-76597d6855-vxzxx"] Nov 04 03:17:06.456307 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:17:06.456280 4405 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-76597d6855-vxzxx"] Nov 04 03:17:06.884415 ip-10-0-134-125 kubenswrapper[4405]: E1104 03:17:06.884381 4405 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5f2vb" containerName="extractor" Nov 04 03:17:07.701429 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:17:07.701382 4405 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="eeec2ff5-38c0-4e67-aeed-9f0baa9fa1d1" path="/var/lib/kubelet/pods/eeec2ff5-38c0-4e67-aeed-9f0baa9fa1d1/volumes" Nov 04 03:17:10.952419 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:17:10.952371 4405 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-bl5mg"] Nov 04 03:17:12.088110 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:17:12.088074 4405 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-095081-6c6bf65b7b-rdq44"] Nov 04 03:17:12.199713 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:17:12.199678 4405 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-095081-bd58cc787-xz5kc"] Nov 04 03:17:15.088791 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:17:15.088754 4405 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-4cbcf738-7b5dfbb998-d5bb8"] Nov 04 03:17:15.091463 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:17:15.089559 4405 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-mt-4cbcf738-7b5dfbb998-d5bb8" podUID="73c5ca67-bc51-492d-b055-0cbaf3a85b07" containerName="maas-api" containerID="cri-o://e8d1a5666eb7317340bf5fce328af705481496dfdeffbcb56490374b3b7e1181" gracePeriod=30 Nov 04 03:17:15.343975 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:17:15.343938 4405 generic.go:356] "Generic (PLEG): container finished" podID="73c5ca67-bc51-492d-b055-0cbaf3a85b07" containerID="e8d1a5666eb7317340bf5fce328af705481496dfdeffbcb56490374b3b7e1181" exitCode=0 Nov 04 03:17:15.344156 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:17:15.344014 4405 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-4cbcf738-7b5dfbb998-d5bb8" event={"ID":"73c5ca67-bc51-492d-b055-0cbaf3a85b07","Type":"ContainerDied","Data":"e8d1a5666eb7317340bf5fce328af705481496dfdeffbcb56490374b3b7e1181"} Nov 04 03:17:15.344156 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:17:15.344095 4405 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-4cbcf738-7b5dfbb998-d5bb8" event={"ID":"73c5ca67-bc51-492d-b055-0cbaf3a85b07","Type":"ContainerDied","Data":"350c684d614d271b303a4dedd03348238e88c0d56f047c2a82d39d38f97d5087"} Nov 04 03:17:15.344156 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:17:15.344122 4405 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="350c684d614d271b303a4dedd03348238e88c0d56f047c2a82d39d38f97d5087" Nov 04 03:17:15.359998 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:17:15.359970 4405 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-4cbcf738-7b5dfbb998-d5bb8" Nov 04 03:17:15.505982 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:17:15.505942 4405 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/73c5ca67-bc51-492d-b055-0cbaf3a85b07-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/73c5ca67-bc51-492d-b055-0cbaf3a85b07-openshift-service-ca\") pod \"73c5ca67-bc51-492d-b055-0cbaf3a85b07\" (UID: \"73c5ca67-bc51-492d-b055-0cbaf3a85b07\") " Nov 04 03:17:15.506180 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:17:15.505999 4405 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/73c5ca67-bc51-492d-b055-0cbaf3a85b07-kube-api-access-5vgz7\" (UniqueName: \"kubernetes.io/projected/73c5ca67-bc51-492d-b055-0cbaf3a85b07-kube-api-access-5vgz7\") pod \"73c5ca67-bc51-492d-b055-0cbaf3a85b07\" (UID: \"73c5ca67-bc51-492d-b055-0cbaf3a85b07\") " Nov 04 03:17:15.506180 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:17:15.506095 4405 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/73c5ca67-bc51-492d-b055-0cbaf3a85b07-metrics-certs\" (UniqueName: \"kubernetes.io/secret/73c5ca67-bc51-492d-b055-0cbaf3a85b07-metrics-certs\") pod \"73c5ca67-bc51-492d-b055-0cbaf3a85b07\" (UID: \"73c5ca67-bc51-492d-b055-0cbaf3a85b07\") " Nov 04 03:17:15.506313 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:17:15.506215 4405 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/73c5ca67-bc51-492d-b055-0cbaf3a85b07-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/73c5ca67-bc51-492d-b055-0cbaf3a85b07-maas-api-tls\") pod \"73c5ca67-bc51-492d-b055-0cbaf3a85b07\" (UID: \"73c5ca67-bc51-492d-b055-0cbaf3a85b07\") " Nov 04 03:17:15.506373 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:17:15.506352 4405 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/73c5ca67-bc51-492d-b055-0cbaf3a85b07-openshift-service-ca" pod "73c5ca67-bc51-492d-b055-0cbaf3a85b07" (UID: "73c5ca67-bc51-492d-b055-0cbaf3a85b07"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 03:17:15.506638 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:17:15.506616 4405 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/73c5ca67-bc51-492d-b055-0cbaf3a85b07-openshift-service-ca\") on node \"ip-10-0-134-125.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:17:15.508518 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:17:15.508486 4405 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/73c5ca67-bc51-492d-b055-0cbaf3a85b07-maas-api-tls" pod "73c5ca67-bc51-492d-b055-0cbaf3a85b07" (UID: "73c5ca67-bc51-492d-b055-0cbaf3a85b07"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 04 03:17:15.508518 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:17:15.508495 4405 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/73c5ca67-bc51-492d-b055-0cbaf3a85b07-kube-api-access-5vgz7" pod "73c5ca67-bc51-492d-b055-0cbaf3a85b07" (UID: "73c5ca67-bc51-492d-b055-0cbaf3a85b07"). InnerVolumeSpecName "kube-api-access-5vgz7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 03:17:15.508813 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:17:15.508793 4405 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/73c5ca67-bc51-492d-b055-0cbaf3a85b07-metrics-certs" pod "73c5ca67-bc51-492d-b055-0cbaf3a85b07" (UID: "73c5ca67-bc51-492d-b055-0cbaf3a85b07"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 04 03:17:15.608349 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:17:15.608256 4405 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5vgz7\" (UniqueName: \"kubernetes.io/projected/73c5ca67-bc51-492d-b055-0cbaf3a85b07-kube-api-access-5vgz7\") on node \"ip-10-0-134-125.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:17:15.608349 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:17:15.608299 4405 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/73c5ca67-bc51-492d-b055-0cbaf3a85b07-metrics-certs\") on node \"ip-10-0-134-125.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:17:15.608349 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:17:15.608317 4405 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/73c5ca67-bc51-492d-b055-0cbaf3a85b07-maas-api-tls\") on node \"ip-10-0-134-125.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:17:16.348971 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:17:16.348936 4405 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-4cbcf738-7b5dfbb998-d5bb8" Nov 04 03:17:16.513344 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:17:16.513300 4405 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-4cbcf738-7b5dfbb998-d5bb8"] Nov 04 03:17:16.563343 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:17:16.563300 4405 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-4cbcf738-7b5dfbb998-d5bb8"] Nov 04 03:17:17.089142 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:17:17.089106 4405 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-b-095081-6c6bf65b7b-rdq44" podUID="a6f009b4-0c0b-4652-812f-a7d3333f3880" containerName="payload-processing" containerID="cri-o://a41597d148d56db195437cd58783bce0bf781c5694672f6f5389d6ac4bf1b751" gracePeriod=25 Nov 04 03:17:17.200798 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:17:17.200714 4405 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-095081-bd58cc787-xz5kc" podUID="3eb56d77-29c0-4a72-863d-354d2cb764a0" containerName="payload-pre-processing" containerID="cri-o://661b7e606486e657909844b5e698bca6cd782df0cfdfc06b6b55264c70e08797" gracePeriod=25 Nov 04 03:17:17.352781 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:17:17.352711 4405 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-095081-6c6bf65b7b-rdq44" Nov 04 03:17:17.355699 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:17:17.355631 4405 generic.go:356] "Generic (PLEG): container finished" podID="a6f009b4-0c0b-4652-812f-a7d3333f3880" containerID="a41597d148d56db195437cd58783bce0bf781c5694672f6f5389d6ac4bf1b751" exitCode=0 Nov 04 03:17:17.355699 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:17:17.355691 4405 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-095081-6c6bf65b7b-rdq44" event={"ID":"a6f009b4-0c0b-4652-812f-a7d3333f3880","Type":"ContainerDied","Data":"a41597d148d56db195437cd58783bce0bf781c5694672f6f5389d6ac4bf1b751"} Nov 04 03:17:17.355699 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:17:17.355697 4405 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-095081-6c6bf65b7b-rdq44" Nov 04 03:17:17.355884 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:17:17.355745 4405 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-095081-6c6bf65b7b-rdq44" event={"ID":"a6f009b4-0c0b-4652-812f-a7d3333f3880","Type":"ContainerDied","Data":"4065c260e2f0a317e6667660d1bb84218fcf338289171feeb8d79b1a47b9ef91"} Nov 04 03:17:17.355884 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:17:17.355774 4405 scope.go:122] "RemoveContainer" containerID="a41597d148d56db195437cd58783bce0bf781c5694672f6f5389d6ac4bf1b751" Nov 04 03:17:17.367796 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:17:17.367774 4405 scope.go:122] "RemoveContainer" containerID="a41597d148d56db195437cd58783bce0bf781c5694672f6f5389d6ac4bf1b751" Nov 04 03:17:17.368099 ip-10-0-134-125 kubenswrapper[4405]: E1104 03:17:17.368075 4405 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a41597d148d56db195437cd58783bce0bf781c5694672f6f5389d6ac4bf1b751\": container with ID starting with a41597d148d56db195437cd58783bce0bf781c5694672f6f5389d6ac4bf1b751 not found: ID does not exist" containerID="a41597d148d56db195437cd58783bce0bf781c5694672f6f5389d6ac4bf1b751" Nov 04 03:17:17.368182 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:17:17.368108 4405 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a41597d148d56db195437cd58783bce0bf781c5694672f6f5389d6ac4bf1b751"} err="failed to get container status \"a41597d148d56db195437cd58783bce0bf781c5694672f6f5389d6ac4bf1b751\": rpc error: code = NotFound desc = could not find container \"a41597d148d56db195437cd58783bce0bf781c5694672f6f5389d6ac4bf1b751\": container with ID starting with a41597d148d56db195437cd58783bce0bf781c5694672f6f5389d6ac4bf1b751 not found: ID does not exist" Nov 04 03:17:17.530645 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:17:17.530614 4405 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a6f009b4-0c0b-4652-812f-a7d3333f3880-kube-api-access-qp4qw\" (UniqueName: \"kubernetes.io/projected/a6f009b4-0c0b-4652-812f-a7d3333f3880-kube-api-access-qp4qw\") pod \"a6f009b4-0c0b-4652-812f-a7d3333f3880\" (UID: \"a6f009b4-0c0b-4652-812f-a7d3333f3880\") " Nov 04 03:17:17.530821 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:17:17.530746 4405 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a6f009b4-0c0b-4652-812f-a7d3333f3880-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a6f009b4-0c0b-4652-812f-a7d3333f3880-plugins-config-volume\") pod \"a6f009b4-0c0b-4652-812f-a7d3333f3880\" (UID: \"a6f009b4-0c0b-4652-812f-a7d3333f3880\") " Nov 04 03:17:17.531151 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:17:17.531127 4405 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a6f009b4-0c0b-4652-812f-a7d3333f3880-plugins-config-volume" pod "a6f009b4-0c0b-4652-812f-a7d3333f3880" (UID: "a6f009b4-0c0b-4652-812f-a7d3333f3880"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 03:17:17.533052 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:17:17.533025 4405 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a6f009b4-0c0b-4652-812f-a7d3333f3880-kube-api-access-qp4qw" pod "a6f009b4-0c0b-4652-812f-a7d3333f3880" (UID: "a6f009b4-0c0b-4652-812f-a7d3333f3880"). InnerVolumeSpecName "kube-api-access-qp4qw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 03:17:17.631795 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:17:17.631704 4405 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qp4qw\" (UniqueName: \"kubernetes.io/projected/a6f009b4-0c0b-4652-812f-a7d3333f3880-kube-api-access-qp4qw\") on node \"ip-10-0-134-125.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:17:17.631795 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:17:17.631737 4405 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a6f009b4-0c0b-4652-812f-a7d3333f3880-plugins-config-volume\") on node \"ip-10-0-134-125.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:17:17.701352 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:17:17.701319 4405 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="73c5ca67-bc51-492d-b055-0cbaf3a85b07" path="/var/lib/kubelet/pods/73c5ca67-bc51-492d-b055-0cbaf3a85b07/volumes" Nov 04 03:17:17.776755 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:17:17.776722 4405 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-095081-6c6bf65b7b-rdq44"] Nov 04 03:17:17.780514 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:17:17.780484 4405 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-095081-6c6bf65b7b-rdq44"] Nov 04 03:17:19.699955 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:17:19.699919 4405 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a6f009b4-0c0b-4652-812f-a7d3333f3880" path="/var/lib/kubelet/pods/a6f009b4-0c0b-4652-812f-a7d3333f3880/volumes" Nov 04 03:17:24.053661 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:17:24.053619 4405 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-095081-bd58cc787-xz5kc" podUID="3eb56d77-29c0-4a72-863d-354d2cb764a0" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 04 03:17:27.031859 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:17:27.031824 4405 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-4cbcf738-openshift-default-68fb497c9c-2ndqr"] Nov 04 03:17:27.032376 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:17:27.032041 4405 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-4cbcf738-openshift-default-68fb497c9c-2ndqr" podUID="a77157ba-0aed-46cf-90de-ffc15074a50f" containerName="istio-proxy" containerID="cri-o://380cfa2a332239970422898ec218bee16d39f4fc2106c20638fa7e0982c35c0e" gracePeriod=30 Nov 04 03:17:30.946848 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:17:30.946816 4405 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-76597d6855-brntt"] Nov 04 03:17:30.947320 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:17:30.947005 4405 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-brntt" podUID="423b62dd-792f-442c-b1ac-3248072565e8" containerName="istio-proxy" containerID="cri-o://e24a32ba8e02e59a6d4901acdaf8b888fd3d35821730e056c2e4aec27ad72fe5" gracePeriod=30 Nov 04 03:17:31.725518 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:17:31.725476 4405 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76597d6855-brntt container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 04 03:17:31.725744 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:17:31.725572 4405 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-brntt" podUID="423b62dd-792f-442c-b1ac-3248072565e8" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 04 03:17:32.312113 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:17:32.312050 4405 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-4cbcf738-openshift-default-68fb497c9c-2ndqr" Nov 04 03:17:32.393642 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:17:32.393603 4405 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a77157ba-0aed-46cf-90de-ffc15074a50f-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a77157ba-0aed-46cf-90de-ffc15074a50f-istiod-ca-cert\") pod \"a77157ba-0aed-46cf-90de-ffc15074a50f\" (UID: \"a77157ba-0aed-46cf-90de-ffc15074a50f\") " Nov 04 03:17:32.393838 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:17:32.393753 4405 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/a77157ba-0aed-46cf-90de-ffc15074a50f-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a77157ba-0aed-46cf-90de-ffc15074a50f-istio-podinfo\") pod \"a77157ba-0aed-46cf-90de-ffc15074a50f\" (UID: \"a77157ba-0aed-46cf-90de-ffc15074a50f\") " Nov 04 03:17:32.393838 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:17:32.393781 4405 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a77157ba-0aed-46cf-90de-ffc15074a50f-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a77157ba-0aed-46cf-90de-ffc15074a50f-istio-data\") pod \"a77157ba-0aed-46cf-90de-ffc15074a50f\" (UID: \"a77157ba-0aed-46cf-90de-ffc15074a50f\") " Nov 04 03:17:32.393838 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:17:32.393815 4405 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a77157ba-0aed-46cf-90de-ffc15074a50f-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a77157ba-0aed-46cf-90de-ffc15074a50f-workload-socket\") pod \"a77157ba-0aed-46cf-90de-ffc15074a50f\" (UID: \"a77157ba-0aed-46cf-90de-ffc15074a50f\") " Nov 04 03:17:32.394003 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:17:32.393841 4405 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a77157ba-0aed-46cf-90de-ffc15074a50f-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a77157ba-0aed-46cf-90de-ffc15074a50f-credential-socket\") pod \"a77157ba-0aed-46cf-90de-ffc15074a50f\" (UID: \"a77157ba-0aed-46cf-90de-ffc15074a50f\") " Nov 04 03:17:32.394003 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:17:32.393874 4405 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a77157ba-0aed-46cf-90de-ffc15074a50f-istio-token\" (UniqueName: \"kubernetes.io/projected/a77157ba-0aed-46cf-90de-ffc15074a50f-istio-token\") pod \"a77157ba-0aed-46cf-90de-ffc15074a50f\" (UID: \"a77157ba-0aed-46cf-90de-ffc15074a50f\") " Nov 04 03:17:32.394003 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:17:32.393909 4405 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a77157ba-0aed-46cf-90de-ffc15074a50f-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a77157ba-0aed-46cf-90de-ffc15074a50f-workload-certs\") pod \"a77157ba-0aed-46cf-90de-ffc15074a50f\" (UID: \"a77157ba-0aed-46cf-90de-ffc15074a50f\") " Nov 04 03:17:32.394003 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:17:32.393977 4405 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a77157ba-0aed-46cf-90de-ffc15074a50f-kube-api-access-xxs4z\" (UniqueName: \"kubernetes.io/projected/a77157ba-0aed-46cf-90de-ffc15074a50f-kube-api-access-xxs4z\") pod \"a77157ba-0aed-46cf-90de-ffc15074a50f\" (UID: \"a77157ba-0aed-46cf-90de-ffc15074a50f\") " Nov 04 03:17:32.394210 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:17:32.394011 4405 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a77157ba-0aed-46cf-90de-ffc15074a50f-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a77157ba-0aed-46cf-90de-ffc15074a50f-istio-envoy\") pod \"a77157ba-0aed-46cf-90de-ffc15074a50f\" (UID: \"a77157ba-0aed-46cf-90de-ffc15074a50f\") " Nov 04 03:17:32.394210 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:17:32.394101 4405 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a77157ba-0aed-46cf-90de-ffc15074a50f-istiod-ca-cert" pod "a77157ba-0aed-46cf-90de-ffc15074a50f" (UID: "a77157ba-0aed-46cf-90de-ffc15074a50f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 03:17:32.394469 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:17:32.394336 4405 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a77157ba-0aed-46cf-90de-ffc15074a50f-credential-socket" pod "a77157ba-0aed-46cf-90de-ffc15074a50f" (UID: "a77157ba-0aed-46cf-90de-ffc15074a50f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 03:17:32.394469 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:17:32.394395 4405 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a77157ba-0aed-46cf-90de-ffc15074a50f-workload-certs" pod "a77157ba-0aed-46cf-90de-ffc15074a50f" (UID: "a77157ba-0aed-46cf-90de-ffc15074a50f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 03:17:32.394469 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:17:32.394403 4405 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a77157ba-0aed-46cf-90de-ffc15074a50f-workload-socket" pod "a77157ba-0aed-46cf-90de-ffc15074a50f" (UID: "a77157ba-0aed-46cf-90de-ffc15074a50f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 03:17:32.394469 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:17:32.394428 4405 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a77157ba-0aed-46cf-90de-ffc15074a50f-istio-data" pod "a77157ba-0aed-46cf-90de-ffc15074a50f" (UID: "a77157ba-0aed-46cf-90de-ffc15074a50f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 03:17:32.394857 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:17:32.394518 4405 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a77157ba-0aed-46cf-90de-ffc15074a50f-credential-socket\") on node \"ip-10-0-134-125.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:17:32.394857 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:17:32.394544 4405 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a77157ba-0aed-46cf-90de-ffc15074a50f-workload-certs\") on node \"ip-10-0-134-125.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:17:32.394857 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:17:32.394558 4405 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a77157ba-0aed-46cf-90de-ffc15074a50f-istiod-ca-cert\") on node \"ip-10-0-134-125.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:17:32.396872 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:17:32.396842 4405 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a77157ba-0aed-46cf-90de-ffc15074a50f-istio-token" pod "a77157ba-0aed-46cf-90de-ffc15074a50f" (UID: "a77157ba-0aed-46cf-90de-ffc15074a50f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 03:17:32.397261 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:17:32.397236 4405 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a77157ba-0aed-46cf-90de-ffc15074a50f-kube-api-access-xxs4z" pod "a77157ba-0aed-46cf-90de-ffc15074a50f" (UID: "a77157ba-0aed-46cf-90de-ffc15074a50f"). InnerVolumeSpecName "kube-api-access-xxs4z". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 03:17:32.397515 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:17:32.397487 4405 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a77157ba-0aed-46cf-90de-ffc15074a50f-istio-envoy" pod "a77157ba-0aed-46cf-90de-ffc15074a50f" (UID: "a77157ba-0aed-46cf-90de-ffc15074a50f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 03:17:32.397623 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:17:32.397571 4405 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a77157ba-0aed-46cf-90de-ffc15074a50f-istio-podinfo" pod "a77157ba-0aed-46cf-90de-ffc15074a50f" (UID: "a77157ba-0aed-46cf-90de-ffc15074a50f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 04 03:17:32.445197 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:17:32.445161 4405 generic.go:356] "Generic (PLEG): container finished" podID="a77157ba-0aed-46cf-90de-ffc15074a50f" containerID="380cfa2a332239970422898ec218bee16d39f4fc2106c20638fa7e0982c35c0e" exitCode=0 Nov 04 03:17:32.445532 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:17:32.445295 4405 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-4cbcf738-openshift-default-68fb497c9c-2ndqr" event={"ID":"a77157ba-0aed-46cf-90de-ffc15074a50f","Type":"ContainerDied","Data":"380cfa2a332239970422898ec218bee16d39f4fc2106c20638fa7e0982c35c0e"} Nov 04 03:17:32.445532 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:17:32.445346 4405 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-4cbcf738-openshift-default-68fb497c9c-2ndqr" event={"ID":"a77157ba-0aed-46cf-90de-ffc15074a50f","Type":"ContainerDied","Data":"e57f052b1c1e308e3db04076ce75ab0841ad6d56803824fa0ad5b0ee0a050b4b"} Nov 04 03:17:32.445532 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:17:32.445378 4405 scope.go:122] "RemoveContainer" containerID="380cfa2a332239970422898ec218bee16d39f4fc2106c20638fa7e0982c35c0e" Nov 04 03:17:32.445785 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:17:32.445544 4405 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-4cbcf738-openshift-default-68fb497c9c-2ndqr" Nov 04 03:17:32.461187 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:17:32.461159 4405 scope.go:122] "RemoveContainer" containerID="81aa272b24db3e2aac8307e85ec1c538754b7333cddcdf89cdfbbdaaf0382d0e" Nov 04 03:17:32.470188 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:17:32.470167 4405 scope.go:122] "RemoveContainer" containerID="380cfa2a332239970422898ec218bee16d39f4fc2106c20638fa7e0982c35c0e" Nov 04 03:17:32.470488 ip-10-0-134-125 kubenswrapper[4405]: E1104 03:17:32.470464 4405 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"380cfa2a332239970422898ec218bee16d39f4fc2106c20638fa7e0982c35c0e\": container with ID starting with 380cfa2a332239970422898ec218bee16d39f4fc2106c20638fa7e0982c35c0e not found: ID does not exist" containerID="380cfa2a332239970422898ec218bee16d39f4fc2106c20638fa7e0982c35c0e" Nov 04 03:17:32.470572 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:17:32.470496 4405 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"380cfa2a332239970422898ec218bee16d39f4fc2106c20638fa7e0982c35c0e"} err="failed to get container status \"380cfa2a332239970422898ec218bee16d39f4fc2106c20638fa7e0982c35c0e\": rpc error: code = NotFound desc = could not find container \"380cfa2a332239970422898ec218bee16d39f4fc2106c20638fa7e0982c35c0e\": container with ID starting with 380cfa2a332239970422898ec218bee16d39f4fc2106c20638fa7e0982c35c0e not found: ID does not exist" Nov 04 03:17:32.470572 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:17:32.470516 4405 scope.go:122] "RemoveContainer" containerID="81aa272b24db3e2aac8307e85ec1c538754b7333cddcdf89cdfbbdaaf0382d0e" Nov 04 03:17:32.470838 ip-10-0-134-125 kubenswrapper[4405]: E1104 03:17:32.470817 4405 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"81aa272b24db3e2aac8307e85ec1c538754b7333cddcdf89cdfbbdaaf0382d0e\": container with ID starting with 81aa272b24db3e2aac8307e85ec1c538754b7333cddcdf89cdfbbdaaf0382d0e not found: ID does not exist" containerID="81aa272b24db3e2aac8307e85ec1c538754b7333cddcdf89cdfbbdaaf0382d0e" Nov 04 03:17:32.470892 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:17:32.470846 4405 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"81aa272b24db3e2aac8307e85ec1c538754b7333cddcdf89cdfbbdaaf0382d0e"} err="failed to get container status \"81aa272b24db3e2aac8307e85ec1c538754b7333cddcdf89cdfbbdaaf0382d0e\": rpc error: code = NotFound desc = could not find container \"81aa272b24db3e2aac8307e85ec1c538754b7333cddcdf89cdfbbdaaf0382d0e\": container with ID starting with 81aa272b24db3e2aac8307e85ec1c538754b7333cddcdf89cdfbbdaaf0382d0e not found: ID does not exist" Nov 04 03:17:32.495467 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:17:32.495423 4405 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a77157ba-0aed-46cf-90de-ffc15074a50f-istio-podinfo\") on node \"ip-10-0-134-125.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:17:32.495467 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:17:32.495451 4405 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a77157ba-0aed-46cf-90de-ffc15074a50f-istio-data\") on node \"ip-10-0-134-125.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:17:32.495467 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:17:32.495465 4405 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a77157ba-0aed-46cf-90de-ffc15074a50f-workload-socket\") on node \"ip-10-0-134-125.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:17:32.495753 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:17:32.495477 4405 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a77157ba-0aed-46cf-90de-ffc15074a50f-istio-token\") on node \"ip-10-0-134-125.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:17:32.495753 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:17:32.495494 4405 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xxs4z\" (UniqueName: \"kubernetes.io/projected/a77157ba-0aed-46cf-90de-ffc15074a50f-kube-api-access-xxs4z\") on node \"ip-10-0-134-125.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:17:32.495753 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:17:32.495509 4405 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a77157ba-0aed-46cf-90de-ffc15074a50f-istio-envoy\") on node \"ip-10-0-134-125.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:17:32.497750 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:17:32.497725 4405 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-a0bcbc/distinct-w2-a0bcbc-kserve-596f94c7cd-szl4g"] Nov 04 03:17:32.499605 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:17:32.499584 4405 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-a0bcbc/premium-w2-a0bcbc-kserve-78dfd6d78c-lwth2"] Nov 04 03:17:32.662055 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:17:32.662010 4405 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-bl5mg"] Nov 04 03:17:32.728644 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:17:32.728594 4405 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-4cbcf738-openshift-default-68fb497c9c-2ndqr"] Nov 04 03:17:32.733319 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:17:32.733287 4405 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-4cbcf738-openshift-default-68fb497c9c-2ndqr"] Nov 04 03:17:33.700752 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:17:33.700718 4405 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a77157ba-0aed-46cf-90de-ffc15074a50f" path="/var/lib/kubelet/pods/a77157ba-0aed-46cf-90de-ffc15074a50f/volumes" Nov 04 03:17:34.046846 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:17:34.046759 4405 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-vwcx8_ffbb2d83-5a93-4a5b-bce3-e40afb8f58ea/ovn-acl-logging/0.log" Nov 04 03:17:34.051990 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:17:34.051970 4405 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-vwcx8_ffbb2d83-5a93-4a5b-bce3-e40afb8f58ea/ovn-acl-logging/0.log" Nov 04 03:17:34.053287 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:17:34.053254 4405 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-095081-bd58cc787-xz5kc" podUID="3eb56d77-29c0-4a72-863d-354d2cb764a0" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 04 03:17:36.210169 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:17:36.210144 4405 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-brntt" Nov 04 03:17:36.339331 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:17:36.339292 4405 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/423b62dd-792f-442c-b1ac-3248072565e8-istio-token\" (UniqueName: \"kubernetes.io/projected/423b62dd-792f-442c-b1ac-3248072565e8-istio-token\") pod \"423b62dd-792f-442c-b1ac-3248072565e8\" (UID: \"423b62dd-792f-442c-b1ac-3248072565e8\") " Nov 04 03:17:36.339540 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:17:36.339368 4405 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/423b62dd-792f-442c-b1ac-3248072565e8-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/423b62dd-792f-442c-b1ac-3248072565e8-istiod-ca-cert\") pod \"423b62dd-792f-442c-b1ac-3248072565e8\" (UID: \"423b62dd-792f-442c-b1ac-3248072565e8\") " Nov 04 03:17:36.339540 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:17:36.339474 4405 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/423b62dd-792f-442c-b1ac-3248072565e8-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/423b62dd-792f-442c-b1ac-3248072565e8-istio-envoy\") pod \"423b62dd-792f-442c-b1ac-3248072565e8\" (UID: \"423b62dd-792f-442c-b1ac-3248072565e8\") " Nov 04 03:17:36.339540 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:17:36.339517 4405 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/423b62dd-792f-442c-b1ac-3248072565e8-kube-api-access-lw6pt\" (UniqueName: \"kubernetes.io/projected/423b62dd-792f-442c-b1ac-3248072565e8-kube-api-access-lw6pt\") pod \"423b62dd-792f-442c-b1ac-3248072565e8\" (UID: \"423b62dd-792f-442c-b1ac-3248072565e8\") " Nov 04 03:17:36.339764 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:17:36.339560 4405 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/423b62dd-792f-442c-b1ac-3248072565e8-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/423b62dd-792f-442c-b1ac-3248072565e8-istio-podinfo\") pod \"423b62dd-792f-442c-b1ac-3248072565e8\" (UID: \"423b62dd-792f-442c-b1ac-3248072565e8\") " Nov 04 03:17:36.339764 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:17:36.339595 4405 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/423b62dd-792f-442c-b1ac-3248072565e8-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/423b62dd-792f-442c-b1ac-3248072565e8-workload-certs\") pod \"423b62dd-792f-442c-b1ac-3248072565e8\" (UID: \"423b62dd-792f-442c-b1ac-3248072565e8\") " Nov 04 03:17:36.339764 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:17:36.339624 4405 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/423b62dd-792f-442c-b1ac-3248072565e8-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/423b62dd-792f-442c-b1ac-3248072565e8-credential-socket\") pod \"423b62dd-792f-442c-b1ac-3248072565e8\" (UID: \"423b62dd-792f-442c-b1ac-3248072565e8\") " Nov 04 03:17:36.339764 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:17:36.339736 4405 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/423b62dd-792f-442c-b1ac-3248072565e8-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/423b62dd-792f-442c-b1ac-3248072565e8-istio-data\") pod \"423b62dd-792f-442c-b1ac-3248072565e8\" (UID: \"423b62dd-792f-442c-b1ac-3248072565e8\") " Nov 04 03:17:36.339984 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:17:36.339791 4405 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/423b62dd-792f-442c-b1ac-3248072565e8-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/423b62dd-792f-442c-b1ac-3248072565e8-workload-socket\") pod \"423b62dd-792f-442c-b1ac-3248072565e8\" (UID: \"423b62dd-792f-442c-b1ac-3248072565e8\") " Nov 04 03:17:36.339984 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:17:36.339954 4405 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/423b62dd-792f-442c-b1ac-3248072565e8-credential-socket" pod "423b62dd-792f-442c-b1ac-3248072565e8" (UID: "423b62dd-792f-442c-b1ac-3248072565e8"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 03:17:36.340101 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:17:36.339889 4405 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/423b62dd-792f-442c-b1ac-3248072565e8-istiod-ca-cert" pod "423b62dd-792f-442c-b1ac-3248072565e8" (UID: "423b62dd-792f-442c-b1ac-3248072565e8"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 03:17:36.340235 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:17:36.340191 4405 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/423b62dd-792f-442c-b1ac-3248072565e8-workload-socket" pod "423b62dd-792f-442c-b1ac-3248072565e8" (UID: "423b62dd-792f-442c-b1ac-3248072565e8"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 03:17:36.340235 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:17:36.340200 4405 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/423b62dd-792f-442c-b1ac-3248072565e8-istio-data" pod "423b62dd-792f-442c-b1ac-3248072565e8" (UID: "423b62dd-792f-442c-b1ac-3248072565e8"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 03:17:36.340430 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:17:36.340293 4405 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/423b62dd-792f-442c-b1ac-3248072565e8-istiod-ca-cert\") on node \"ip-10-0-134-125.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:17:36.340430 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:17:36.340315 4405 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/423b62dd-792f-442c-b1ac-3248072565e8-credential-socket\") on node \"ip-10-0-134-125.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:17:36.340430 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:17:36.340361 4405 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/423b62dd-792f-442c-b1ac-3248072565e8-workload-certs" pod "423b62dd-792f-442c-b1ac-3248072565e8" (UID: "423b62dd-792f-442c-b1ac-3248072565e8"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 03:17:36.342434 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:17:36.342406 4405 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/423b62dd-792f-442c-b1ac-3248072565e8-istio-envoy" pod "423b62dd-792f-442c-b1ac-3248072565e8" (UID: "423b62dd-792f-442c-b1ac-3248072565e8"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 03:17:36.342534 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:17:36.342428 4405 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/423b62dd-792f-442c-b1ac-3248072565e8-istio-token" pod "423b62dd-792f-442c-b1ac-3248072565e8" (UID: "423b62dd-792f-442c-b1ac-3248072565e8"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 03:17:36.342534 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:17:36.342463 4405 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/423b62dd-792f-442c-b1ac-3248072565e8-istio-podinfo" pod "423b62dd-792f-442c-b1ac-3248072565e8" (UID: "423b62dd-792f-442c-b1ac-3248072565e8"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 04 03:17:36.342606 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:17:36.342523 4405 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/423b62dd-792f-442c-b1ac-3248072565e8-kube-api-access-lw6pt" pod "423b62dd-792f-442c-b1ac-3248072565e8" (UID: "423b62dd-792f-442c-b1ac-3248072565e8"). InnerVolumeSpecName "kube-api-access-lw6pt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 03:17:36.441644 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:17:36.441607 4405 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/423b62dd-792f-442c-b1ac-3248072565e8-istio-data\") on node \"ip-10-0-134-125.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:17:36.441644 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:17:36.441638 4405 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/423b62dd-792f-442c-b1ac-3248072565e8-workload-socket\") on node \"ip-10-0-134-125.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:17:36.441644 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:17:36.441648 4405 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/423b62dd-792f-442c-b1ac-3248072565e8-istio-token\") on node \"ip-10-0-134-125.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:17:36.441924 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:17:36.441656 4405 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/423b62dd-792f-442c-b1ac-3248072565e8-istio-envoy\") on node \"ip-10-0-134-125.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:17:36.441924 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:17:36.441692 4405 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lw6pt\" (UniqueName: \"kubernetes.io/projected/423b62dd-792f-442c-b1ac-3248072565e8-kube-api-access-lw6pt\") on node \"ip-10-0-134-125.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:17:36.441924 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:17:36.441708 4405 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/423b62dd-792f-442c-b1ac-3248072565e8-istio-podinfo\") on node \"ip-10-0-134-125.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:17:36.441924 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:17:36.441717 4405 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/423b62dd-792f-442c-b1ac-3248072565e8-workload-certs\") on node \"ip-10-0-134-125.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:17:36.469175 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:17:36.469141 4405 generic.go:356] "Generic (PLEG): container finished" podID="423b62dd-792f-442c-b1ac-3248072565e8" containerID="e24a32ba8e02e59a6d4901acdaf8b888fd3d35821730e056c2e4aec27ad72fe5" exitCode=0 Nov 04 03:17:36.469372 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:17:36.469205 4405 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-brntt" event={"ID":"423b62dd-792f-442c-b1ac-3248072565e8","Type":"ContainerDied","Data":"e24a32ba8e02e59a6d4901acdaf8b888fd3d35821730e056c2e4aec27ad72fe5"} Nov 04 03:17:36.469372 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:17:36.469250 4405 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-brntt" event={"ID":"423b62dd-792f-442c-b1ac-3248072565e8","Type":"ContainerDied","Data":"578adc0e4480fcca480ac0029d68baeb2c3e1f719b0330dd5387c9d245860c9d"} Nov 04 03:17:36.469372 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:17:36.469278 4405 scope.go:122] "RemoveContainer" containerID="e24a32ba8e02e59a6d4901acdaf8b888fd3d35821730e056c2e4aec27ad72fe5" Nov 04 03:17:36.469372 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:17:36.469206 4405 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-brntt" Nov 04 03:17:36.479878 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:17:36.479856 4405 scope.go:122] "RemoveContainer" containerID="e24a32ba8e02e59a6d4901acdaf8b888fd3d35821730e056c2e4aec27ad72fe5" Nov 04 03:17:36.480164 ip-10-0-134-125 kubenswrapper[4405]: E1104 03:17:36.480143 4405 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e24a32ba8e02e59a6d4901acdaf8b888fd3d35821730e056c2e4aec27ad72fe5\": container with ID starting with e24a32ba8e02e59a6d4901acdaf8b888fd3d35821730e056c2e4aec27ad72fe5 not found: ID does not exist" containerID="e24a32ba8e02e59a6d4901acdaf8b888fd3d35821730e056c2e4aec27ad72fe5" Nov 04 03:17:36.480219 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:17:36.480173 4405 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e24a32ba8e02e59a6d4901acdaf8b888fd3d35821730e056c2e4aec27ad72fe5"} err="failed to get container status \"e24a32ba8e02e59a6d4901acdaf8b888fd3d35821730e056c2e4aec27ad72fe5\": rpc error: code = NotFound desc = could not find container \"e24a32ba8e02e59a6d4901acdaf8b888fd3d35821730e056c2e4aec27ad72fe5\": container with ID starting with e24a32ba8e02e59a6d4901acdaf8b888fd3d35821730e056c2e4aec27ad72fe5 not found: ID does not exist" Nov 04 03:17:36.558857 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:17:36.558823 4405 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-095081-85f7c6c868-qh2c7"] Nov 04 03:17:36.559059 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:17:36.559030 4405 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-095081-85f7c6c868-qh2c7" podUID="239f941f-cf45-4550-90cd-e2be9f961e07" containerName="maas-api" containerID="cri-o://6158d11aea76349ce7eeae299fa573bd28f033422a27089a724230063d24d76f" gracePeriod=30 Nov 04 03:17:36.634593 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:17:36.634508 4405 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-76597d6855-brntt"] Nov 04 03:17:36.638929 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:17:36.638905 4405 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-76597d6855-brntt"] Nov 04 03:17:36.815105 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:17:36.815081 4405 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-095081-85f7c6c868-qh2c7" Nov 04 03:17:36.884151 ip-10-0-134-125 kubenswrapper[4405]: E1104 03:17:36.884110 4405 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5f2vb" containerName="extractor" Nov 04 03:17:36.946851 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:17:36.946483 4405 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/239f941f-cf45-4550-90cd-e2be9f961e07-kube-api-access-jd9j2\" (UniqueName: \"kubernetes.io/projected/239f941f-cf45-4550-90cd-e2be9f961e07-kube-api-access-jd9j2\") pod \"239f941f-cf45-4550-90cd-e2be9f961e07\" (UID: \"239f941f-cf45-4550-90cd-e2be9f961e07\") " Nov 04 03:17:36.946851 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:17:36.946583 4405 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/239f941f-cf45-4550-90cd-e2be9f961e07-metrics-certs\" (UniqueName: \"kubernetes.io/secret/239f941f-cf45-4550-90cd-e2be9f961e07-metrics-certs\") pod \"239f941f-cf45-4550-90cd-e2be9f961e07\" (UID: \"239f941f-cf45-4550-90cd-e2be9f961e07\") " Nov 04 03:17:36.946851 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:17:36.946635 4405 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/239f941f-cf45-4550-90cd-e2be9f961e07-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/239f941f-cf45-4550-90cd-e2be9f961e07-openshift-service-ca\") pod \"239f941f-cf45-4550-90cd-e2be9f961e07\" (UID: \"239f941f-cf45-4550-90cd-e2be9f961e07\") " Nov 04 03:17:36.946851 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:17:36.946740 4405 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/239f941f-cf45-4550-90cd-e2be9f961e07-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/239f941f-cf45-4550-90cd-e2be9f961e07-maas-api-tls\") pod \"239f941f-cf45-4550-90cd-e2be9f961e07\" (UID: \"239f941f-cf45-4550-90cd-e2be9f961e07\") " Nov 04 03:17:36.948011 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:17:36.947979 4405 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/239f941f-cf45-4550-90cd-e2be9f961e07-openshift-service-ca" pod "239f941f-cf45-4550-90cd-e2be9f961e07" (UID: "239f941f-cf45-4550-90cd-e2be9f961e07"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 03:17:36.950123 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:17:36.950095 4405 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/239f941f-cf45-4550-90cd-e2be9f961e07-metrics-certs" pod "239f941f-cf45-4550-90cd-e2be9f961e07" (UID: "239f941f-cf45-4550-90cd-e2be9f961e07"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 04 03:17:36.950483 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:17:36.950458 4405 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/239f941f-cf45-4550-90cd-e2be9f961e07-maas-api-tls" pod "239f941f-cf45-4550-90cd-e2be9f961e07" (UID: "239f941f-cf45-4550-90cd-e2be9f961e07"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 04 03:17:36.952532 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:17:36.952502 4405 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/239f941f-cf45-4550-90cd-e2be9f961e07-kube-api-access-jd9j2" pod "239f941f-cf45-4550-90cd-e2be9f961e07" (UID: "239f941f-cf45-4550-90cd-e2be9f961e07"). InnerVolumeSpecName "kube-api-access-jd9j2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 03:17:37.048292 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:17:37.048256 4405 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/239f941f-cf45-4550-90cd-e2be9f961e07-openshift-service-ca\") on node \"ip-10-0-134-125.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:17:37.048292 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:17:37.048295 4405 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/239f941f-cf45-4550-90cd-e2be9f961e07-maas-api-tls\") on node \"ip-10-0-134-125.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:17:37.048532 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:17:37.048311 4405 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jd9j2\" (UniqueName: \"kubernetes.io/projected/239f941f-cf45-4550-90cd-e2be9f961e07-kube-api-access-jd9j2\") on node \"ip-10-0-134-125.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:17:37.048532 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:17:37.048325 4405 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/239f941f-cf45-4550-90cd-e2be9f961e07-metrics-certs\") on node \"ip-10-0-134-125.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:17:37.448242 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:17:37.448214 4405 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-095081-bd58cc787-xz5kc" Nov 04 03:17:37.475930 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:17:37.475846 4405 generic.go:356] "Generic (PLEG): container finished" podID="3eb56d77-29c0-4a72-863d-354d2cb764a0" containerID="661b7e606486e657909844b5e698bca6cd782df0cfdfc06b6b55264c70e08797" exitCode=0 Nov 04 03:17:37.475930 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:17:37.475897 4405 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-095081-bd58cc787-xz5kc" Nov 04 03:17:37.476122 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:17:37.475927 4405 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-095081-bd58cc787-xz5kc" event={"ID":"3eb56d77-29c0-4a72-863d-354d2cb764a0","Type":"ContainerDied","Data":"661b7e606486e657909844b5e698bca6cd782df0cfdfc06b6b55264c70e08797"} Nov 04 03:17:37.476122 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:17:37.475975 4405 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-095081-bd58cc787-xz5kc" event={"ID":"3eb56d77-29c0-4a72-863d-354d2cb764a0","Type":"ContainerDied","Data":"c3b82e72aefd1a76a6ce8a5c671fefd798211e8dafabac17809a5d6b94211499"} Nov 04 03:17:37.476122 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:17:37.475997 4405 scope.go:122] "RemoveContainer" containerID="661b7e606486e657909844b5e698bca6cd782df0cfdfc06b6b55264c70e08797" Nov 04 03:17:37.477272 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:17:37.477251 4405 generic.go:356] "Generic (PLEG): container finished" podID="239f941f-cf45-4550-90cd-e2be9f961e07" containerID="6158d11aea76349ce7eeae299fa573bd28f033422a27089a724230063d24d76f" exitCode=0 Nov 04 03:17:37.477381 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:17:37.477307 4405 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-095081-85f7c6c868-qh2c7" Nov 04 03:17:37.477381 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:17:37.477322 4405 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-095081-85f7c6c868-qh2c7" event={"ID":"239f941f-cf45-4550-90cd-e2be9f961e07","Type":"ContainerDied","Data":"6158d11aea76349ce7eeae299fa573bd28f033422a27089a724230063d24d76f"} Nov 04 03:17:37.477381 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:17:37.477362 4405 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-095081-85f7c6c868-qh2c7" event={"ID":"239f941f-cf45-4550-90cd-e2be9f961e07","Type":"ContainerDied","Data":"93c80311be729d7708ea18f3a8b3a87e46cbc881265eae5eed9c5f08cfc10b2f"} Nov 04 03:17:37.488989 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:17:37.488119 4405 scope.go:122] "RemoveContainer" containerID="661b7e606486e657909844b5e698bca6cd782df0cfdfc06b6b55264c70e08797" Nov 04 03:17:37.492093 ip-10-0-134-125 kubenswrapper[4405]: E1104 03:17:37.492065 4405 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"661b7e606486e657909844b5e698bca6cd782df0cfdfc06b6b55264c70e08797\": container with ID starting with 661b7e606486e657909844b5e698bca6cd782df0cfdfc06b6b55264c70e08797 not found: ID does not exist" containerID="661b7e606486e657909844b5e698bca6cd782df0cfdfc06b6b55264c70e08797" Nov 04 03:17:37.492224 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:17:37.492103 4405 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"661b7e606486e657909844b5e698bca6cd782df0cfdfc06b6b55264c70e08797"} err="failed to get container status \"661b7e606486e657909844b5e698bca6cd782df0cfdfc06b6b55264c70e08797\": rpc error: code = NotFound desc = could not find container \"661b7e606486e657909844b5e698bca6cd782df0cfdfc06b6b55264c70e08797\": container with ID starting with 661b7e606486e657909844b5e698bca6cd782df0cfdfc06b6b55264c70e08797 not found: ID does not exist" Nov 04 03:17:37.492224 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:17:37.492128 4405 scope.go:122] "RemoveContainer" containerID="6158d11aea76349ce7eeae299fa573bd28f033422a27089a724230063d24d76f" Nov 04 03:17:37.505136 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:17:37.505108 4405 scope.go:122] "RemoveContainer" containerID="6158d11aea76349ce7eeae299fa573bd28f033422a27089a724230063d24d76f" Nov 04 03:17:37.505791 ip-10-0-134-125 kubenswrapper[4405]: E1104 03:17:37.505733 4405 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6158d11aea76349ce7eeae299fa573bd28f033422a27089a724230063d24d76f\": container with ID starting with 6158d11aea76349ce7eeae299fa573bd28f033422a27089a724230063d24d76f not found: ID does not exist" containerID="6158d11aea76349ce7eeae299fa573bd28f033422a27089a724230063d24d76f" Nov 04 03:17:37.505791 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:17:37.505778 4405 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6158d11aea76349ce7eeae299fa573bd28f033422a27089a724230063d24d76f"} err="failed to get container status \"6158d11aea76349ce7eeae299fa573bd28f033422a27089a724230063d24d76f\": rpc error: code = NotFound desc = could not find container \"6158d11aea76349ce7eeae299fa573bd28f033422a27089a724230063d24d76f\": container with ID starting with 6158d11aea76349ce7eeae299fa573bd28f033422a27089a724230063d24d76f not found: ID does not exist" Nov 04 03:17:37.553814 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:17:37.553778 4405 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3eb56d77-29c0-4a72-863d-354d2cb764a0-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3eb56d77-29c0-4a72-863d-354d2cb764a0-plugins-config-volume\") pod \"3eb56d77-29c0-4a72-863d-354d2cb764a0\" (UID: \"3eb56d77-29c0-4a72-863d-354d2cb764a0\") " Nov 04 03:17:37.553980 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:17:37.553900 4405 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3eb56d77-29c0-4a72-863d-354d2cb764a0-kube-api-access-vqvjz\" (UniqueName: \"kubernetes.io/projected/3eb56d77-29c0-4a72-863d-354d2cb764a0-kube-api-access-vqvjz\") pod \"3eb56d77-29c0-4a72-863d-354d2cb764a0\" (UID: \"3eb56d77-29c0-4a72-863d-354d2cb764a0\") " Nov 04 03:17:37.554172 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:17:37.554151 4405 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3eb56d77-29c0-4a72-863d-354d2cb764a0-plugins-config-volume" pod "3eb56d77-29c0-4a72-863d-354d2cb764a0" (UID: "3eb56d77-29c0-4a72-863d-354d2cb764a0"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 03:17:37.556112 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:17:37.556086 4405 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3eb56d77-29c0-4a72-863d-354d2cb764a0-kube-api-access-vqvjz" pod "3eb56d77-29c0-4a72-863d-354d2cb764a0" (UID: "3eb56d77-29c0-4a72-863d-354d2cb764a0"). InnerVolumeSpecName "kube-api-access-vqvjz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 03:17:37.655381 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:17:37.655334 4405 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vqvjz\" (UniqueName: \"kubernetes.io/projected/3eb56d77-29c0-4a72-863d-354d2cb764a0-kube-api-access-vqvjz\") on node \"ip-10-0-134-125.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:17:37.655381 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:17:37.655377 4405 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3eb56d77-29c0-4a72-863d-354d2cb764a0-plugins-config-volume\") on node \"ip-10-0-134-125.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:17:37.675102 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:17:37.675069 4405 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-095081-85f7c6c868-qh2c7"] Nov 04 03:17:37.678853 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:17:37.678829 4405 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-095081-85f7c6c868-qh2c7"] Nov 04 03:17:37.700239 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:17:37.700212 4405 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="239f941f-cf45-4550-90cd-e2be9f961e07" path="/var/lib/kubelet/pods/239f941f-cf45-4550-90cd-e2be9f961e07/volumes" Nov 04 03:17:37.700784 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:17:37.700763 4405 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="423b62dd-792f-442c-b1ac-3248072565e8" path="/var/lib/kubelet/pods/423b62dd-792f-442c-b1ac-3248072565e8/volumes" Nov 04 03:17:37.890271 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:17:37.890237 4405 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-095081-bd58cc787-xz5kc"] Nov 04 03:17:37.894542 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:17:37.894517 4405 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-095081-bd58cc787-xz5kc"] Nov 04 03:17:39.699618 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:17:39.699586 4405 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3eb56d77-29c0-4a72-863d-354d2cb764a0" path="/var/lib/kubelet/pods/3eb56d77-29c0-4a72-863d-354d2cb764a0/volumes" Nov 04 03:17:45.692433 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:17:45.692397 4405 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w2-a0bcbc-69fdc5f5c9-d4d6x"] Nov 04 03:17:47.518601 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:17:47.518534 4405 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w2-a0bcbc/distinct-w2-a0bcbc-kserve-596f94c7cd-szl4g" podUID="bac32934-8a57-4919-b435-13f367cbea3e" containerName="main" containerID="cri-o://1ca1b0a45ab98ebeb37408d4b3f67c175e2691ff9f4b7cfd0b75093140603b9f" gracePeriod=45 Nov 04 03:17:47.519395 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:17:47.519368 4405 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w2-a0bcbc/premium-w2-a0bcbc-kserve-78dfd6d78c-lwth2" podUID="33b620c1-389c-48b4-b277-90252018c4b6" containerName="main" containerID="cri-o://1249ce7eac538936bf2968a97afdd41b3b45bf0686f95dc6312213addef6436f" gracePeriod=45 Nov 04 03:17:47.788375 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:17:47.788308 4405 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-a0bcbc/premium-w2-a0bcbc-kserve-78dfd6d78c-lwth2" Nov 04 03:17:47.828826 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:17:47.828797 4405 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-a0bcbc/distinct-w2-a0bcbc-kserve-596f94c7cd-szl4g" Nov 04 03:17:47.966379 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:17:47.966347 4405 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/33b620c1-389c-48b4-b277-90252018c4b6-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/33b620c1-389c-48b4-b277-90252018c4b6-tmp-dir\") pod \"33b620c1-389c-48b4-b277-90252018c4b6\" (UID: \"33b620c1-389c-48b4-b277-90252018c4b6\") " Nov 04 03:17:47.966582 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:17:47.966396 4405 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bac32934-8a57-4919-b435-13f367cbea3e-home\" (UniqueName: \"kubernetes.io/empty-dir/bac32934-8a57-4919-b435-13f367cbea3e-home\") pod \"bac32934-8a57-4919-b435-13f367cbea3e\" (UID: \"bac32934-8a57-4919-b435-13f367cbea3e\") " Nov 04 03:17:47.966582 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:17:47.966419 4405 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/33b620c1-389c-48b4-b277-90252018c4b6-tls-certs\" (UniqueName: \"kubernetes.io/secret/33b620c1-389c-48b4-b277-90252018c4b6-tls-certs\") pod \"33b620c1-389c-48b4-b277-90252018c4b6\" (UID: \"33b620c1-389c-48b4-b277-90252018c4b6\") " Nov 04 03:17:47.966582 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:17:47.966451 4405 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bac32934-8a57-4919-b435-13f367cbea3e-dshm\" (UniqueName: \"kubernetes.io/empty-dir/bac32934-8a57-4919-b435-13f367cbea3e-dshm\") pod \"bac32934-8a57-4919-b435-13f367cbea3e\" (UID: \"bac32934-8a57-4919-b435-13f367cbea3e\") " Nov 04 03:17:47.966582 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:17:47.966487 4405 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/bac32934-8a57-4919-b435-13f367cbea3e-tls-certs\" (UniqueName: \"kubernetes.io/secret/bac32934-8a57-4919-b435-13f367cbea3e-tls-certs\") pod \"bac32934-8a57-4919-b435-13f367cbea3e\" (UID: \"bac32934-8a57-4919-b435-13f367cbea3e\") " Nov 04 03:17:47.966582 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:17:47.966521 4405 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/33b620c1-389c-48b4-b277-90252018c4b6-kube-api-access-45xsf\" (UniqueName: \"kubernetes.io/projected/33b620c1-389c-48b4-b277-90252018c4b6-kube-api-access-45xsf\") pod \"33b620c1-389c-48b4-b277-90252018c4b6\" (UID: \"33b620c1-389c-48b4-b277-90252018c4b6\") " Nov 04 03:17:47.966582 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:17:47.966555 4405 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/33b620c1-389c-48b4-b277-90252018c4b6-dshm\" (UniqueName: \"kubernetes.io/empty-dir/33b620c1-389c-48b4-b277-90252018c4b6-dshm\") pod \"33b620c1-389c-48b4-b277-90252018c4b6\" (UID: \"33b620c1-389c-48b4-b277-90252018c4b6\") " Nov 04 03:17:47.967009 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:17:47.966594 4405 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/33b620c1-389c-48b4-b277-90252018c4b6-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/33b620c1-389c-48b4-b277-90252018c4b6-model-cache\") pod \"33b620c1-389c-48b4-b277-90252018c4b6\" (UID: \"33b620c1-389c-48b4-b277-90252018c4b6\") " Nov 04 03:17:47.967009 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:17:47.966629 4405 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/33b620c1-389c-48b4-b277-90252018c4b6-tmp-dir" pod "33b620c1-389c-48b4-b277-90252018c4b6" (UID: "33b620c1-389c-48b4-b277-90252018c4b6"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 03:17:47.967009 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:17:47.966645 4405 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bac32934-8a57-4919-b435-13f367cbea3e-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bac32934-8a57-4919-b435-13f367cbea3e-model-cache\") pod \"bac32934-8a57-4919-b435-13f367cbea3e\" (UID: \"bac32934-8a57-4919-b435-13f367cbea3e\") " Nov 04 03:17:47.967009 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:17:47.966710 4405 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bac32934-8a57-4919-b435-13f367cbea3e-kube-api-access-tv4lr\" (UniqueName: \"kubernetes.io/projected/bac32934-8a57-4919-b435-13f367cbea3e-kube-api-access-tv4lr\") pod \"bac32934-8a57-4919-b435-13f367cbea3e\" (UID: \"bac32934-8a57-4919-b435-13f367cbea3e\") " Nov 04 03:17:47.967009 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:17:47.966742 4405 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bac32934-8a57-4919-b435-13f367cbea3e-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bac32934-8a57-4919-b435-13f367cbea3e-tmp-dir\") pod \"bac32934-8a57-4919-b435-13f367cbea3e\" (UID: \"bac32934-8a57-4919-b435-13f367cbea3e\") " Nov 04 03:17:47.967009 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:17:47.966765 4405 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bac32934-8a57-4919-b435-13f367cbea3e-home" pod "bac32934-8a57-4919-b435-13f367cbea3e" (UID: "bac32934-8a57-4919-b435-13f367cbea3e"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 03:17:47.967009 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:17:47.966836 4405 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/33b620c1-389c-48b4-b277-90252018c4b6-home\" (UniqueName: \"kubernetes.io/empty-dir/33b620c1-389c-48b4-b277-90252018c4b6-home\") pod \"33b620c1-389c-48b4-b277-90252018c4b6\" (UID: \"33b620c1-389c-48b4-b277-90252018c4b6\") " Nov 04 03:17:47.967009 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:17:47.966986 4405 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bac32934-8a57-4919-b435-13f367cbea3e-model-cache" pod "bac32934-8a57-4919-b435-13f367cbea3e" (UID: "bac32934-8a57-4919-b435-13f367cbea3e"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 03:17:47.967450 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:17:47.967068 4405 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/33b620c1-389c-48b4-b277-90252018c4b6-model-cache" pod "33b620c1-389c-48b4-b277-90252018c4b6" (UID: "33b620c1-389c-48b4-b277-90252018c4b6"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 03:17:47.967450 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:17:47.967196 4405 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bac32934-8a57-4919-b435-13f367cbea3e-tmp-dir" pod "bac32934-8a57-4919-b435-13f367cbea3e" (UID: "bac32934-8a57-4919-b435-13f367cbea3e"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 03:17:47.967450 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:17:47.967302 4405 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/33b620c1-389c-48b4-b277-90252018c4b6-home" pod "33b620c1-389c-48b4-b277-90252018c4b6" (UID: "33b620c1-389c-48b4-b277-90252018c4b6"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 03:17:47.967628 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:17:47.967561 4405 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/33b620c1-389c-48b4-b277-90252018c4b6-model-cache\") on node \"ip-10-0-134-125.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:17:47.967628 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:17:47.967581 4405 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bac32934-8a57-4919-b435-13f367cbea3e-model-cache\") on node \"ip-10-0-134-125.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:17:47.967628 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:17:47.967598 4405 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bac32934-8a57-4919-b435-13f367cbea3e-tmp-dir\") on node \"ip-10-0-134-125.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:17:47.967628 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:17:47.967613 4405 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/33b620c1-389c-48b4-b277-90252018c4b6-home\") on node \"ip-10-0-134-125.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:17:47.967628 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:17:47.967625 4405 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/33b620c1-389c-48b4-b277-90252018c4b6-tmp-dir\") on node \"ip-10-0-134-125.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:17:47.968110 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:17:47.967639 4405 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bac32934-8a57-4919-b435-13f367cbea3e-home\") on node \"ip-10-0-134-125.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:17:47.969458 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:17:47.969427 4405 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bac32934-8a57-4919-b435-13f367cbea3e-tls-certs" pod "bac32934-8a57-4919-b435-13f367cbea3e" (UID: "bac32934-8a57-4919-b435-13f367cbea3e"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 04 03:17:47.969590 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:17:47.969481 4405 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bac32934-8a57-4919-b435-13f367cbea3e-dshm" pod "bac32934-8a57-4919-b435-13f367cbea3e" (UID: "bac32934-8a57-4919-b435-13f367cbea3e"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 03:17:47.969590 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:17:47.969510 4405 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/33b620c1-389c-48b4-b277-90252018c4b6-tls-certs" pod "33b620c1-389c-48b4-b277-90252018c4b6" (UID: "33b620c1-389c-48b4-b277-90252018c4b6"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 04 03:17:47.969590 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:17:47.969534 4405 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bac32934-8a57-4919-b435-13f367cbea3e-kube-api-access-tv4lr" pod "bac32934-8a57-4919-b435-13f367cbea3e" (UID: "bac32934-8a57-4919-b435-13f367cbea3e"). InnerVolumeSpecName "kube-api-access-tv4lr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 03:17:47.969739 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:17:47.969595 4405 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/33b620c1-389c-48b4-b277-90252018c4b6-dshm" pod "33b620c1-389c-48b4-b277-90252018c4b6" (UID: "33b620c1-389c-48b4-b277-90252018c4b6"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 03:17:47.970365 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:17:47.970345 4405 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/33b620c1-389c-48b4-b277-90252018c4b6-kube-api-access-45xsf" pod "33b620c1-389c-48b4-b277-90252018c4b6" (UID: "33b620c1-389c-48b4-b277-90252018c4b6"). InnerVolumeSpecName "kube-api-access-45xsf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 03:17:48.068803 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:17:48.068714 4405 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-45xsf\" (UniqueName: \"kubernetes.io/projected/33b620c1-389c-48b4-b277-90252018c4b6-kube-api-access-45xsf\") on node \"ip-10-0-134-125.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:17:48.068803 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:17:48.068743 4405 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/33b620c1-389c-48b4-b277-90252018c4b6-dshm\") on node \"ip-10-0-134-125.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:17:48.068803 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:17:48.068753 4405 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tv4lr\" (UniqueName: \"kubernetes.io/projected/bac32934-8a57-4919-b435-13f367cbea3e-kube-api-access-tv4lr\") on node \"ip-10-0-134-125.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:17:48.068803 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:17:48.068762 4405 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/33b620c1-389c-48b4-b277-90252018c4b6-tls-certs\") on node \"ip-10-0-134-125.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:17:48.068803 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:17:48.068771 4405 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bac32934-8a57-4919-b435-13f367cbea3e-dshm\") on node \"ip-10-0-134-125.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:17:48.068803 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:17:48.068781 4405 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bac32934-8a57-4919-b435-13f367cbea3e-tls-certs\") on node \"ip-10-0-134-125.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:17:48.545855 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:17:48.545819 4405 generic.go:356] "Generic (PLEG): container finished" podID="33b620c1-389c-48b4-b277-90252018c4b6" containerID="1249ce7eac538936bf2968a97afdd41b3b45bf0686f95dc6312213addef6436f" exitCode=0 Nov 04 03:17:48.546323 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:17:48.545872 4405 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-a0bcbc/premium-w2-a0bcbc-kserve-78dfd6d78c-lwth2" Nov 04 03:17:48.546323 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:17:48.545900 4405 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-a0bcbc/premium-w2-a0bcbc-kserve-78dfd6d78c-lwth2" event={"ID":"33b620c1-389c-48b4-b277-90252018c4b6","Type":"ContainerDied","Data":"1249ce7eac538936bf2968a97afdd41b3b45bf0686f95dc6312213addef6436f"} Nov 04 03:17:48.546323 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:17:48.545952 4405 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-a0bcbc/premium-w2-a0bcbc-kserve-78dfd6d78c-lwth2" event={"ID":"33b620c1-389c-48b4-b277-90252018c4b6","Type":"ContainerDied","Data":"ef6737ae624d86a300660bbba32e94d789f873fd430f2a839933742afa1ddc0b"} Nov 04 03:17:48.546323 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:17:48.545973 4405 scope.go:122] "RemoveContainer" containerID="1249ce7eac538936bf2968a97afdd41b3b45bf0686f95dc6312213addef6436f" Nov 04 03:17:48.547356 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:17:48.547334 4405 generic.go:356] "Generic (PLEG): container finished" podID="bac32934-8a57-4919-b435-13f367cbea3e" containerID="1ca1b0a45ab98ebeb37408d4b3f67c175e2691ff9f4b7cfd0b75093140603b9f" exitCode=0 Nov 04 03:17:48.547445 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:17:48.547378 4405 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-a0bcbc/distinct-w2-a0bcbc-kserve-596f94c7cd-szl4g" Nov 04 03:17:48.547445 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:17:48.547409 4405 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-a0bcbc/distinct-w2-a0bcbc-kserve-596f94c7cd-szl4g" event={"ID":"bac32934-8a57-4919-b435-13f367cbea3e","Type":"ContainerDied","Data":"1ca1b0a45ab98ebeb37408d4b3f67c175e2691ff9f4b7cfd0b75093140603b9f"} Nov 04 03:17:48.547445 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:17:48.547436 4405 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-a0bcbc/distinct-w2-a0bcbc-kserve-596f94c7cd-szl4g" event={"ID":"bac32934-8a57-4919-b435-13f367cbea3e","Type":"ContainerDied","Data":"b4f4a6104efcef2678635173d0214e0719a8cced4f0bfb6bb0893d54defe05e1"} Nov 04 03:17:48.555714 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:17:48.555696 4405 scope.go:122] "RemoveContainer" containerID="1249ce7eac538936bf2968a97afdd41b3b45bf0686f95dc6312213addef6436f" Nov 04 03:17:48.556018 ip-10-0-134-125 kubenswrapper[4405]: E1104 03:17:48.555999 4405 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1249ce7eac538936bf2968a97afdd41b3b45bf0686f95dc6312213addef6436f\": container with ID starting with 1249ce7eac538936bf2968a97afdd41b3b45bf0686f95dc6312213addef6436f not found: ID does not exist" containerID="1249ce7eac538936bf2968a97afdd41b3b45bf0686f95dc6312213addef6436f" Nov 04 03:17:48.556112 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:17:48.556025 4405 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1249ce7eac538936bf2968a97afdd41b3b45bf0686f95dc6312213addef6436f"} err="failed to get container status \"1249ce7eac538936bf2968a97afdd41b3b45bf0686f95dc6312213addef6436f\": rpc error: code = NotFound desc = could not find container \"1249ce7eac538936bf2968a97afdd41b3b45bf0686f95dc6312213addef6436f\": container with ID starting with 1249ce7eac538936bf2968a97afdd41b3b45bf0686f95dc6312213addef6436f not found: ID does not exist" Nov 04 03:17:48.556112 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:17:48.556040 4405 scope.go:122] "RemoveContainer" containerID="1ca1b0a45ab98ebeb37408d4b3f67c175e2691ff9f4b7cfd0b75093140603b9f" Nov 04 03:17:48.564402 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:17:48.564386 4405 scope.go:122] "RemoveContainer" containerID="1ca1b0a45ab98ebeb37408d4b3f67c175e2691ff9f4b7cfd0b75093140603b9f" Nov 04 03:17:48.564716 ip-10-0-134-125 kubenswrapper[4405]: E1104 03:17:48.564697 4405 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1ca1b0a45ab98ebeb37408d4b3f67c175e2691ff9f4b7cfd0b75093140603b9f\": container with ID starting with 1ca1b0a45ab98ebeb37408d4b3f67c175e2691ff9f4b7cfd0b75093140603b9f not found: ID does not exist" containerID="1ca1b0a45ab98ebeb37408d4b3f67c175e2691ff9f4b7cfd0b75093140603b9f" Nov 04 03:17:48.564773 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:17:48.564723 4405 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1ca1b0a45ab98ebeb37408d4b3f67c175e2691ff9f4b7cfd0b75093140603b9f"} err="failed to get container status \"1ca1b0a45ab98ebeb37408d4b3f67c175e2691ff9f4b7cfd0b75093140603b9f\": rpc error: code = NotFound desc = could not find container \"1ca1b0a45ab98ebeb37408d4b3f67c175e2691ff9f4b7cfd0b75093140603b9f\": container with ID starting with 1ca1b0a45ab98ebeb37408d4b3f67c175e2691ff9f4b7cfd0b75093140603b9f not found: ID does not exist" Nov 04 03:17:48.709810 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:17:48.709773 4405 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-a0bcbc/premium-w2-a0bcbc-kserve-78dfd6d78c-lwth2"] Nov 04 03:17:48.714640 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:17:48.714614 4405 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w2-a0bcbc/premium-w2-a0bcbc-kserve-78dfd6d78c-lwth2"] Nov 04 03:17:48.822909 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:17:48.822824 4405 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-a0bcbc/distinct-w2-a0bcbc-kserve-596f94c7cd-szl4g"] Nov 04 03:17:48.829249 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:17:48.829215 4405 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w2-a0bcbc/distinct-w2-a0bcbc-kserve-596f94c7cd-szl4g"] Nov 04 03:17:49.701277 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:17:49.701241 4405 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="33b620c1-389c-48b4-b277-90252018c4b6" path="/var/lib/kubelet/pods/33b620c1-389c-48b4-b277-90252018c4b6/volumes" Nov 04 03:17:49.701784 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:17:49.701655 4405 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bac32934-8a57-4919-b435-13f367cbea3e" path="/var/lib/kubelet/pods/bac32934-8a57-4919-b435-13f367cbea3e/volumes" Nov 04 03:17:50.692970 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:17:50.692926 4405 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-a0bcbc-69fdc5f5c9-d4d6x" podUID="ed86675a-731f-4625-bca5-02ba4200e7ad" containerName="payload-pre-processing" containerID="cri-o://8da84aeee90456df8fa2e6415da3074463772dbec91eedcf12da9984dc29d885" gracePeriod=25 Nov 04 03:17:50.950744 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:17:50.950659 4405 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-a0bcbc-69fdc5f5c9-d4d6x" Nov 04 03:17:51.100762 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:17:51.100731 4405 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ed86675a-731f-4625-bca5-02ba4200e7ad-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ed86675a-731f-4625-bca5-02ba4200e7ad-plugins-config-volume\") pod \"ed86675a-731f-4625-bca5-02ba4200e7ad\" (UID: \"ed86675a-731f-4625-bca5-02ba4200e7ad\") " Nov 04 03:17:51.100945 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:17:51.100783 4405 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ed86675a-731f-4625-bca5-02ba4200e7ad-kube-api-access-5f4mh\" (UniqueName: \"kubernetes.io/projected/ed86675a-731f-4625-bca5-02ba4200e7ad-kube-api-access-5f4mh\") pod \"ed86675a-731f-4625-bca5-02ba4200e7ad\" (UID: \"ed86675a-731f-4625-bca5-02ba4200e7ad\") " Nov 04 03:17:51.101123 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:17:51.101100 4405 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ed86675a-731f-4625-bca5-02ba4200e7ad-plugins-config-volume" pod "ed86675a-731f-4625-bca5-02ba4200e7ad" (UID: "ed86675a-731f-4625-bca5-02ba4200e7ad"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 03:17:51.103157 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:17:51.103130 4405 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ed86675a-731f-4625-bca5-02ba4200e7ad-kube-api-access-5f4mh" pod "ed86675a-731f-4625-bca5-02ba4200e7ad" (UID: "ed86675a-731f-4625-bca5-02ba4200e7ad"). InnerVolumeSpecName "kube-api-access-5f4mh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 03:17:51.201925 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:17:51.201844 4405 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ed86675a-731f-4625-bca5-02ba4200e7ad-plugins-config-volume\") on node \"ip-10-0-134-125.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:17:51.201925 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:17:51.201872 4405 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5f4mh\" (UniqueName: \"kubernetes.io/projected/ed86675a-731f-4625-bca5-02ba4200e7ad-kube-api-access-5f4mh\") on node \"ip-10-0-134-125.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:17:51.567946 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:17:51.567855 4405 generic.go:356] "Generic (PLEG): container finished" podID="ed86675a-731f-4625-bca5-02ba4200e7ad" containerID="8da84aeee90456df8fa2e6415da3074463772dbec91eedcf12da9984dc29d885" exitCode=0 Nov 04 03:17:51.567946 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:17:51.567923 4405 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-a0bcbc-69fdc5f5c9-d4d6x" Nov 04 03:17:51.568198 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:17:51.567945 4405 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-a0bcbc-69fdc5f5c9-d4d6x" event={"ID":"ed86675a-731f-4625-bca5-02ba4200e7ad","Type":"ContainerDied","Data":"8da84aeee90456df8fa2e6415da3074463772dbec91eedcf12da9984dc29d885"} Nov 04 03:17:51.568198 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:17:51.568045 4405 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-a0bcbc-69fdc5f5c9-d4d6x" event={"ID":"ed86675a-731f-4625-bca5-02ba4200e7ad","Type":"ContainerDied","Data":"24e4046d6a8c209ba6f6a2597803f1667bc4d006f21add2f7b11f9d3c4f3131b"} Nov 04 03:17:51.568198 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:17:51.568078 4405 scope.go:122] "RemoveContainer" containerID="8da84aeee90456df8fa2e6415da3074463772dbec91eedcf12da9984dc29d885" Nov 04 03:17:51.577411 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:17:51.577384 4405 scope.go:122] "RemoveContainer" containerID="8da84aeee90456df8fa2e6415da3074463772dbec91eedcf12da9984dc29d885" Nov 04 03:17:51.577685 ip-10-0-134-125 kubenswrapper[4405]: E1104 03:17:51.577650 4405 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8da84aeee90456df8fa2e6415da3074463772dbec91eedcf12da9984dc29d885\": container with ID starting with 8da84aeee90456df8fa2e6415da3074463772dbec91eedcf12da9984dc29d885 not found: ID does not exist" containerID="8da84aeee90456df8fa2e6415da3074463772dbec91eedcf12da9984dc29d885" Nov 04 03:17:51.577764 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:17:51.577699 4405 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8da84aeee90456df8fa2e6415da3074463772dbec91eedcf12da9984dc29d885"} err="failed to get container status \"8da84aeee90456df8fa2e6415da3074463772dbec91eedcf12da9984dc29d885\": rpc error: code = NotFound desc = could not find container \"8da84aeee90456df8fa2e6415da3074463772dbec91eedcf12da9984dc29d885\": container with ID starting with 8da84aeee90456df8fa2e6415da3074463772dbec91eedcf12da9984dc29d885 not found: ID does not exist" Nov 04 03:17:51.730843 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:17:51.730810 4405 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w2-a0bcbc-69fdc5f5c9-d4d6x"] Nov 04 03:17:51.739152 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:17:51.739122 4405 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w2-a0bcbc-69fdc5f5c9-d4d6x"] Nov 04 03:17:53.461700 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:17:53.461647 4405 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-bl5mg"] Nov 04 03:17:53.700144 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:17:53.700109 4405 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ed86675a-731f-4625-bca5-02ba4200e7ad" path="/var/lib/kubelet/pods/ed86675a-731f-4625-bca5-02ba4200e7ad/volumes" Nov 04 03:17:58.341071 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:17:58.341033 4405 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w2-a0bcbc-5b45966b8d-dgchr"] Nov 04 03:17:58.341542 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:17:58.341291 4405 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-a0bcbc-5b45966b8d-dgchr" podUID="44902e0e-53e2-443d-8e78-881319ea6fef" containerName="maas-api" containerID="cri-o://d85309cb93ec8c62aeab9ca9c8b11a86d8098192bc207876944852bcd7572204" gracePeriod=30 Nov 04 03:17:58.596102 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:17:58.596040 4405 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-a0bcbc-5b45966b8d-dgchr" Nov 04 03:17:58.604779 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:17:58.604749 4405 generic.go:356] "Generic (PLEG): container finished" podID="44902e0e-53e2-443d-8e78-881319ea6fef" containerID="d85309cb93ec8c62aeab9ca9c8b11a86d8098192bc207876944852bcd7572204" exitCode=0 Nov 04 03:17:58.604923 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:17:58.604799 4405 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-a0bcbc-5b45966b8d-dgchr" Nov 04 03:17:58.604923 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:17:58.604842 4405 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-a0bcbc-5b45966b8d-dgchr" event={"ID":"44902e0e-53e2-443d-8e78-881319ea6fef","Type":"ContainerDied","Data":"d85309cb93ec8c62aeab9ca9c8b11a86d8098192bc207876944852bcd7572204"} Nov 04 03:17:58.604923 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:17:58.604901 4405 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-a0bcbc-5b45966b8d-dgchr" event={"ID":"44902e0e-53e2-443d-8e78-881319ea6fef","Type":"ContainerDied","Data":"6526e97b42cb7f9a4031dfc3f520afd50445b2f1ae411a27d7175712c2b321e3"} Nov 04 03:17:58.605032 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:17:58.604927 4405 scope.go:122] "RemoveContainer" containerID="d85309cb93ec8c62aeab9ca9c8b11a86d8098192bc207876944852bcd7572204" Nov 04 03:17:58.615349 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:17:58.615329 4405 scope.go:122] "RemoveContainer" containerID="d85309cb93ec8c62aeab9ca9c8b11a86d8098192bc207876944852bcd7572204" Nov 04 03:17:58.615633 ip-10-0-134-125 kubenswrapper[4405]: E1104 03:17:58.615615 4405 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d85309cb93ec8c62aeab9ca9c8b11a86d8098192bc207876944852bcd7572204\": container with ID starting with d85309cb93ec8c62aeab9ca9c8b11a86d8098192bc207876944852bcd7572204 not found: ID does not exist" containerID="d85309cb93ec8c62aeab9ca9c8b11a86d8098192bc207876944852bcd7572204" Nov 04 03:17:58.615722 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:17:58.615642 4405 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d85309cb93ec8c62aeab9ca9c8b11a86d8098192bc207876944852bcd7572204"} err="failed to get container status \"d85309cb93ec8c62aeab9ca9c8b11a86d8098192bc207876944852bcd7572204\": rpc error: code = NotFound desc = could not find container \"d85309cb93ec8c62aeab9ca9c8b11a86d8098192bc207876944852bcd7572204\": container with ID starting with d85309cb93ec8c62aeab9ca9c8b11a86d8098192bc207876944852bcd7572204 not found: ID does not exist" Nov 04 03:17:58.676514 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:17:58.676483 4405 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/44902e0e-53e2-443d-8e78-881319ea6fef-metrics-certs\" (UniqueName: \"kubernetes.io/secret/44902e0e-53e2-443d-8e78-881319ea6fef-metrics-certs\") pod \"44902e0e-53e2-443d-8e78-881319ea6fef\" (UID: \"44902e0e-53e2-443d-8e78-881319ea6fef\") " Nov 04 03:17:58.676731 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:17:58.676543 4405 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/44902e0e-53e2-443d-8e78-881319ea6fef-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/44902e0e-53e2-443d-8e78-881319ea6fef-maas-api-tls\") pod \"44902e0e-53e2-443d-8e78-881319ea6fef\" (UID: \"44902e0e-53e2-443d-8e78-881319ea6fef\") " Nov 04 03:17:58.676731 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:17:58.676571 4405 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/44902e0e-53e2-443d-8e78-881319ea6fef-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/44902e0e-53e2-443d-8e78-881319ea6fef-openshift-service-ca\") pod \"44902e0e-53e2-443d-8e78-881319ea6fef\" (UID: \"44902e0e-53e2-443d-8e78-881319ea6fef\") " Nov 04 03:17:58.676731 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:17:58.676619 4405 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/44902e0e-53e2-443d-8e78-881319ea6fef-kube-api-access-tvhcv\" (UniqueName: \"kubernetes.io/projected/44902e0e-53e2-443d-8e78-881319ea6fef-kube-api-access-tvhcv\") pod \"44902e0e-53e2-443d-8e78-881319ea6fef\" (UID: \"44902e0e-53e2-443d-8e78-881319ea6fef\") " Nov 04 03:17:58.677008 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:17:58.676983 4405 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/44902e0e-53e2-443d-8e78-881319ea6fef-openshift-service-ca" pod "44902e0e-53e2-443d-8e78-881319ea6fef" (UID: "44902e0e-53e2-443d-8e78-881319ea6fef"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 03:17:58.678833 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:17:58.678808 4405 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/44902e0e-53e2-443d-8e78-881319ea6fef-metrics-certs" pod "44902e0e-53e2-443d-8e78-881319ea6fef" (UID: "44902e0e-53e2-443d-8e78-881319ea6fef"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 04 03:17:58.678935 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:17:58.678831 4405 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/44902e0e-53e2-443d-8e78-881319ea6fef-kube-api-access-tvhcv" pod "44902e0e-53e2-443d-8e78-881319ea6fef" (UID: "44902e0e-53e2-443d-8e78-881319ea6fef"). InnerVolumeSpecName "kube-api-access-tvhcv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 03:17:58.679016 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:17:58.678976 4405 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/44902e0e-53e2-443d-8e78-881319ea6fef-maas-api-tls" pod "44902e0e-53e2-443d-8e78-881319ea6fef" (UID: "44902e0e-53e2-443d-8e78-881319ea6fef"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 04 03:17:58.778024 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:17:58.777990 4405 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/44902e0e-53e2-443d-8e78-881319ea6fef-metrics-certs\") on node \"ip-10-0-134-125.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:17:58.778024 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:17:58.778018 4405 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/44902e0e-53e2-443d-8e78-881319ea6fef-maas-api-tls\") on node \"ip-10-0-134-125.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:17:58.778024 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:17:58.778028 4405 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/44902e0e-53e2-443d-8e78-881319ea6fef-openshift-service-ca\") on node \"ip-10-0-134-125.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:17:58.778266 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:17:58.778036 4405 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tvhcv\" (UniqueName: \"kubernetes.io/projected/44902e0e-53e2-443d-8e78-881319ea6fef-kube-api-access-tvhcv\") on node \"ip-10-0-134-125.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:17:59.025084 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:17:59.025047 4405 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w2-a0bcbc-5b45966b8d-dgchr"] Nov 04 03:17:59.028738 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:17:59.028706 4405 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w2-a0bcbc-5b45966b8d-dgchr"] Nov 04 03:17:59.551477 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:17:59.551446 4405 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-a0bcbc-openshift-default-65dc9d9b6c-g5hw6"] Nov 04 03:17:59.551999 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:17:59.551923 4405 cpu_manager.go:414] "containerMap: removing container" podUID="ed86675a-731f-4625-bca5-02ba4200e7ad" containerName="payload-pre-processing" Nov 04 03:17:59.551999 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:17:59.551941 4405 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ed86675a-731f-4625-bca5-02ba4200e7ad" containerName="payload-pre-processing" Nov 04 03:17:59.551999 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:17:59.551969 4405 cpu_manager.go:414] "containerMap: removing container" podUID="bac32934-8a57-4919-b435-13f367cbea3e" containerName="main" Nov 04 03:17:59.551999 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:17:59.551978 4405 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bac32934-8a57-4919-b435-13f367cbea3e" containerName="main" Nov 04 03:17:59.551999 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:17:59.551991 4405 cpu_manager.go:414] "containerMap: removing container" podUID="a061347f-23bc-43c9-9b1a-594d184ebe7c" containerName="payload-processing" Nov 04 03:17:59.551999 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:17:59.551998 4405 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a061347f-23bc-43c9-9b1a-594d184ebe7c" containerName="payload-processing" Nov 04 03:17:59.552427 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:17:59.552020 4405 cpu_manager.go:414] "containerMap: removing container" podUID="239f941f-cf45-4550-90cd-e2be9f961e07" containerName="maas-api" Nov 04 03:17:59.552427 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:17:59.552028 4405 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="239f941f-cf45-4550-90cd-e2be9f961e07" containerName="maas-api" Nov 04 03:17:59.552427 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:17:59.552038 4405 cpu_manager.go:414] "containerMap: removing container" podUID="3d80d037-3e20-4898-855b-86fbcf2bb9aa" containerName="maas-api" Nov 04 03:17:59.552427 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:17:59.552045 4405 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3d80d037-3e20-4898-855b-86fbcf2bb9aa" containerName="maas-api" Nov 04 03:17:59.552427 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:17:59.552059 4405 cpu_manager.go:414] "containerMap: removing container" podUID="9a57520d-dae3-45e9-8558-fa824bd746cd" containerName="payload-pre-processing" Nov 04 03:17:59.552427 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:17:59.552063 4405 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9a57520d-dae3-45e9-8558-fa824bd746cd" containerName="payload-pre-processing" Nov 04 03:17:59.552427 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:17:59.552071 4405 cpu_manager.go:414] "containerMap: removing container" podUID="44902e0e-53e2-443d-8e78-881319ea6fef" containerName="maas-api" Nov 04 03:17:59.552427 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:17:59.552075 4405 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="44902e0e-53e2-443d-8e78-881319ea6fef" containerName="maas-api" Nov 04 03:17:59.552427 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:17:59.552082 4405 cpu_manager.go:414] "containerMap: removing container" podUID="c1ed6380-62fc-4239-81a7-b1368b9a529b" containerName="main" Nov 04 03:17:59.552427 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:17:59.552086 4405 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c1ed6380-62fc-4239-81a7-b1368b9a529b" containerName="main" Nov 04 03:17:59.552427 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:17:59.552094 4405 cpu_manager.go:414] "containerMap: removing container" podUID="73c5ca67-bc51-492d-b055-0cbaf3a85b07" containerName="maas-api" Nov 04 03:17:59.552427 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:17:59.552098 4405 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="73c5ca67-bc51-492d-b055-0cbaf3a85b07" containerName="maas-api" Nov 04 03:17:59.552427 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:17:59.552105 4405 cpu_manager.go:414] "containerMap: removing container" podUID="a6f009b4-0c0b-4652-812f-a7d3333f3880" containerName="payload-processing" Nov 04 03:17:59.552427 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:17:59.552109 4405 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a6f009b4-0c0b-4652-812f-a7d3333f3880" containerName="payload-processing" Nov 04 03:17:59.552427 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:17:59.552122 4405 cpu_manager.go:414] "containerMap: removing container" podUID="a77157ba-0aed-46cf-90de-ffc15074a50f" containerName="istio-proxy" Nov 04 03:17:59.552427 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:17:59.552129 4405 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a77157ba-0aed-46cf-90de-ffc15074a50f" containerName="istio-proxy" Nov 04 03:17:59.552427 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:17:59.552140 4405 cpu_manager.go:414] "containerMap: removing container" podUID="3eb56d77-29c0-4a72-863d-354d2cb764a0" containerName="payload-pre-processing" Nov 04 03:17:59.552427 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:17:59.552145 4405 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3eb56d77-29c0-4a72-863d-354d2cb764a0" containerName="payload-pre-processing" Nov 04 03:17:59.552427 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:17:59.552152 4405 cpu_manager.go:414] "containerMap: removing container" podUID="423b62dd-792f-442c-b1ac-3248072565e8" containerName="istio-proxy" Nov 04 03:17:59.552427 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:17:59.552157 4405 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="423b62dd-792f-442c-b1ac-3248072565e8" containerName="istio-proxy" Nov 04 03:17:59.552427 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:17:59.552163 4405 cpu_manager.go:414] "containerMap: removing container" podUID="eeec2ff5-38c0-4e67-aeed-9f0baa9fa1d1" containerName="istio-proxy" Nov 04 03:17:59.552427 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:17:59.552169 4405 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="eeec2ff5-38c0-4e67-aeed-9f0baa9fa1d1" containerName="istio-proxy" Nov 04 03:17:59.552427 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:17:59.552179 4405 cpu_manager.go:414] "containerMap: removing container" podUID="a77157ba-0aed-46cf-90de-ffc15074a50f" containerName="istio-proxy" Nov 04 03:17:59.552427 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:17:59.552183 4405 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a77157ba-0aed-46cf-90de-ffc15074a50f" containerName="istio-proxy" Nov 04 03:17:59.552427 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:17:59.552192 4405 cpu_manager.go:414] "containerMap: removing container" podUID="33b620c1-389c-48b4-b277-90252018c4b6" containerName="main" Nov 04 03:17:59.552427 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:17:59.552197 4405 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="33b620c1-389c-48b4-b277-90252018c4b6" containerName="main" Nov 04 03:17:59.552427 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:17:59.552259 4405 memory_manager.go:365] "RemoveStaleState removing state" podUID="a6f009b4-0c0b-4652-812f-a7d3333f3880" containerName="payload-processing" Nov 04 03:17:59.552427 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:17:59.552266 4405 memory_manager.go:365] "RemoveStaleState removing state" podUID="a77157ba-0aed-46cf-90de-ffc15074a50f" containerName="istio-proxy" Nov 04 03:17:59.552427 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:17:59.552274 4405 memory_manager.go:365] "RemoveStaleState removing state" podUID="3eb56d77-29c0-4a72-863d-354d2cb764a0" containerName="payload-pre-processing" Nov 04 03:17:59.552427 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:17:59.552280 4405 memory_manager.go:365] "RemoveStaleState removing state" podUID="423b62dd-792f-442c-b1ac-3248072565e8" containerName="istio-proxy" Nov 04 03:17:59.552427 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:17:59.552285 4405 memory_manager.go:365] "RemoveStaleState removing state" podUID="eeec2ff5-38c0-4e67-aeed-9f0baa9fa1d1" containerName="istio-proxy" Nov 04 03:17:59.552427 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:17:59.552292 4405 memory_manager.go:365] "RemoveStaleState removing state" podUID="a77157ba-0aed-46cf-90de-ffc15074a50f" containerName="istio-proxy" Nov 04 03:17:59.552427 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:17:59.552300 4405 memory_manager.go:365] "RemoveStaleState removing state" podUID="33b620c1-389c-48b4-b277-90252018c4b6" containerName="main" Nov 04 03:17:59.552427 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:17:59.552306 4405 memory_manager.go:365] "RemoveStaleState removing state" podUID="ed86675a-731f-4625-bca5-02ba4200e7ad" containerName="payload-pre-processing" Nov 04 03:17:59.552427 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:17:59.552313 4405 memory_manager.go:365] "RemoveStaleState removing state" podUID="bac32934-8a57-4919-b435-13f367cbea3e" containerName="main" Nov 04 03:17:59.552427 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:17:59.552318 4405 memory_manager.go:365] "RemoveStaleState removing state" podUID="a061347f-23bc-43c9-9b1a-594d184ebe7c" containerName="payload-processing" Nov 04 03:17:59.552427 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:17:59.552324 4405 memory_manager.go:365] "RemoveStaleState removing state" podUID="239f941f-cf45-4550-90cd-e2be9f961e07" containerName="maas-api" Nov 04 03:17:59.552427 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:17:59.552330 4405 memory_manager.go:365] "RemoveStaleState removing state" podUID="3d80d037-3e20-4898-855b-86fbcf2bb9aa" containerName="maas-api" Nov 04 03:17:59.552427 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:17:59.552335 4405 memory_manager.go:365] "RemoveStaleState removing state" podUID="9a57520d-dae3-45e9-8558-fa824bd746cd" containerName="payload-pre-processing" Nov 04 03:17:59.552427 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:17:59.552343 4405 memory_manager.go:365] "RemoveStaleState removing state" podUID="44902e0e-53e2-443d-8e78-881319ea6fef" containerName="maas-api" Nov 04 03:17:59.552427 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:17:59.552350 4405 memory_manager.go:365] "RemoveStaleState removing state" podUID="c1ed6380-62fc-4239-81a7-b1368b9a529b" containerName="main" Nov 04 03:17:59.552427 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:17:59.552357 4405 memory_manager.go:365] "RemoveStaleState removing state" podUID="73c5ca67-bc51-492d-b055-0cbaf3a85b07" containerName="maas-api" Nov 04 03:17:59.556764 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:17:59.556741 4405 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-a0bcbc-openshift-default-65dc9d9b6c-g5hw6" Nov 04 03:17:59.585912 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:17:59.585878 4405 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d0362f05-c556-4c71-9b54-b360072d0758-credential-socket\") pod \"e2e-worker-w2-a0bcbc-openshift-default-65dc9d9b6c-g5hw6\" (UID: \"d0362f05-c556-4c71-9b54-b360072d0758\") " pod="openshift-ingress/e2e-worker-w2-a0bcbc-openshift-default-65dc9d9b6c-g5hw6" Nov 04 03:17:59.586074 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:17:59.585919 4405 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d0362f05-c556-4c71-9b54-b360072d0758-istio-envoy\") pod \"e2e-worker-w2-a0bcbc-openshift-default-65dc9d9b6c-g5hw6\" (UID: \"d0362f05-c556-4c71-9b54-b360072d0758\") " pod="openshift-ingress/e2e-worker-w2-a0bcbc-openshift-default-65dc9d9b6c-g5hw6" Nov 04 03:17:59.586074 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:17:59.585941 4405 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cxd6x\" (UniqueName: \"kubernetes.io/projected/d0362f05-c556-4c71-9b54-b360072d0758-kube-api-access-cxd6x\") pod \"e2e-worker-w2-a0bcbc-openshift-default-65dc9d9b6c-g5hw6\" (UID: \"d0362f05-c556-4c71-9b54-b360072d0758\") " pod="openshift-ingress/e2e-worker-w2-a0bcbc-openshift-default-65dc9d9b6c-g5hw6" Nov 04 03:17:59.586193 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:17:59.586068 4405 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d0362f05-c556-4c71-9b54-b360072d0758-istio-token\") pod \"e2e-worker-w2-a0bcbc-openshift-default-65dc9d9b6c-g5hw6\" (UID: \"d0362f05-c556-4c71-9b54-b360072d0758\") " pod="openshift-ingress/e2e-worker-w2-a0bcbc-openshift-default-65dc9d9b6c-g5hw6" Nov 04 03:17:59.586193 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:17:59.586163 4405 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d0362f05-c556-4c71-9b54-b360072d0758-workload-certs\") pod \"e2e-worker-w2-a0bcbc-openshift-default-65dc9d9b6c-g5hw6\" (UID: \"d0362f05-c556-4c71-9b54-b360072d0758\") " pod="openshift-ingress/e2e-worker-w2-a0bcbc-openshift-default-65dc9d9b6c-g5hw6" Nov 04 03:17:59.586293 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:17:59.586220 4405 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d0362f05-c556-4c71-9b54-b360072d0758-istio-podinfo\") pod \"e2e-worker-w2-a0bcbc-openshift-default-65dc9d9b6c-g5hw6\" (UID: \"d0362f05-c556-4c71-9b54-b360072d0758\") " pod="openshift-ingress/e2e-worker-w2-a0bcbc-openshift-default-65dc9d9b6c-g5hw6" Nov 04 03:17:59.586352 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:17:59.586333 4405 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d0362f05-c556-4c71-9b54-b360072d0758-istiod-ca-cert\") pod \"e2e-worker-w2-a0bcbc-openshift-default-65dc9d9b6c-g5hw6\" (UID: \"d0362f05-c556-4c71-9b54-b360072d0758\") " pod="openshift-ingress/e2e-worker-w2-a0bcbc-openshift-default-65dc9d9b6c-g5hw6" Nov 04 03:17:59.586414 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:17:59.586399 4405 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d0362f05-c556-4c71-9b54-b360072d0758-workload-socket\") pod \"e2e-worker-w2-a0bcbc-openshift-default-65dc9d9b6c-g5hw6\" (UID: \"d0362f05-c556-4c71-9b54-b360072d0758\") " pod="openshift-ingress/e2e-worker-w2-a0bcbc-openshift-default-65dc9d9b6c-g5hw6" Nov 04 03:17:59.586457 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:17:59.586433 4405 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d0362f05-c556-4c71-9b54-b360072d0758-istio-data\") pod \"e2e-worker-w2-a0bcbc-openshift-default-65dc9d9b6c-g5hw6\" (UID: \"d0362f05-c556-4c71-9b54-b360072d0758\") " pod="openshift-ingress/e2e-worker-w2-a0bcbc-openshift-default-65dc9d9b6c-g5hw6" Nov 04 03:17:59.608986 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:17:59.608955 4405 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w2-a0bcbc-openshift-default-dockercfg-z8ns2\"" Nov 04 03:17:59.660145 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:17:59.660099 4405 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-a0bcbc-openshift-default-65dc9d9b6c-g5hw6"] Nov 04 03:17:59.687527 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:17:59.687486 4405 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d0362f05-c556-4c71-9b54-b360072d0758-istio-token\") pod \"e2e-worker-w2-a0bcbc-openshift-default-65dc9d9b6c-g5hw6\" (UID: \"d0362f05-c556-4c71-9b54-b360072d0758\") " pod="openshift-ingress/e2e-worker-w2-a0bcbc-openshift-default-65dc9d9b6c-g5hw6" Nov 04 03:17:59.687769 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:17:59.687541 4405 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d0362f05-c556-4c71-9b54-b360072d0758-workload-certs\") pod \"e2e-worker-w2-a0bcbc-openshift-default-65dc9d9b6c-g5hw6\" (UID: \"d0362f05-c556-4c71-9b54-b360072d0758\") " pod="openshift-ingress/e2e-worker-w2-a0bcbc-openshift-default-65dc9d9b6c-g5hw6" Nov 04 03:17:59.687769 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:17:59.687586 4405 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d0362f05-c556-4c71-9b54-b360072d0758-istio-podinfo\") pod \"e2e-worker-w2-a0bcbc-openshift-default-65dc9d9b6c-g5hw6\" (UID: \"d0362f05-c556-4c71-9b54-b360072d0758\") " pod="openshift-ingress/e2e-worker-w2-a0bcbc-openshift-default-65dc9d9b6c-g5hw6" Nov 04 03:17:59.687769 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:17:59.687653 4405 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d0362f05-c556-4c71-9b54-b360072d0758-istiod-ca-cert\") pod \"e2e-worker-w2-a0bcbc-openshift-default-65dc9d9b6c-g5hw6\" (UID: \"d0362f05-c556-4c71-9b54-b360072d0758\") " pod="openshift-ingress/e2e-worker-w2-a0bcbc-openshift-default-65dc9d9b6c-g5hw6" Nov 04 03:17:59.687769 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:17:59.687747 4405 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d0362f05-c556-4c71-9b54-b360072d0758-workload-socket\") pod \"e2e-worker-w2-a0bcbc-openshift-default-65dc9d9b6c-g5hw6\" (UID: \"d0362f05-c556-4c71-9b54-b360072d0758\") " pod="openshift-ingress/e2e-worker-w2-a0bcbc-openshift-default-65dc9d9b6c-g5hw6" Nov 04 03:17:59.688023 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:17:59.687776 4405 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d0362f05-c556-4c71-9b54-b360072d0758-istio-data\") pod \"e2e-worker-w2-a0bcbc-openshift-default-65dc9d9b6c-g5hw6\" (UID: \"d0362f05-c556-4c71-9b54-b360072d0758\") " pod="openshift-ingress/e2e-worker-w2-a0bcbc-openshift-default-65dc9d9b6c-g5hw6" Nov 04 03:17:59.688023 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:17:59.687803 4405 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d0362f05-c556-4c71-9b54-b360072d0758-credential-socket\") pod \"e2e-worker-w2-a0bcbc-openshift-default-65dc9d9b6c-g5hw6\" (UID: \"d0362f05-c556-4c71-9b54-b360072d0758\") " pod="openshift-ingress/e2e-worker-w2-a0bcbc-openshift-default-65dc9d9b6c-g5hw6" Nov 04 03:17:59.688023 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:17:59.687830 4405 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d0362f05-c556-4c71-9b54-b360072d0758-istio-envoy\") pod \"e2e-worker-w2-a0bcbc-openshift-default-65dc9d9b6c-g5hw6\" (UID: \"d0362f05-c556-4c71-9b54-b360072d0758\") " pod="openshift-ingress/e2e-worker-w2-a0bcbc-openshift-default-65dc9d9b6c-g5hw6" Nov 04 03:17:59.688023 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:17:59.687883 4405 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cxd6x\" (UniqueName: \"kubernetes.io/projected/d0362f05-c556-4c71-9b54-b360072d0758-kube-api-access-cxd6x\") pod \"e2e-worker-w2-a0bcbc-openshift-default-65dc9d9b6c-g5hw6\" (UID: \"d0362f05-c556-4c71-9b54-b360072d0758\") " pod="openshift-ingress/e2e-worker-w2-a0bcbc-openshift-default-65dc9d9b6c-g5hw6" Nov 04 03:17:59.688226 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:17:59.688066 4405 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d0362f05-c556-4c71-9b54-b360072d0758-workload-certs\") pod \"e2e-worker-w2-a0bcbc-openshift-default-65dc9d9b6c-g5hw6\" (UID: \"d0362f05-c556-4c71-9b54-b360072d0758\") " pod="openshift-ingress/e2e-worker-w2-a0bcbc-openshift-default-65dc9d9b6c-g5hw6" Nov 04 03:17:59.688226 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:17:59.688099 4405 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d0362f05-c556-4c71-9b54-b360072d0758-workload-socket\") pod \"e2e-worker-w2-a0bcbc-openshift-default-65dc9d9b6c-g5hw6\" (UID: \"d0362f05-c556-4c71-9b54-b360072d0758\") " pod="openshift-ingress/e2e-worker-w2-a0bcbc-openshift-default-65dc9d9b6c-g5hw6" Nov 04 03:17:59.688360 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:17:59.688336 4405 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d0362f05-c556-4c71-9b54-b360072d0758-credential-socket\") pod \"e2e-worker-w2-a0bcbc-openshift-default-65dc9d9b6c-g5hw6\" (UID: \"d0362f05-c556-4c71-9b54-b360072d0758\") " pod="openshift-ingress/e2e-worker-w2-a0bcbc-openshift-default-65dc9d9b6c-g5hw6" Nov 04 03:17:59.688360 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:17:59.688347 4405 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d0362f05-c556-4c71-9b54-b360072d0758-istio-data\") pod \"e2e-worker-w2-a0bcbc-openshift-default-65dc9d9b6c-g5hw6\" (UID: \"d0362f05-c556-4c71-9b54-b360072d0758\") " pod="openshift-ingress/e2e-worker-w2-a0bcbc-openshift-default-65dc9d9b6c-g5hw6" Nov 04 03:17:59.688690 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:17:59.688564 4405 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d0362f05-c556-4c71-9b54-b360072d0758-istiod-ca-cert\") pod \"e2e-worker-w2-a0bcbc-openshift-default-65dc9d9b6c-g5hw6\" (UID: \"d0362f05-c556-4c71-9b54-b360072d0758\") " pod="openshift-ingress/e2e-worker-w2-a0bcbc-openshift-default-65dc9d9b6c-g5hw6" Nov 04 03:17:59.690252 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:17:59.690226 4405 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d0362f05-c556-4c71-9b54-b360072d0758-istio-envoy\") pod \"e2e-worker-w2-a0bcbc-openshift-default-65dc9d9b6c-g5hw6\" (UID: \"d0362f05-c556-4c71-9b54-b360072d0758\") " pod="openshift-ingress/e2e-worker-w2-a0bcbc-openshift-default-65dc9d9b6c-g5hw6" Nov 04 03:17:59.690377 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:17:59.690358 4405 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d0362f05-c556-4c71-9b54-b360072d0758-istio-podinfo\") pod \"e2e-worker-w2-a0bcbc-openshift-default-65dc9d9b6c-g5hw6\" (UID: \"d0362f05-c556-4c71-9b54-b360072d0758\") " pod="openshift-ingress/e2e-worker-w2-a0bcbc-openshift-default-65dc9d9b6c-g5hw6" Nov 04 03:17:59.702813 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:17:59.702785 4405 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="44902e0e-53e2-443d-8e78-881319ea6fef" path="/var/lib/kubelet/pods/44902e0e-53e2-443d-8e78-881319ea6fef/volumes" Nov 04 03:17:59.748738 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:17:59.748704 4405 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d0362f05-c556-4c71-9b54-b360072d0758-istio-token\") pod \"e2e-worker-w2-a0bcbc-openshift-default-65dc9d9b6c-g5hw6\" (UID: \"d0362f05-c556-4c71-9b54-b360072d0758\") " pod="openshift-ingress/e2e-worker-w2-a0bcbc-openshift-default-65dc9d9b6c-g5hw6" Nov 04 03:17:59.749069 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:17:59.749044 4405 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cxd6x\" (UniqueName: \"kubernetes.io/projected/d0362f05-c556-4c71-9b54-b360072d0758-kube-api-access-cxd6x\") pod \"e2e-worker-w2-a0bcbc-openshift-default-65dc9d9b6c-g5hw6\" (UID: \"d0362f05-c556-4c71-9b54-b360072d0758\") " pod="openshift-ingress/e2e-worker-w2-a0bcbc-openshift-default-65dc9d9b6c-g5hw6" Nov 04 03:17:59.872579 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:17:59.872537 4405 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-a0bcbc-openshift-default-65dc9d9b6c-g5hw6" Nov 04 03:18:00.309014 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:18:00.308935 4405 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-a0bcbc-openshift-default-65dc9d9b6c-g5hw6"] Nov 04 03:18:00.346892 ip-10-0-134-125 kubenswrapper[4405]: W1104 03:18:00.346862 4405 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd0362f05_c556_4c71_9b54_b360072d0758.slice/crio-80846d84e267028032ac689318c25d1ab60baf25272b03cb41e3b9785873cf66 WatchSource:0}: Error finding container 80846d84e267028032ac689318c25d1ab60baf25272b03cb41e3b9785873cf66: Status 404 returned error can't find the container with id 80846d84e267028032ac689318c25d1ab60baf25272b03cb41e3b9785873cf66 Nov 04 03:18:00.349434 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:18:00.349397 4405 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 04 03:18:00.349530 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:18:00.349478 4405 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 04 03:18:00.349530 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:18:00.349508 4405 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 04 03:18:00.618972 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:18:00.618938 4405 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-a0bcbc-openshift-default-65dc9d9b6c-g5hw6" event={"ID":"d0362f05-c556-4c71-9b54-b360072d0758","Type":"ContainerStarted","Data":"30c4684c5addc51fb80dee95aba95b1bf3cbeef5c1127dfc4b592b8892857c6e"} Nov 04 03:18:00.619490 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:18:00.618994 4405 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-a0bcbc-openshift-default-65dc9d9b6c-g5hw6" event={"ID":"d0362f05-c556-4c71-9b54-b360072d0758","Type":"ContainerStarted","Data":"80846d84e267028032ac689318c25d1ab60baf25272b03cb41e3b9785873cf66"} Nov 04 03:18:00.741183 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:18:00.741119 4405 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-a0bcbc-openshift-default-65dc9d9b6c-g5hw6" podStartSLOduration=1.741101716 podStartE2EDuration="1.741101716s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 03:17:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 03:18:00.738955862 +0000 UTC m=+2727.169655984" watchObservedRunningTime="2026-11-04 03:18:00.741101716 +0000 UTC m=+2727.171801836" Nov 04 03:18:00.872766 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:18:00.872638 4405 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-a0bcbc-openshift-default-65dc9d9b6c-g5hw6" Nov 04 03:18:01.874395 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:18:01.874352 4405 patch_prober.go:29] interesting pod/e2e-worker-w2-a0bcbc-openshift-default-65dc9d9b6c-g5hw6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.136:15021/healthz/ready\": dial tcp 10.134.0.136:15021: connect: connection refused" start-of-body= Nov 04 03:18:01.874847 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:18:01.874415 4405 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-a0bcbc-openshift-default-65dc9d9b6c-g5hw6" podUID="d0362f05-c556-4c71-9b54-b360072d0758" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.136:15021/healthz/ready\": dial tcp 10.134.0.136:15021: connect: connection refused" Nov 04 03:18:02.873450 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:18:02.873413 4405 patch_prober.go:29] interesting pod/e2e-worker-w2-a0bcbc-openshift-default-65dc9d9b6c-g5hw6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.136:15021/healthz/ready\": dial tcp 10.134.0.136:15021: connect: connection refused" start-of-body= Nov 04 03:18:02.873642 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:18:02.873470 4405 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-a0bcbc-openshift-default-65dc9d9b6c-g5hw6" podUID="d0362f05-c556-4c71-9b54-b360072d0758" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.136:15021/healthz/ready\": dial tcp 10.134.0.136:15021: connect: connection refused" Nov 04 03:18:03.873855 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:18:03.873814 4405 patch_prober.go:29] interesting pod/e2e-worker-w2-a0bcbc-openshift-default-65dc9d9b6c-g5hw6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.136:15021/healthz/ready\": dial tcp 10.134.0.136:15021: connect: connection refused" start-of-body= Nov 04 03:18:03.874274 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:18:03.873884 4405 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-a0bcbc-openshift-default-65dc9d9b6c-g5hw6" podUID="d0362f05-c556-4c71-9b54-b360072d0758" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.136:15021/healthz/ready\": dial tcp 10.134.0.136:15021: connect: connection refused" Nov 04 03:18:04.873161 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:18:04.873122 4405 patch_prober.go:29] interesting pod/e2e-worker-w2-a0bcbc-openshift-default-65dc9d9b6c-g5hw6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.136:15021/healthz/ready\": dial tcp 10.134.0.136:15021: connect: connection refused" start-of-body= Nov 04 03:18:04.873366 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:18:04.873182 4405 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-a0bcbc-openshift-default-65dc9d9b6c-g5hw6" podUID="d0362f05-c556-4c71-9b54-b360072d0758" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.136:15021/healthz/ready\": dial tcp 10.134.0.136:15021: connect: connection refused" Nov 04 03:18:05.873266 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:18:05.873230 4405 patch_prober.go:29] interesting pod/e2e-worker-w2-a0bcbc-openshift-default-65dc9d9b6c-g5hw6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.136:15021/healthz/ready\": dial tcp 10.134.0.136:15021: connect: connection refused" start-of-body= Nov 04 03:18:05.873711 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:18:05.873288 4405 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-a0bcbc-openshift-default-65dc9d9b6c-g5hw6" podUID="d0362f05-c556-4c71-9b54-b360072d0758" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.136:15021/healthz/ready\": dial tcp 10.134.0.136:15021: connect: connection refused" Nov 04 03:18:06.075761 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:18:06.075723 4405 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-072ece-openshift-default-78b784fd8c-k2vfq"] Nov 04 03:18:06.076007 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:18:06.075983 4405 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w4-072ece-openshift-default-78b784fd8c-k2vfq" podUID="652d018b-8d07-49ae-b9fb-a87cfc70b0c8" containerName="istio-proxy" containerID="cri-o://9c2f7cc08dbbb4aa2b4f90dd96c5db012ffa85b0590ed3c70e56029b97f22278" gracePeriod=30 Nov 04 03:18:06.873075 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:18:06.873031 4405 patch_prober.go:29] interesting pod/e2e-worker-w2-a0bcbc-openshift-default-65dc9d9b6c-g5hw6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.136:15021/healthz/ready\": dial tcp 10.134.0.136:15021: connect: connection refused" start-of-body= Nov 04 03:18:06.873295 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:18:06.873104 4405 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-a0bcbc-openshift-default-65dc9d9b6c-g5hw6" podUID="d0362f05-c556-4c71-9b54-b360072d0758" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.136:15021/healthz/ready\": dial tcp 10.134.0.136:15021: connect: connection refused" Nov 04 03:18:06.887574 ip-10-0-134-125 kubenswrapper[4405]: E1104 03:18:06.887535 4405 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5f2vb" containerName="extractor" Nov 04 03:18:08.165734 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:18:08.165700 4405 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-a0bcbc-openshift-default-65dc9d9b6c-g5hw6" Nov 04 03:18:08.166221 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:18:08.166054 4405 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-a0bcbc-openshift-default-65dc9d9b6c-g5hw6" Nov 04 03:18:08.167165 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:18:08.167143 4405 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-a0bcbc-openshift-default-65dc9d9b6c-g5hw6" Nov 04 03:18:11.334740 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:18:11.334715 4405 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-072ece-openshift-default-78b784fd8c-k2vfq" Nov 04 03:18:11.432273 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:18:11.432172 4405 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/652d018b-8d07-49ae-b9fb-a87cfc70b0c8-istio-token\" (UniqueName: \"kubernetes.io/projected/652d018b-8d07-49ae-b9fb-a87cfc70b0c8-istio-token\") pod \"652d018b-8d07-49ae-b9fb-a87cfc70b0c8\" (UID: \"652d018b-8d07-49ae-b9fb-a87cfc70b0c8\") " Nov 04 03:18:11.432273 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:18:11.432235 4405 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/652d018b-8d07-49ae-b9fb-a87cfc70b0c8-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/652d018b-8d07-49ae-b9fb-a87cfc70b0c8-istio-envoy\") pod \"652d018b-8d07-49ae-b9fb-a87cfc70b0c8\" (UID: \"652d018b-8d07-49ae-b9fb-a87cfc70b0c8\") " Nov 04 03:18:11.432273 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:18:11.432263 4405 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/652d018b-8d07-49ae-b9fb-a87cfc70b0c8-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/652d018b-8d07-49ae-b9fb-a87cfc70b0c8-workload-socket\") pod \"652d018b-8d07-49ae-b9fb-a87cfc70b0c8\" (UID: \"652d018b-8d07-49ae-b9fb-a87cfc70b0c8\") " Nov 04 03:18:11.432565 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:18:11.432315 4405 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/652d018b-8d07-49ae-b9fb-a87cfc70b0c8-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/652d018b-8d07-49ae-b9fb-a87cfc70b0c8-workload-certs\") pod \"652d018b-8d07-49ae-b9fb-a87cfc70b0c8\" (UID: \"652d018b-8d07-49ae-b9fb-a87cfc70b0c8\") " Nov 04 03:18:11.432565 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:18:11.432335 4405 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/652d018b-8d07-49ae-b9fb-a87cfc70b0c8-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/652d018b-8d07-49ae-b9fb-a87cfc70b0c8-istio-data\") pod \"652d018b-8d07-49ae-b9fb-a87cfc70b0c8\" (UID: \"652d018b-8d07-49ae-b9fb-a87cfc70b0c8\") " Nov 04 03:18:11.432565 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:18:11.432407 4405 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/652d018b-8d07-49ae-b9fb-a87cfc70b0c8-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/652d018b-8d07-49ae-b9fb-a87cfc70b0c8-credential-socket\") pod \"652d018b-8d07-49ae-b9fb-a87cfc70b0c8\" (UID: \"652d018b-8d07-49ae-b9fb-a87cfc70b0c8\") " Nov 04 03:18:11.432565 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:18:11.432466 4405 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/652d018b-8d07-49ae-b9fb-a87cfc70b0c8-kube-api-access-2h5lc\" (UniqueName: \"kubernetes.io/projected/652d018b-8d07-49ae-b9fb-a87cfc70b0c8-kube-api-access-2h5lc\") pod \"652d018b-8d07-49ae-b9fb-a87cfc70b0c8\" (UID: \"652d018b-8d07-49ae-b9fb-a87cfc70b0c8\") " Nov 04 03:18:11.432565 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:18:11.432503 4405 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/652d018b-8d07-49ae-b9fb-a87cfc70b0c8-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/652d018b-8d07-49ae-b9fb-a87cfc70b0c8-istio-podinfo\") pod \"652d018b-8d07-49ae-b9fb-a87cfc70b0c8\" (UID: \"652d018b-8d07-49ae-b9fb-a87cfc70b0c8\") " Nov 04 03:18:11.432565 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:18:11.432530 4405 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/652d018b-8d07-49ae-b9fb-a87cfc70b0c8-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/652d018b-8d07-49ae-b9fb-a87cfc70b0c8-istiod-ca-cert\") pod \"652d018b-8d07-49ae-b9fb-a87cfc70b0c8\" (UID: \"652d018b-8d07-49ae-b9fb-a87cfc70b0c8\") " Nov 04 03:18:11.432565 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:18:11.432549 4405 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/652d018b-8d07-49ae-b9fb-a87cfc70b0c8-workload-socket" pod "652d018b-8d07-49ae-b9fb-a87cfc70b0c8" (UID: "652d018b-8d07-49ae-b9fb-a87cfc70b0c8"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 03:18:11.433004 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:18:11.432743 4405 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/652d018b-8d07-49ae-b9fb-a87cfc70b0c8-workload-certs" pod "652d018b-8d07-49ae-b9fb-a87cfc70b0c8" (UID: "652d018b-8d07-49ae-b9fb-a87cfc70b0c8"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 03:18:11.433004 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:18:11.432763 4405 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/652d018b-8d07-49ae-b9fb-a87cfc70b0c8-istio-data" pod "652d018b-8d07-49ae-b9fb-a87cfc70b0c8" (UID: "652d018b-8d07-49ae-b9fb-a87cfc70b0c8"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 03:18:11.433004 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:18:11.432989 4405 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/652d018b-8d07-49ae-b9fb-a87cfc70b0c8-credential-socket" pod "652d018b-8d07-49ae-b9fb-a87cfc70b0c8" (UID: "652d018b-8d07-49ae-b9fb-a87cfc70b0c8"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 03:18:11.433159 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:18:11.433065 4405 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/652d018b-8d07-49ae-b9fb-a87cfc70b0c8-istiod-ca-cert" pod "652d018b-8d07-49ae-b9fb-a87cfc70b0c8" (UID: "652d018b-8d07-49ae-b9fb-a87cfc70b0c8"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 03:18:11.433257 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:18:11.433206 4405 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/652d018b-8d07-49ae-b9fb-a87cfc70b0c8-workload-socket\") on node \"ip-10-0-134-125.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:18:11.433369 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:18:11.433263 4405 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/652d018b-8d07-49ae-b9fb-a87cfc70b0c8-workload-certs\") on node \"ip-10-0-134-125.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:18:11.433369 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:18:11.433279 4405 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/652d018b-8d07-49ae-b9fb-a87cfc70b0c8-istio-data\") on node \"ip-10-0-134-125.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:18:11.435073 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:18:11.435019 4405 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/652d018b-8d07-49ae-b9fb-a87cfc70b0c8-istio-envoy" pod "652d018b-8d07-49ae-b9fb-a87cfc70b0c8" (UID: "652d018b-8d07-49ae-b9fb-a87cfc70b0c8"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 03:18:11.435073 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:18:11.435032 4405 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/652d018b-8d07-49ae-b9fb-a87cfc70b0c8-istio-podinfo" pod "652d018b-8d07-49ae-b9fb-a87cfc70b0c8" (UID: "652d018b-8d07-49ae-b9fb-a87cfc70b0c8"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 04 03:18:11.435361 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:18:11.435331 4405 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/652d018b-8d07-49ae-b9fb-a87cfc70b0c8-istio-token" pod "652d018b-8d07-49ae-b9fb-a87cfc70b0c8" (UID: "652d018b-8d07-49ae-b9fb-a87cfc70b0c8"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 03:18:11.435470 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:18:11.435333 4405 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/652d018b-8d07-49ae-b9fb-a87cfc70b0c8-kube-api-access-2h5lc" pod "652d018b-8d07-49ae-b9fb-a87cfc70b0c8" (UID: "652d018b-8d07-49ae-b9fb-a87cfc70b0c8"). InnerVolumeSpecName "kube-api-access-2h5lc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 03:18:11.534560 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:18:11.534521 4405 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/652d018b-8d07-49ae-b9fb-a87cfc70b0c8-credential-socket\") on node \"ip-10-0-134-125.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:18:11.534560 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:18:11.534554 4405 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2h5lc\" (UniqueName: \"kubernetes.io/projected/652d018b-8d07-49ae-b9fb-a87cfc70b0c8-kube-api-access-2h5lc\") on node \"ip-10-0-134-125.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:18:11.534560 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:18:11.534565 4405 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/652d018b-8d07-49ae-b9fb-a87cfc70b0c8-istio-podinfo\") on node \"ip-10-0-134-125.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:18:11.534950 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:18:11.534574 4405 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/652d018b-8d07-49ae-b9fb-a87cfc70b0c8-istiod-ca-cert\") on node \"ip-10-0-134-125.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:18:11.534950 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:18:11.534582 4405 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/652d018b-8d07-49ae-b9fb-a87cfc70b0c8-istio-token\") on node \"ip-10-0-134-125.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:18:11.534950 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:18:11.534594 4405 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/652d018b-8d07-49ae-b9fb-a87cfc70b0c8-istio-envoy\") on node \"ip-10-0-134-125.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:18:11.694239 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:18:11.694135 4405 generic.go:356] "Generic (PLEG): container finished" podID="652d018b-8d07-49ae-b9fb-a87cfc70b0c8" containerID="9c2f7cc08dbbb4aa2b4f90dd96c5db012ffa85b0590ed3c70e56029b97f22278" exitCode=0 Nov 04 03:18:11.694420 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:18:11.694260 4405 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-072ece-openshift-default-78b784fd8c-k2vfq" Nov 04 03:18:11.699799 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:18:11.699769 4405 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-072ece-openshift-default-78b784fd8c-k2vfq" event={"ID":"652d018b-8d07-49ae-b9fb-a87cfc70b0c8","Type":"ContainerDied","Data":"9c2f7cc08dbbb4aa2b4f90dd96c5db012ffa85b0590ed3c70e56029b97f22278"} Nov 04 03:18:11.699950 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:18:11.699817 4405 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-072ece-openshift-default-78b784fd8c-k2vfq" event={"ID":"652d018b-8d07-49ae-b9fb-a87cfc70b0c8","Type":"ContainerDied","Data":"992aab1001ece4f8d6e5a5e38cf6a111e232936fb9f8c812606eb7135d361d71"} Nov 04 03:18:11.699950 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:18:11.699843 4405 scope.go:122] "RemoveContainer" containerID="9c2f7cc08dbbb4aa2b4f90dd96c5db012ffa85b0590ed3c70e56029b97f22278" Nov 04 03:18:11.709482 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:18:11.709458 4405 scope.go:122] "RemoveContainer" containerID="a30b7638525ea4ce26e47f3f8e99113b4d9df664c679a6e02855af22859fe884" Nov 04 03:18:11.720174 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:18:11.720149 4405 scope.go:122] "RemoveContainer" containerID="9c2f7cc08dbbb4aa2b4f90dd96c5db012ffa85b0590ed3c70e56029b97f22278" Nov 04 03:18:11.720571 ip-10-0-134-125 kubenswrapper[4405]: E1104 03:18:11.720538 4405 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9c2f7cc08dbbb4aa2b4f90dd96c5db012ffa85b0590ed3c70e56029b97f22278\": container with ID starting with 9c2f7cc08dbbb4aa2b4f90dd96c5db012ffa85b0590ed3c70e56029b97f22278 not found: ID does not exist" containerID="9c2f7cc08dbbb4aa2b4f90dd96c5db012ffa85b0590ed3c70e56029b97f22278" Nov 04 03:18:11.720791 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:18:11.720582 4405 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9c2f7cc08dbbb4aa2b4f90dd96c5db012ffa85b0590ed3c70e56029b97f22278"} err="failed to get container status \"9c2f7cc08dbbb4aa2b4f90dd96c5db012ffa85b0590ed3c70e56029b97f22278\": rpc error: code = NotFound desc = could not find container \"9c2f7cc08dbbb4aa2b4f90dd96c5db012ffa85b0590ed3c70e56029b97f22278\": container with ID starting with 9c2f7cc08dbbb4aa2b4f90dd96c5db012ffa85b0590ed3c70e56029b97f22278 not found: ID does not exist" Nov 04 03:18:11.720791 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:18:11.720617 4405 scope.go:122] "RemoveContainer" containerID="a30b7638525ea4ce26e47f3f8e99113b4d9df664c679a6e02855af22859fe884" Nov 04 03:18:11.721027 ip-10-0-134-125 kubenswrapper[4405]: E1104 03:18:11.720997 4405 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a30b7638525ea4ce26e47f3f8e99113b4d9df664c679a6e02855af22859fe884\": container with ID starting with a30b7638525ea4ce26e47f3f8e99113b4d9df664c679a6e02855af22859fe884 not found: ID does not exist" containerID="a30b7638525ea4ce26e47f3f8e99113b4d9df664c679a6e02855af22859fe884" Nov 04 03:18:11.721109 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:18:11.721038 4405 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a30b7638525ea4ce26e47f3f8e99113b4d9df664c679a6e02855af22859fe884"} err="failed to get container status \"a30b7638525ea4ce26e47f3f8e99113b4d9df664c679a6e02855af22859fe884\": rpc error: code = NotFound desc = could not find container \"a30b7638525ea4ce26e47f3f8e99113b4d9df664c679a6e02855af22859fe884\": container with ID starting with a30b7638525ea4ce26e47f3f8e99113b4d9df664c679a6e02855af22859fe884 not found: ID does not exist" Nov 04 03:18:12.202181 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:18:12.202144 4405 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-072ece-openshift-default-78b784fd8c-k2vfq"] Nov 04 03:18:12.252006 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:18:12.251965 4405 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-072ece-openshift-default-78b784fd8c-k2vfq"] Nov 04 03:18:12.504556 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:18:12.504467 4405 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-a0bcbc-openshift-default-65dc9d9b6c-g5hw6"] Nov 04 03:18:12.504993 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:18:12.504739 4405 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-a0bcbc-openshift-default-65dc9d9b6c-g5hw6" podUID="d0362f05-c556-4c71-9b54-b360072d0758" containerName="istio-proxy" containerID="cri-o://30c4684c5addc51fb80dee95aba95b1bf3cbeef5c1127dfc4b592b8892857c6e" gracePeriod=30 Nov 04 03:18:13.414656 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:18:13.414623 4405 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w4-35df3e-7969f6bd6b-ptzqg"] Nov 04 03:18:13.550593 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:18:13.550553 4405 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w4-35df3e-7d67b7f575-zbnm5"] Nov 04 03:18:13.701235 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:18:13.701140 4405 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="652d018b-8d07-49ae-b9fb-a87cfc70b0c8" path="/var/lib/kubelet/pods/652d018b-8d07-49ae-b9fb-a87cfc70b0c8/volumes" Nov 04 03:18:16.328374 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:18:16.328276 4405 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-bl5mg"] Nov 04 03:18:17.726847 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:18:17.726814 4405 generic.go:356] "Generic (PLEG): container finished" podID="d0362f05-c556-4c71-9b54-b360072d0758" containerID="30c4684c5addc51fb80dee95aba95b1bf3cbeef5c1127dfc4b592b8892857c6e" exitCode=0 Nov 04 03:18:17.727242 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:18:17.726893 4405 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-a0bcbc-openshift-default-65dc9d9b6c-g5hw6" event={"ID":"d0362f05-c556-4c71-9b54-b360072d0758","Type":"ContainerDied","Data":"30c4684c5addc51fb80dee95aba95b1bf3cbeef5c1127dfc4b592b8892857c6e"} Nov 04 03:18:17.759900 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:18:17.759871 4405 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-a0bcbc-openshift-default-65dc9d9b6c-g5hw6" Nov 04 03:18:17.895768 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:18:17.895730 4405 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d0362f05-c556-4c71-9b54-b360072d0758-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d0362f05-c556-4c71-9b54-b360072d0758-credential-socket\") pod \"d0362f05-c556-4c71-9b54-b360072d0758\" (UID: \"d0362f05-c556-4c71-9b54-b360072d0758\") " Nov 04 03:18:17.895977 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:18:17.895799 4405 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d0362f05-c556-4c71-9b54-b360072d0758-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d0362f05-c556-4c71-9b54-b360072d0758-workload-certs\") pod \"d0362f05-c556-4c71-9b54-b360072d0758\" (UID: \"d0362f05-c556-4c71-9b54-b360072d0758\") " Nov 04 03:18:17.895977 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:18:17.895841 4405 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d0362f05-c556-4c71-9b54-b360072d0758-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d0362f05-c556-4c71-9b54-b360072d0758-istio-envoy\") pod \"d0362f05-c556-4c71-9b54-b360072d0758\" (UID: \"d0362f05-c556-4c71-9b54-b360072d0758\") " Nov 04 03:18:17.895977 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:18:17.895890 4405 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d0362f05-c556-4c71-9b54-b360072d0758-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d0362f05-c556-4c71-9b54-b360072d0758-workload-socket\") pod \"d0362f05-c556-4c71-9b54-b360072d0758\" (UID: \"d0362f05-c556-4c71-9b54-b360072d0758\") " Nov 04 03:18:17.895977 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:18:17.895933 4405 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/d0362f05-c556-4c71-9b54-b360072d0758-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d0362f05-c556-4c71-9b54-b360072d0758-istio-podinfo\") pod \"d0362f05-c556-4c71-9b54-b360072d0758\" (UID: \"d0362f05-c556-4c71-9b54-b360072d0758\") " Nov 04 03:18:17.895977 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:18:17.895967 4405 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d0362f05-c556-4c71-9b54-b360072d0758-istio-token\" (UniqueName: \"kubernetes.io/projected/d0362f05-c556-4c71-9b54-b360072d0758-istio-token\") pod \"d0362f05-c556-4c71-9b54-b360072d0758\" (UID: \"d0362f05-c556-4c71-9b54-b360072d0758\") " Nov 04 03:18:17.896259 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:18:17.896008 4405 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d0362f05-c556-4c71-9b54-b360072d0758-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d0362f05-c556-4c71-9b54-b360072d0758-istiod-ca-cert\") pod \"d0362f05-c556-4c71-9b54-b360072d0758\" (UID: \"d0362f05-c556-4c71-9b54-b360072d0758\") " Nov 04 03:18:17.896259 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:18:17.896016 4405 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d0362f05-c556-4c71-9b54-b360072d0758-credential-socket" pod "d0362f05-c556-4c71-9b54-b360072d0758" (UID: "d0362f05-c556-4c71-9b54-b360072d0758"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 03:18:17.896259 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:18:17.896049 4405 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d0362f05-c556-4c71-9b54-b360072d0758-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d0362f05-c556-4c71-9b54-b360072d0758-istio-data\") pod \"d0362f05-c556-4c71-9b54-b360072d0758\" (UID: \"d0362f05-c556-4c71-9b54-b360072d0758\") " Nov 04 03:18:17.896259 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:18:17.896084 4405 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d0362f05-c556-4c71-9b54-b360072d0758-kube-api-access-cxd6x\" (UniqueName: \"kubernetes.io/projected/d0362f05-c556-4c71-9b54-b360072d0758-kube-api-access-cxd6x\") pod \"d0362f05-c556-4c71-9b54-b360072d0758\" (UID: \"d0362f05-c556-4c71-9b54-b360072d0758\") " Nov 04 03:18:17.896259 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:18:17.896147 4405 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d0362f05-c556-4c71-9b54-b360072d0758-workload-certs" pod "d0362f05-c556-4c71-9b54-b360072d0758" (UID: "d0362f05-c556-4c71-9b54-b360072d0758"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 03:18:17.896259 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:18:17.896256 4405 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d0362f05-c556-4c71-9b54-b360072d0758-workload-socket" pod "d0362f05-c556-4c71-9b54-b360072d0758" (UID: "d0362f05-c556-4c71-9b54-b360072d0758"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 03:18:17.896544 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:18:17.896344 4405 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d0362f05-c556-4c71-9b54-b360072d0758-istio-data" pod "d0362f05-c556-4c71-9b54-b360072d0758" (UID: "d0362f05-c556-4c71-9b54-b360072d0758"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 03:18:17.898343 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:18:17.897033 4405 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d0362f05-c556-4c71-9b54-b360072d0758-istiod-ca-cert" pod "d0362f05-c556-4c71-9b54-b360072d0758" (UID: "d0362f05-c556-4c71-9b54-b360072d0758"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 03:18:17.898343 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:18:17.898313 4405 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d0362f05-c556-4c71-9b54-b360072d0758-workload-certs\") on node \"ip-10-0-134-125.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:18:17.898547 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:18:17.898350 4405 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d0362f05-c556-4c71-9b54-b360072d0758-workload-socket\") on node \"ip-10-0-134-125.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:18:17.898547 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:18:17.898365 4405 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d0362f05-c556-4c71-9b54-b360072d0758-istio-data\") on node \"ip-10-0-134-125.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:18:17.898547 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:18:17.898378 4405 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d0362f05-c556-4c71-9b54-b360072d0758-credential-socket\") on node \"ip-10-0-134-125.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:18:17.898829 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:18:17.898803 4405 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d0362f05-c556-4c71-9b54-b360072d0758-istio-envoy" pod "d0362f05-c556-4c71-9b54-b360072d0758" (UID: "d0362f05-c556-4c71-9b54-b360072d0758"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 03:18:17.899978 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:18:17.899285 4405 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d0362f05-c556-4c71-9b54-b360072d0758-istio-podinfo" pod "d0362f05-c556-4c71-9b54-b360072d0758" (UID: "d0362f05-c556-4c71-9b54-b360072d0758"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 04 03:18:17.905538 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:18:17.900481 4405 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d0362f05-c556-4c71-9b54-b360072d0758-istio-token" pod "d0362f05-c556-4c71-9b54-b360072d0758" (UID: "d0362f05-c556-4c71-9b54-b360072d0758"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 03:18:17.905538 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:18:17.900803 4405 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d0362f05-c556-4c71-9b54-b360072d0758-kube-api-access-cxd6x" pod "d0362f05-c556-4c71-9b54-b360072d0758" (UID: "d0362f05-c556-4c71-9b54-b360072d0758"). InnerVolumeSpecName "kube-api-access-cxd6x". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 03:18:17.999745 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:18:17.999657 4405 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d0362f05-c556-4c71-9b54-b360072d0758-istio-envoy\") on node \"ip-10-0-134-125.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:18:17.999905 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:18:17.999751 4405 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d0362f05-c556-4c71-9b54-b360072d0758-istio-podinfo\") on node \"ip-10-0-134-125.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:18:17.999905 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:18:17.999768 4405 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d0362f05-c556-4c71-9b54-b360072d0758-istio-token\") on node \"ip-10-0-134-125.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:18:17.999905 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:18:17.999782 4405 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d0362f05-c556-4c71-9b54-b360072d0758-istiod-ca-cert\") on node \"ip-10-0-134-125.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:18:17.999905 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:18:17.999797 4405 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cxd6x\" (UniqueName: \"kubernetes.io/projected/d0362f05-c556-4c71-9b54-b360072d0758-kube-api-access-cxd6x\") on node \"ip-10-0-134-125.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:18:18.415407 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:18:18.415346 4405 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-35df3e-7969f6bd6b-ptzqg" podUID="8d7f28e3-d62b-4813-81e3-364d4700983a" containerName="payload-processing" containerID="cri-o://a574bdaf53a239cbffbcb9c5f7d32def80a9d146817eeca5bccdc1ea96d3fd74" gracePeriod=25 Nov 04 03:18:18.551306 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:18:18.551245 4405 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-35df3e-7d67b7f575-zbnm5" podUID="317d0b2a-1ab0-4278-add3-cfaee93aa69c" containerName="payload-pre-processing" containerID="cri-o://a35ede810afa95faec96e58c54ae75899966b69c6c91d76a4a40173fd5a4c274" gracePeriod=25 Nov 04 03:18:18.677625 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:18:18.677559 4405 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-35df3e-7969f6bd6b-ptzqg" Nov 04 03:18:18.733661 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:18:18.733626 4405 generic.go:356] "Generic (PLEG): container finished" podID="317d0b2a-1ab0-4278-add3-cfaee93aa69c" containerID="a35ede810afa95faec96e58c54ae75899966b69c6c91d76a4a40173fd5a4c274" exitCode=0 Nov 04 03:18:18.734159 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:18:18.733702 4405 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-35df3e-7d67b7f575-zbnm5" event={"ID":"317d0b2a-1ab0-4278-add3-cfaee93aa69c","Type":"ContainerDied","Data":"a35ede810afa95faec96e58c54ae75899966b69c6c91d76a4a40173fd5a4c274"} Nov 04 03:18:18.735112 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:18:18.735088 4405 generic.go:356] "Generic (PLEG): container finished" podID="8d7f28e3-d62b-4813-81e3-364d4700983a" containerID="a574bdaf53a239cbffbcb9c5f7d32def80a9d146817eeca5bccdc1ea96d3fd74" exitCode=0 Nov 04 03:18:18.735251 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:18:18.735154 4405 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-35df3e-7969f6bd6b-ptzqg" Nov 04 03:18:18.735251 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:18:18.735166 4405 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-35df3e-7969f6bd6b-ptzqg" event={"ID":"8d7f28e3-d62b-4813-81e3-364d4700983a","Type":"ContainerDied","Data":"a574bdaf53a239cbffbcb9c5f7d32def80a9d146817eeca5bccdc1ea96d3fd74"} Nov 04 03:18:18.735251 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:18:18.735220 4405 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-35df3e-7969f6bd6b-ptzqg" event={"ID":"8d7f28e3-d62b-4813-81e3-364d4700983a","Type":"ContainerDied","Data":"4241ec894de25db5e18ebb33d31b9b8a9aa5e7106dd987c7ad88d19dd91c78d5"} Nov 04 03:18:18.735251 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:18:18.735247 4405 scope.go:122] "RemoveContainer" containerID="a574bdaf53a239cbffbcb9c5f7d32def80a9d146817eeca5bccdc1ea96d3fd74" Nov 04 03:18:18.736694 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:18:18.736654 4405 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-a0bcbc-openshift-default-65dc9d9b6c-g5hw6" event={"ID":"d0362f05-c556-4c71-9b54-b360072d0758","Type":"ContainerDied","Data":"80846d84e267028032ac689318c25d1ab60baf25272b03cb41e3b9785873cf66"} Nov 04 03:18:18.736808 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:18:18.736738 4405 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-a0bcbc-openshift-default-65dc9d9b6c-g5hw6" Nov 04 03:18:18.748135 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:18:18.748113 4405 scope.go:122] "RemoveContainer" containerID="a574bdaf53a239cbffbcb9c5f7d32def80a9d146817eeca5bccdc1ea96d3fd74" Nov 04 03:18:18.748471 ip-10-0-134-125 kubenswrapper[4405]: E1104 03:18:18.748452 4405 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a574bdaf53a239cbffbcb9c5f7d32def80a9d146817eeca5bccdc1ea96d3fd74\": container with ID starting with a574bdaf53a239cbffbcb9c5f7d32def80a9d146817eeca5bccdc1ea96d3fd74 not found: ID does not exist" containerID="a574bdaf53a239cbffbcb9c5f7d32def80a9d146817eeca5bccdc1ea96d3fd74" Nov 04 03:18:18.748536 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:18:18.748481 4405 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a574bdaf53a239cbffbcb9c5f7d32def80a9d146817eeca5bccdc1ea96d3fd74"} err="failed to get container status \"a574bdaf53a239cbffbcb9c5f7d32def80a9d146817eeca5bccdc1ea96d3fd74\": rpc error: code = NotFound desc = could not find container \"a574bdaf53a239cbffbcb9c5f7d32def80a9d146817eeca5bccdc1ea96d3fd74\": container with ID starting with a574bdaf53a239cbffbcb9c5f7d32def80a9d146817eeca5bccdc1ea96d3fd74 not found: ID does not exist" Nov 04 03:18:18.748536 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:18:18.748499 4405 scope.go:122] "RemoveContainer" containerID="30c4684c5addc51fb80dee95aba95b1bf3cbeef5c1127dfc4b592b8892857c6e" Nov 04 03:18:18.791515 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:18:18.791487 4405 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-35df3e-7d67b7f575-zbnm5" Nov 04 03:18:18.808111 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:18:18.808081 4405 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8d7f28e3-d62b-4813-81e3-364d4700983a-kube-api-access-qr8hr\" (UniqueName: \"kubernetes.io/projected/8d7f28e3-d62b-4813-81e3-364d4700983a-kube-api-access-qr8hr\") pod \"8d7f28e3-d62b-4813-81e3-364d4700983a\" (UID: \"8d7f28e3-d62b-4813-81e3-364d4700983a\") " Nov 04 03:18:18.808292 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:18:18.808148 4405 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8d7f28e3-d62b-4813-81e3-364d4700983a-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8d7f28e3-d62b-4813-81e3-364d4700983a-plugins-config-volume\") pod \"8d7f28e3-d62b-4813-81e3-364d4700983a\" (UID: \"8d7f28e3-d62b-4813-81e3-364d4700983a\") " Nov 04 03:18:18.808555 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:18:18.808532 4405 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8d7f28e3-d62b-4813-81e3-364d4700983a-plugins-config-volume" pod "8d7f28e3-d62b-4813-81e3-364d4700983a" (UID: "8d7f28e3-d62b-4813-81e3-364d4700983a"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 03:18:18.810394 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:18:18.810371 4405 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8d7f28e3-d62b-4813-81e3-364d4700983a-kube-api-access-qr8hr" pod "8d7f28e3-d62b-4813-81e3-364d4700983a" (UID: "8d7f28e3-d62b-4813-81e3-364d4700983a"). InnerVolumeSpecName "kube-api-access-qr8hr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 03:18:18.909023 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:18:18.908980 4405 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/317d0b2a-1ab0-4278-add3-cfaee93aa69c-kube-api-access-jp8pj\" (UniqueName: \"kubernetes.io/projected/317d0b2a-1ab0-4278-add3-cfaee93aa69c-kube-api-access-jp8pj\") pod \"317d0b2a-1ab0-4278-add3-cfaee93aa69c\" (UID: \"317d0b2a-1ab0-4278-add3-cfaee93aa69c\") " Nov 04 03:18:18.909224 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:18:18.909087 4405 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/317d0b2a-1ab0-4278-add3-cfaee93aa69c-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/317d0b2a-1ab0-4278-add3-cfaee93aa69c-plugins-config-volume\") pod \"317d0b2a-1ab0-4278-add3-cfaee93aa69c\" (UID: \"317d0b2a-1ab0-4278-add3-cfaee93aa69c\") " Nov 04 03:18:18.909478 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:18:18.909454 4405 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8d7f28e3-d62b-4813-81e3-364d4700983a-plugins-config-volume\") on node \"ip-10-0-134-125.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:18:18.909598 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:18:18.909479 4405 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qr8hr\" (UniqueName: \"kubernetes.io/projected/8d7f28e3-d62b-4813-81e3-364d4700983a-kube-api-access-qr8hr\") on node \"ip-10-0-134-125.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:18:18.909769 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:18:18.909743 4405 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/317d0b2a-1ab0-4278-add3-cfaee93aa69c-plugins-config-volume" pod "317d0b2a-1ab0-4278-add3-cfaee93aa69c" (UID: "317d0b2a-1ab0-4278-add3-cfaee93aa69c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 03:18:18.911948 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:18:18.911925 4405 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/317d0b2a-1ab0-4278-add3-cfaee93aa69c-kube-api-access-jp8pj" pod "317d0b2a-1ab0-4278-add3-cfaee93aa69c" (UID: "317d0b2a-1ab0-4278-add3-cfaee93aa69c"). InnerVolumeSpecName "kube-api-access-jp8pj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 03:18:19.010541 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:18:19.010458 4405 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jp8pj\" (UniqueName: \"kubernetes.io/projected/317d0b2a-1ab0-4278-add3-cfaee93aa69c-kube-api-access-jp8pj\") on node \"ip-10-0-134-125.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:18:19.010541 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:18:19.010486 4405 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/317d0b2a-1ab0-4278-add3-cfaee93aa69c-plugins-config-volume\") on node \"ip-10-0-134-125.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:18:19.020022 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:18:19.019993 4405 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-a0bcbc-openshift-default-65dc9d9b6c-g5hw6"] Nov 04 03:18:19.069141 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:18:19.069104 4405 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-a0bcbc-openshift-default-65dc9d9b6c-g5hw6"] Nov 04 03:18:19.184336 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:18:19.184297 4405 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w4-35df3e-7969f6bd6b-ptzqg"] Nov 04 03:18:19.188171 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:18:19.188139 4405 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w4-35df3e-7969f6bd6b-ptzqg"] Nov 04 03:18:19.700933 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:18:19.700902 4405 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8d7f28e3-d62b-4813-81e3-364d4700983a" path="/var/lib/kubelet/pods/8d7f28e3-d62b-4813-81e3-364d4700983a/volumes" Nov 04 03:18:19.701357 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:18:19.701344 4405 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d0362f05-c556-4c71-9b54-b360072d0758" path="/var/lib/kubelet/pods/d0362f05-c556-4c71-9b54-b360072d0758/volumes" Nov 04 03:18:19.744700 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:18:19.744650 4405 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-35df3e-7d67b7f575-zbnm5" Nov 04 03:18:19.744700 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:18:19.744682 4405 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-35df3e-7d67b7f575-zbnm5" event={"ID":"317d0b2a-1ab0-4278-add3-cfaee93aa69c","Type":"ContainerDied","Data":"25d7468a99e3df632cdc7709d38822dedbf196fd35b075e3c65ec29cc7233b2c"} Nov 04 03:18:19.745317 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:18:19.744751 4405 scope.go:122] "RemoveContainer" containerID="a35ede810afa95faec96e58c54ae75899966b69c6c91d76a4a40173fd5a4c274" Nov 04 03:18:19.908965 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:18:19.908927 4405 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w4-35df3e-7d67b7f575-zbnm5"] Nov 04 03:18:19.958061 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:18:19.958026 4405 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w4-35df3e-7d67b7f575-zbnm5"] Nov 04 03:18:21.042331 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:18:21.042286 4405 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-35df3e-8675b47557-ms62p"] Nov 04 03:18:21.042850 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:18:21.042526 4405 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-35df3e-8675b47557-ms62p" podUID="c75d3a55-98c5-4af1-96ff-596d556597b9" containerName="maas-api" containerID="cri-o://5edb540a5308ac2a9393bc43d1395c33fe254a732fd71a5cd37eceed82b49fdd" gracePeriod=30 Nov 04 03:18:21.300490 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:18:21.300423 4405 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-35df3e-8675b47557-ms62p" Nov 04 03:18:21.433411 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:18:21.433380 4405 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c75d3a55-98c5-4af1-96ff-596d556597b9-metrics-certs\" (UniqueName: \"kubernetes.io/secret/c75d3a55-98c5-4af1-96ff-596d556597b9-metrics-certs\") pod \"c75d3a55-98c5-4af1-96ff-596d556597b9\" (UID: \"c75d3a55-98c5-4af1-96ff-596d556597b9\") " Nov 04 03:18:21.433611 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:18:21.433451 4405 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c75d3a55-98c5-4af1-96ff-596d556597b9-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c75d3a55-98c5-4af1-96ff-596d556597b9-openshift-service-ca\") pod \"c75d3a55-98c5-4af1-96ff-596d556597b9\" (UID: \"c75d3a55-98c5-4af1-96ff-596d556597b9\") " Nov 04 03:18:21.433611 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:18:21.433554 4405 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c75d3a55-98c5-4af1-96ff-596d556597b9-kube-api-access-kttb5\" (UniqueName: \"kubernetes.io/projected/c75d3a55-98c5-4af1-96ff-596d556597b9-kube-api-access-kttb5\") pod \"c75d3a55-98c5-4af1-96ff-596d556597b9\" (UID: \"c75d3a55-98c5-4af1-96ff-596d556597b9\") " Nov 04 03:18:21.433611 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:18:21.433572 4405 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c75d3a55-98c5-4af1-96ff-596d556597b9-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c75d3a55-98c5-4af1-96ff-596d556597b9-maas-api-tls\") pod \"c75d3a55-98c5-4af1-96ff-596d556597b9\" (UID: \"c75d3a55-98c5-4af1-96ff-596d556597b9\") " Nov 04 03:18:21.433898 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:18:21.433874 4405 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c75d3a55-98c5-4af1-96ff-596d556597b9-openshift-service-ca" pod "c75d3a55-98c5-4af1-96ff-596d556597b9" (UID: "c75d3a55-98c5-4af1-96ff-596d556597b9"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 03:18:21.435961 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:18:21.435931 4405 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c75d3a55-98c5-4af1-96ff-596d556597b9-metrics-certs" pod "c75d3a55-98c5-4af1-96ff-596d556597b9" (UID: "c75d3a55-98c5-4af1-96ff-596d556597b9"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 04 03:18:21.436088 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:18:21.435996 4405 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c75d3a55-98c5-4af1-96ff-596d556597b9-kube-api-access-kttb5" pod "c75d3a55-98c5-4af1-96ff-596d556597b9" (UID: "c75d3a55-98c5-4af1-96ff-596d556597b9"). InnerVolumeSpecName "kube-api-access-kttb5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 03:18:21.436088 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:18:21.436028 4405 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c75d3a55-98c5-4af1-96ff-596d556597b9-maas-api-tls" pod "c75d3a55-98c5-4af1-96ff-596d556597b9" (UID: "c75d3a55-98c5-4af1-96ff-596d556597b9"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 04 03:18:21.535030 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:18:21.534992 4405 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c75d3a55-98c5-4af1-96ff-596d556597b9-openshift-service-ca\") on node \"ip-10-0-134-125.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:18:21.535030 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:18:21.535031 4405 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kttb5\" (UniqueName: \"kubernetes.io/projected/c75d3a55-98c5-4af1-96ff-596d556597b9-kube-api-access-kttb5\") on node \"ip-10-0-134-125.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:18:21.535333 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:18:21.535047 4405 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c75d3a55-98c5-4af1-96ff-596d556597b9-maas-api-tls\") on node \"ip-10-0-134-125.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:18:21.535333 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:18:21.535060 4405 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c75d3a55-98c5-4af1-96ff-596d556597b9-metrics-certs\") on node \"ip-10-0-134-125.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:18:21.700054 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:18:21.700022 4405 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="317d0b2a-1ab0-4278-add3-cfaee93aa69c" path="/var/lib/kubelet/pods/317d0b2a-1ab0-4278-add3-cfaee93aa69c/volumes" Nov 04 03:18:21.757248 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:18:21.757214 4405 generic.go:356] "Generic (PLEG): container finished" podID="c75d3a55-98c5-4af1-96ff-596d556597b9" containerID="5edb540a5308ac2a9393bc43d1395c33fe254a732fd71a5cd37eceed82b49fdd" exitCode=0 Nov 04 03:18:21.757420 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:18:21.757260 4405 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-35df3e-8675b47557-ms62p" Nov 04 03:18:21.757420 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:18:21.757295 4405 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-35df3e-8675b47557-ms62p" event={"ID":"c75d3a55-98c5-4af1-96ff-596d556597b9","Type":"ContainerDied","Data":"5edb540a5308ac2a9393bc43d1395c33fe254a732fd71a5cd37eceed82b49fdd"} Nov 04 03:18:21.757420 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:18:21.757350 4405 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-35df3e-8675b47557-ms62p" event={"ID":"c75d3a55-98c5-4af1-96ff-596d556597b9","Type":"ContainerDied","Data":"3e7bd26893707ea04d49a05a2246db235c5a1567d4ae5ef0b97e573b927d58cf"} Nov 04 03:18:21.757420 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:18:21.757373 4405 scope.go:122] "RemoveContainer" containerID="5edb540a5308ac2a9393bc43d1395c33fe254a732fd71a5cd37eceed82b49fdd" Nov 04 03:18:21.766351 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:18:21.766328 4405 scope.go:122] "RemoveContainer" containerID="5edb540a5308ac2a9393bc43d1395c33fe254a732fd71a5cd37eceed82b49fdd" Nov 04 03:18:21.766701 ip-10-0-134-125 kubenswrapper[4405]: E1104 03:18:21.766655 4405 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5edb540a5308ac2a9393bc43d1395c33fe254a732fd71a5cd37eceed82b49fdd\": container with ID starting with 5edb540a5308ac2a9393bc43d1395c33fe254a732fd71a5cd37eceed82b49fdd not found: ID does not exist" containerID="5edb540a5308ac2a9393bc43d1395c33fe254a732fd71a5cd37eceed82b49fdd" Nov 04 03:18:21.766803 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:18:21.766708 4405 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5edb540a5308ac2a9393bc43d1395c33fe254a732fd71a5cd37eceed82b49fdd"} err="failed to get container status \"5edb540a5308ac2a9393bc43d1395c33fe254a732fd71a5cd37eceed82b49fdd\": rpc error: code = NotFound desc = could not find container \"5edb540a5308ac2a9393bc43d1395c33fe254a732fd71a5cd37eceed82b49fdd\": container with ID starting with 5edb540a5308ac2a9393bc43d1395c33fe254a732fd71a5cd37eceed82b49fdd not found: ID does not exist" Nov 04 03:18:21.934233 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:18:21.934200 4405 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-35df3e-8675b47557-ms62p"] Nov 04 03:18:21.943830 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:18:21.939275 4405 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-35df3e-8675b47557-ms62p"] Nov 04 03:18:23.700920 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:18:23.700884 4405 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c75d3a55-98c5-4af1-96ff-596d556597b9" path="/var/lib/kubelet/pods/c75d3a55-98c5-4af1-96ff-596d556597b9/volumes" Nov 04 03:18:27.220041 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:18:27.220005 4405 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-35df3e-openshift-default-6c585b76f-vl6wz"] Nov 04 03:18:27.220519 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:18:27.220247 4405 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w4-35df3e-openshift-default-6c585b76f-vl6wz" podUID="18ffb552-4917-4c03-a3b5-aa9c6a89f0e7" containerName="istio-proxy" containerID="cri-o://f135664a6bfe876b38d0757d58e9bcb5949fb9a20d7ffc8fad582c1c859c5ee7" gracePeriod=30 Nov 04 03:18:32.481292 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:18:32.481264 4405 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w4-35df3e-openshift-default-6c585b76f-vl6wz" Nov 04 03:18:32.641791 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:18:32.641755 4405 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/18ffb552-4917-4c03-a3b5-aa9c6a89f0e7-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/18ffb552-4917-4c03-a3b5-aa9c6a89f0e7-istio-podinfo\") pod \"18ffb552-4917-4c03-a3b5-aa9c6a89f0e7\" (UID: \"18ffb552-4917-4c03-a3b5-aa9c6a89f0e7\") " Nov 04 03:18:32.641998 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:18:32.641799 4405 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/18ffb552-4917-4c03-a3b5-aa9c6a89f0e7-kube-api-access-lzc4n\" (UniqueName: \"kubernetes.io/projected/18ffb552-4917-4c03-a3b5-aa9c6a89f0e7-kube-api-access-lzc4n\") pod \"18ffb552-4917-4c03-a3b5-aa9c6a89f0e7\" (UID: \"18ffb552-4917-4c03-a3b5-aa9c6a89f0e7\") " Nov 04 03:18:32.641998 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:18:32.641856 4405 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/18ffb552-4917-4c03-a3b5-aa9c6a89f0e7-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/18ffb552-4917-4c03-a3b5-aa9c6a89f0e7-istiod-ca-cert\") pod \"18ffb552-4917-4c03-a3b5-aa9c6a89f0e7\" (UID: \"18ffb552-4917-4c03-a3b5-aa9c6a89f0e7\") " Nov 04 03:18:32.641998 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:18:32.641885 4405 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/18ffb552-4917-4c03-a3b5-aa9c6a89f0e7-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/18ffb552-4917-4c03-a3b5-aa9c6a89f0e7-workload-certs\") pod \"18ffb552-4917-4c03-a3b5-aa9c6a89f0e7\" (UID: \"18ffb552-4917-4c03-a3b5-aa9c6a89f0e7\") " Nov 04 03:18:32.641998 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:18:32.641903 4405 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/18ffb552-4917-4c03-a3b5-aa9c6a89f0e7-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/18ffb552-4917-4c03-a3b5-aa9c6a89f0e7-credential-socket\") pod \"18ffb552-4917-4c03-a3b5-aa9c6a89f0e7\" (UID: \"18ffb552-4917-4c03-a3b5-aa9c6a89f0e7\") " Nov 04 03:18:32.641998 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:18:32.641954 4405 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/18ffb552-4917-4c03-a3b5-aa9c6a89f0e7-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/18ffb552-4917-4c03-a3b5-aa9c6a89f0e7-workload-socket\") pod \"18ffb552-4917-4c03-a3b5-aa9c6a89f0e7\" (UID: \"18ffb552-4917-4c03-a3b5-aa9c6a89f0e7\") " Nov 04 03:18:32.642283 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:18:32.642020 4405 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/18ffb552-4917-4c03-a3b5-aa9c6a89f0e7-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/18ffb552-4917-4c03-a3b5-aa9c6a89f0e7-istio-envoy\") pod \"18ffb552-4917-4c03-a3b5-aa9c6a89f0e7\" (UID: \"18ffb552-4917-4c03-a3b5-aa9c6a89f0e7\") " Nov 04 03:18:32.642283 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:18:32.642046 4405 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/18ffb552-4917-4c03-a3b5-aa9c6a89f0e7-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/18ffb552-4917-4c03-a3b5-aa9c6a89f0e7-istio-data\") pod \"18ffb552-4917-4c03-a3b5-aa9c6a89f0e7\" (UID: \"18ffb552-4917-4c03-a3b5-aa9c6a89f0e7\") " Nov 04 03:18:32.642283 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:18:32.642080 4405 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/18ffb552-4917-4c03-a3b5-aa9c6a89f0e7-istio-token\" (UniqueName: \"kubernetes.io/projected/18ffb552-4917-4c03-a3b5-aa9c6a89f0e7-istio-token\") pod \"18ffb552-4917-4c03-a3b5-aa9c6a89f0e7\" (UID: \"18ffb552-4917-4c03-a3b5-aa9c6a89f0e7\") " Nov 04 03:18:32.642283 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:18:32.642257 4405 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/18ffb552-4917-4c03-a3b5-aa9c6a89f0e7-workload-certs" pod "18ffb552-4917-4c03-a3b5-aa9c6a89f0e7" (UID: "18ffb552-4917-4c03-a3b5-aa9c6a89f0e7"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 03:18:32.642506 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:18:32.642298 4405 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/18ffb552-4917-4c03-a3b5-aa9c6a89f0e7-istiod-ca-cert" pod "18ffb552-4917-4c03-a3b5-aa9c6a89f0e7" (UID: "18ffb552-4917-4c03-a3b5-aa9c6a89f0e7"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 03:18:32.642506 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:18:32.642439 4405 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/18ffb552-4917-4c03-a3b5-aa9c6a89f0e7-istiod-ca-cert\") on node \"ip-10-0-134-125.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:18:32.642506 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:18:32.642459 4405 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/18ffb552-4917-4c03-a3b5-aa9c6a89f0e7-workload-certs\") on node \"ip-10-0-134-125.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:18:32.642506 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:18:32.642499 4405 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/18ffb552-4917-4c03-a3b5-aa9c6a89f0e7-workload-socket" pod "18ffb552-4917-4c03-a3b5-aa9c6a89f0e7" (UID: "18ffb552-4917-4c03-a3b5-aa9c6a89f0e7"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 03:18:32.642765 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:18:32.642658 4405 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/18ffb552-4917-4c03-a3b5-aa9c6a89f0e7-istio-data" pod "18ffb552-4917-4c03-a3b5-aa9c6a89f0e7" (UID: "18ffb552-4917-4c03-a3b5-aa9c6a89f0e7"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 03:18:32.642821 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:18:32.642791 4405 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/18ffb552-4917-4c03-a3b5-aa9c6a89f0e7-credential-socket" pod "18ffb552-4917-4c03-a3b5-aa9c6a89f0e7" (UID: "18ffb552-4917-4c03-a3b5-aa9c6a89f0e7"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 03:18:32.644484 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:18:32.644436 4405 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/18ffb552-4917-4c03-a3b5-aa9c6a89f0e7-istio-token" pod "18ffb552-4917-4c03-a3b5-aa9c6a89f0e7" (UID: "18ffb552-4917-4c03-a3b5-aa9c6a89f0e7"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 03:18:32.644596 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:18:32.644546 4405 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/18ffb552-4917-4c03-a3b5-aa9c6a89f0e7-istio-envoy" pod "18ffb552-4917-4c03-a3b5-aa9c6a89f0e7" (UID: "18ffb552-4917-4c03-a3b5-aa9c6a89f0e7"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 03:18:32.644780 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:18:32.644762 4405 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/18ffb552-4917-4c03-a3b5-aa9c6a89f0e7-istio-podinfo" pod "18ffb552-4917-4c03-a3b5-aa9c6a89f0e7" (UID: "18ffb552-4917-4c03-a3b5-aa9c6a89f0e7"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 04 03:18:32.644833 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:18:32.644785 4405 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/18ffb552-4917-4c03-a3b5-aa9c6a89f0e7-kube-api-access-lzc4n" pod "18ffb552-4917-4c03-a3b5-aa9c6a89f0e7" (UID: "18ffb552-4917-4c03-a3b5-aa9c6a89f0e7"). InnerVolumeSpecName "kube-api-access-lzc4n". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 03:18:32.731376 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:18:32.731328 4405 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-203c35/simulator-w4-203c35-kserve-f9c4bd695-999ns"] Nov 04 03:18:32.743589 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:18:32.743559 4405 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/18ffb552-4917-4c03-a3b5-aa9c6a89f0e7-credential-socket\") on node \"ip-10-0-134-125.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:18:32.743589 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:18:32.743587 4405 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/18ffb552-4917-4c03-a3b5-aa9c6a89f0e7-workload-socket\") on node \"ip-10-0-134-125.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:18:32.743795 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:18:32.743597 4405 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/18ffb552-4917-4c03-a3b5-aa9c6a89f0e7-istio-envoy\") on node \"ip-10-0-134-125.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:18:32.743795 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:18:32.743606 4405 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/18ffb552-4917-4c03-a3b5-aa9c6a89f0e7-istio-data\") on node \"ip-10-0-134-125.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:18:32.743795 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:18:32.743614 4405 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/18ffb552-4917-4c03-a3b5-aa9c6a89f0e7-istio-token\") on node \"ip-10-0-134-125.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:18:32.743795 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:18:32.743622 4405 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/18ffb552-4917-4c03-a3b5-aa9c6a89f0e7-istio-podinfo\") on node \"ip-10-0-134-125.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:18:32.743795 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:18:32.743631 4405 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lzc4n\" (UniqueName: \"kubernetes.io/projected/18ffb552-4917-4c03-a3b5-aa9c6a89f0e7-kube-api-access-lzc4n\") on node \"ip-10-0-134-125.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:18:32.812943 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:18:32.812906 4405 generic.go:356] "Generic (PLEG): container finished" podID="18ffb552-4917-4c03-a3b5-aa9c6a89f0e7" containerID="f135664a6bfe876b38d0757d58e9bcb5949fb9a20d7ffc8fad582c1c859c5ee7" exitCode=0 Nov 04 03:18:32.813155 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:18:32.812977 4405 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w4-35df3e-openshift-default-6c585b76f-vl6wz" Nov 04 03:18:32.813155 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:18:32.812990 4405 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-35df3e-openshift-default-6c585b76f-vl6wz" event={"ID":"18ffb552-4917-4c03-a3b5-aa9c6a89f0e7","Type":"ContainerDied","Data":"f135664a6bfe876b38d0757d58e9bcb5949fb9a20d7ffc8fad582c1c859c5ee7"} Nov 04 03:18:32.813155 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:18:32.813047 4405 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-35df3e-openshift-default-6c585b76f-vl6wz" event={"ID":"18ffb552-4917-4c03-a3b5-aa9c6a89f0e7","Type":"ContainerDied","Data":"97fb8ff966e8335a2d910a8a5b467f8f5867f0880723f76a2a55ed2254797200"} Nov 04 03:18:32.813155 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:18:32.813069 4405 scope.go:122] "RemoveContainer" containerID="f135664a6bfe876b38d0757d58e9bcb5949fb9a20d7ffc8fad582c1c859c5ee7" Nov 04 03:18:32.824064 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:18:32.824037 4405 scope.go:122] "RemoveContainer" containerID="9b9bed2e07dc5af1f69f981953d14d27e67d46f16bd7a3d70d9975fe219dfc75" Nov 04 03:18:32.834457 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:18:32.834431 4405 scope.go:122] "RemoveContainer" containerID="f135664a6bfe876b38d0757d58e9bcb5949fb9a20d7ffc8fad582c1c859c5ee7" Nov 04 03:18:32.834777 ip-10-0-134-125 kubenswrapper[4405]: E1104 03:18:32.834757 4405 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f135664a6bfe876b38d0757d58e9bcb5949fb9a20d7ffc8fad582c1c859c5ee7\": container with ID starting with f135664a6bfe876b38d0757d58e9bcb5949fb9a20d7ffc8fad582c1c859c5ee7 not found: ID does not exist" containerID="f135664a6bfe876b38d0757d58e9bcb5949fb9a20d7ffc8fad582c1c859c5ee7" Nov 04 03:18:32.834877 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:18:32.834792 4405 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f135664a6bfe876b38d0757d58e9bcb5949fb9a20d7ffc8fad582c1c859c5ee7"} err="failed to get container status \"f135664a6bfe876b38d0757d58e9bcb5949fb9a20d7ffc8fad582c1c859c5ee7\": rpc error: code = NotFound desc = could not find container \"f135664a6bfe876b38d0757d58e9bcb5949fb9a20d7ffc8fad582c1c859c5ee7\": container with ID starting with f135664a6bfe876b38d0757d58e9bcb5949fb9a20d7ffc8fad582c1c859c5ee7 not found: ID does not exist" Nov 04 03:18:32.834877 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:18:32.834819 4405 scope.go:122] "RemoveContainer" containerID="9b9bed2e07dc5af1f69f981953d14d27e67d46f16bd7a3d70d9975fe219dfc75" Nov 04 03:18:32.835133 ip-10-0-134-125 kubenswrapper[4405]: E1104 03:18:32.835116 4405 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9b9bed2e07dc5af1f69f981953d14d27e67d46f16bd7a3d70d9975fe219dfc75\": container with ID starting with 9b9bed2e07dc5af1f69f981953d14d27e67d46f16bd7a3d70d9975fe219dfc75 not found: ID does not exist" containerID="9b9bed2e07dc5af1f69f981953d14d27e67d46f16bd7a3d70d9975fe219dfc75" Nov 04 03:18:32.835191 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:18:32.835142 4405 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9b9bed2e07dc5af1f69f981953d14d27e67d46f16bd7a3d70d9975fe219dfc75"} err="failed to get container status \"9b9bed2e07dc5af1f69f981953d14d27e67d46f16bd7a3d70d9975fe219dfc75\": rpc error: code = NotFound desc = could not find container \"9b9bed2e07dc5af1f69f981953d14d27e67d46f16bd7a3d70d9975fe219dfc75\": container with ID starting with 9b9bed2e07dc5af1f69f981953d14d27e67d46f16bd7a3d70d9975fe219dfc75 not found: ID does not exist" Nov 04 03:18:32.959468 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:18:32.959380 4405 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-35df3e-openshift-default-6c585b76f-vl6wz"] Nov 04 03:18:32.962974 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:18:32.962945 4405 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-35df3e-openshift-default-6c585b76f-vl6wz"] Nov 04 03:18:33.699967 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:18:33.699931 4405 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="18ffb552-4917-4c03-a3b5-aa9c6a89f0e7" path="/var/lib/kubelet/pods/18ffb552-4917-4c03-a3b5-aa9c6a89f0e7/volumes" Nov 04 03:18:36.882169 ip-10-0-134-125 kubenswrapper[4405]: E1104 03:18:36.882131 4405 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5f2vb" containerName="extractor" Nov 04 03:18:39.022076 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:18:39.022045 4405 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-bl5mg"] Nov 04 03:18:47.751724 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:18:47.751660 4405 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w4-203c35/simulator-w4-203c35-kserve-f9c4bd695-999ns" podUID="eb0d8bdd-6808-4fd0-b9b5-567dff3e7ca5" containerName="main" containerID="cri-o://829ab80bf344e6089721bbc516ad81919e0c2be8bbe7a1596cd5c5db213f1adb" gracePeriod=45 Nov 04 03:18:47.883984 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:18:47.883950 4405 generic.go:356] "Generic (PLEG): container finished" podID="eb0d8bdd-6808-4fd0-b9b5-567dff3e7ca5" containerID="829ab80bf344e6089721bbc516ad81919e0c2be8bbe7a1596cd5c5db213f1adb" exitCode=0 Nov 04 03:18:47.884130 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:18:47.884003 4405 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-203c35/simulator-w4-203c35-kserve-f9c4bd695-999ns" event={"ID":"eb0d8bdd-6808-4fd0-b9b5-567dff3e7ca5","Type":"ContainerDied","Data":"829ab80bf344e6089721bbc516ad81919e0c2be8bbe7a1596cd5c5db213f1adb"} Nov 04 03:18:48.005438 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:18:48.005376 4405 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-203c35/simulator-w4-203c35-kserve-f9c4bd695-999ns" Nov 04 03:18:48.074767 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:18:48.074732 4405 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/eb0d8bdd-6808-4fd0-b9b5-567dff3e7ca5-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/eb0d8bdd-6808-4fd0-b9b5-567dff3e7ca5-model-cache\") pod \"eb0d8bdd-6808-4fd0-b9b5-567dff3e7ca5\" (UID: \"eb0d8bdd-6808-4fd0-b9b5-567dff3e7ca5\") " Nov 04 03:18:48.074928 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:18:48.074822 4405 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/eb0d8bdd-6808-4fd0-b9b5-567dff3e7ca5-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/eb0d8bdd-6808-4fd0-b9b5-567dff3e7ca5-tmp-dir\") pod \"eb0d8bdd-6808-4fd0-b9b5-567dff3e7ca5\" (UID: \"eb0d8bdd-6808-4fd0-b9b5-567dff3e7ca5\") " Nov 04 03:18:48.074928 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:18:48.074888 4405 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/eb0d8bdd-6808-4fd0-b9b5-567dff3e7ca5-home\" (UniqueName: \"kubernetes.io/empty-dir/eb0d8bdd-6808-4fd0-b9b5-567dff3e7ca5-home\") pod \"eb0d8bdd-6808-4fd0-b9b5-567dff3e7ca5\" (UID: \"eb0d8bdd-6808-4fd0-b9b5-567dff3e7ca5\") " Nov 04 03:18:48.075048 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:18:48.074933 4405 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/eb0d8bdd-6808-4fd0-b9b5-567dff3e7ca5-tls-certs\" (UniqueName: \"kubernetes.io/secret/eb0d8bdd-6808-4fd0-b9b5-567dff3e7ca5-tls-certs\") pod \"eb0d8bdd-6808-4fd0-b9b5-567dff3e7ca5\" (UID: \"eb0d8bdd-6808-4fd0-b9b5-567dff3e7ca5\") " Nov 04 03:18:48.075048 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:18:48.074974 4405 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/eb0d8bdd-6808-4fd0-b9b5-567dff3e7ca5-dshm\" (UniqueName: \"kubernetes.io/empty-dir/eb0d8bdd-6808-4fd0-b9b5-567dff3e7ca5-dshm\") pod \"eb0d8bdd-6808-4fd0-b9b5-567dff3e7ca5\" (UID: \"eb0d8bdd-6808-4fd0-b9b5-567dff3e7ca5\") " Nov 04 03:18:48.075048 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:18:48.075028 4405 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/eb0d8bdd-6808-4fd0-b9b5-567dff3e7ca5-kube-api-access-kqsr7\" (UniqueName: \"kubernetes.io/projected/eb0d8bdd-6808-4fd0-b9b5-567dff3e7ca5-kube-api-access-kqsr7\") pod \"eb0d8bdd-6808-4fd0-b9b5-567dff3e7ca5\" (UID: \"eb0d8bdd-6808-4fd0-b9b5-567dff3e7ca5\") " Nov 04 03:18:48.075209 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:18:48.074974 4405 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eb0d8bdd-6808-4fd0-b9b5-567dff3e7ca5-model-cache" pod "eb0d8bdd-6808-4fd0-b9b5-567dff3e7ca5" (UID: "eb0d8bdd-6808-4fd0-b9b5-567dff3e7ca5"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 03:18:48.075270 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:18:48.075209 4405 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eb0d8bdd-6808-4fd0-b9b5-567dff3e7ca5-home" pod "eb0d8bdd-6808-4fd0-b9b5-567dff3e7ca5" (UID: "eb0d8bdd-6808-4fd0-b9b5-567dff3e7ca5"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 03:18:48.075270 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:18:48.075121 4405 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eb0d8bdd-6808-4fd0-b9b5-567dff3e7ca5-tmp-dir" pod "eb0d8bdd-6808-4fd0-b9b5-567dff3e7ca5" (UID: "eb0d8bdd-6808-4fd0-b9b5-567dff3e7ca5"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 03:18:48.075377 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:18:48.075365 4405 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/eb0d8bdd-6808-4fd0-b9b5-567dff3e7ca5-tmp-dir\") on node \"ip-10-0-134-125.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:18:48.075432 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:18:48.075384 4405 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/eb0d8bdd-6808-4fd0-b9b5-567dff3e7ca5-home\") on node \"ip-10-0-134-125.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:18:48.075432 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:18:48.075399 4405 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/eb0d8bdd-6808-4fd0-b9b5-567dff3e7ca5-model-cache\") on node \"ip-10-0-134-125.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:18:48.077238 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:18:48.077211 4405 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eb0d8bdd-6808-4fd0-b9b5-567dff3e7ca5-kube-api-access-kqsr7" pod "eb0d8bdd-6808-4fd0-b9b5-567dff3e7ca5" (UID: "eb0d8bdd-6808-4fd0-b9b5-567dff3e7ca5"). InnerVolumeSpecName "kube-api-access-kqsr7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 03:18:48.077813 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:18:48.077786 4405 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/eb0d8bdd-6808-4fd0-b9b5-567dff3e7ca5-tls-certs" pod "eb0d8bdd-6808-4fd0-b9b5-567dff3e7ca5" (UID: "eb0d8bdd-6808-4fd0-b9b5-567dff3e7ca5"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 04 03:18:48.077918 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:18:48.077861 4405 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eb0d8bdd-6808-4fd0-b9b5-567dff3e7ca5-dshm" pod "eb0d8bdd-6808-4fd0-b9b5-567dff3e7ca5" (UID: "eb0d8bdd-6808-4fd0-b9b5-567dff3e7ca5"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 03:18:48.176024 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:18:48.175986 4405 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/eb0d8bdd-6808-4fd0-b9b5-567dff3e7ca5-tls-certs\") on node \"ip-10-0-134-125.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:18:48.176024 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:18:48.176018 4405 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/eb0d8bdd-6808-4fd0-b9b5-567dff3e7ca5-dshm\") on node \"ip-10-0-134-125.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:18:48.176024 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:18:48.176031 4405 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kqsr7\" (UniqueName: \"kubernetes.io/projected/eb0d8bdd-6808-4fd0-b9b5-567dff3e7ca5-kube-api-access-kqsr7\") on node \"ip-10-0-134-125.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:18:48.900523 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:18:48.900477 4405 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-203c35/simulator-w4-203c35-kserve-f9c4bd695-999ns" event={"ID":"eb0d8bdd-6808-4fd0-b9b5-567dff3e7ca5","Type":"ContainerDied","Data":"62a9f44f8d8642c7d9d62f72a0791919c0889270b4cef102a86eb5dcf211039b"} Nov 04 03:18:48.900523 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:18:48.900509 4405 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-203c35/simulator-w4-203c35-kserve-f9c4bd695-999ns" Nov 04 03:18:48.901111 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:18:48.900546 4405 scope.go:122] "RemoveContainer" containerID="829ab80bf344e6089721bbc516ad81919e0c2be8bbe7a1596cd5c5db213f1adb" Nov 04 03:18:48.968200 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:18:48.968152 4405 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-203c35/simulator-w4-203c35-kserve-f9c4bd695-999ns" podUID="eb0d8bdd-6808-4fd0-b9b5-567dff3e7ca5" containerName="main" probeResult="failure" output="Get \"https://10.134.0.46:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 03:18:49.065618 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:18:49.065581 4405 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-203c35/simulator-w4-203c35-kserve-f9c4bd695-999ns"] Nov 04 03:18:49.069627 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:18:49.069601 4405 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w4-203c35/simulator-w4-203c35-kserve-f9c4bd695-999ns"] Nov 04 03:18:49.699498 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:18:49.699466 4405 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="eb0d8bdd-6808-4fd0-b9b5-567dff3e7ca5" path="/var/lib/kubelet/pods/eb0d8bdd-6808-4fd0-b9b5-567dff3e7ca5/volumes" Nov 04 03:18:52.559771 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:18:52.559737 4405 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-203c35-854575bcff-vvzfb"] Nov 04 03:18:52.560201 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:18:52.559974 4405 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-203c35-854575bcff-vvzfb" podUID="abcee031-4335-4e23-8ae7-7cd526a633d0" containerName="maas-api" containerID="cri-o://5dae8546b8234161cd91bab1f9efb2fb6f241050981e8cd284615e06cee89a04" gracePeriod=30 Nov 04 03:18:52.804716 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:18:52.804692 4405 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-203c35-854575bcff-vvzfb" Nov 04 03:18:52.920114 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:18:52.920083 4405 generic.go:356] "Generic (PLEG): container finished" podID="abcee031-4335-4e23-8ae7-7cd526a633d0" containerID="5dae8546b8234161cd91bab1f9efb2fb6f241050981e8cd284615e06cee89a04" exitCode=0 Nov 04 03:18:52.920306 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:18:52.920132 4405 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-203c35-854575bcff-vvzfb" Nov 04 03:18:52.920306 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:18:52.920166 4405 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-203c35-854575bcff-vvzfb" event={"ID":"abcee031-4335-4e23-8ae7-7cd526a633d0","Type":"ContainerDied","Data":"5dae8546b8234161cd91bab1f9efb2fb6f241050981e8cd284615e06cee89a04"} Nov 04 03:18:52.920306 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:18:52.920220 4405 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-203c35-854575bcff-vvzfb" event={"ID":"abcee031-4335-4e23-8ae7-7cd526a633d0","Type":"ContainerDied","Data":"15270c53ac73208220376ae3dbd9ad878be87624bddbe4eb607f0d83dae51df6"} Nov 04 03:18:52.920306 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:18:52.920242 4405 scope.go:122] "RemoveContainer" containerID="5dae8546b8234161cd91bab1f9efb2fb6f241050981e8cd284615e06cee89a04" Nov 04 03:18:52.921132 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:18:52.921109 4405 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/abcee031-4335-4e23-8ae7-7cd526a633d0-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/abcee031-4335-4e23-8ae7-7cd526a633d0-openshift-service-ca\") pod \"abcee031-4335-4e23-8ae7-7cd526a633d0\" (UID: \"abcee031-4335-4e23-8ae7-7cd526a633d0\") " Nov 04 03:18:52.921294 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:18:52.921204 4405 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/abcee031-4335-4e23-8ae7-7cd526a633d0-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/abcee031-4335-4e23-8ae7-7cd526a633d0-maas-api-tls\") pod \"abcee031-4335-4e23-8ae7-7cd526a633d0\" (UID: \"abcee031-4335-4e23-8ae7-7cd526a633d0\") " Nov 04 03:18:52.921294 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:18:52.921244 4405 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/abcee031-4335-4e23-8ae7-7cd526a633d0-metrics-certs\" (UniqueName: \"kubernetes.io/secret/abcee031-4335-4e23-8ae7-7cd526a633d0-metrics-certs\") pod \"abcee031-4335-4e23-8ae7-7cd526a633d0\" (UID: \"abcee031-4335-4e23-8ae7-7cd526a633d0\") " Nov 04 03:18:52.921730 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:18:52.921439 4405 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/abcee031-4335-4e23-8ae7-7cd526a633d0-kube-api-access-9ppdv\" (UniqueName: \"kubernetes.io/projected/abcee031-4335-4e23-8ae7-7cd526a633d0-kube-api-access-9ppdv\") pod \"abcee031-4335-4e23-8ae7-7cd526a633d0\" (UID: \"abcee031-4335-4e23-8ae7-7cd526a633d0\") " Nov 04 03:18:52.921730 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:18:52.921550 4405 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/abcee031-4335-4e23-8ae7-7cd526a633d0-openshift-service-ca" pod "abcee031-4335-4e23-8ae7-7cd526a633d0" (UID: "abcee031-4335-4e23-8ae7-7cd526a633d0"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 03:18:52.922309 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:18:52.921999 4405 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/abcee031-4335-4e23-8ae7-7cd526a633d0-openshift-service-ca\") on node \"ip-10-0-134-125.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:18:52.924224 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:18:52.924186 4405 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/abcee031-4335-4e23-8ae7-7cd526a633d0-metrics-certs" pod "abcee031-4335-4e23-8ae7-7cd526a633d0" (UID: "abcee031-4335-4e23-8ae7-7cd526a633d0"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 04 03:18:52.928828 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:18:52.928789 4405 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/abcee031-4335-4e23-8ae7-7cd526a633d0-kube-api-access-9ppdv" pod "abcee031-4335-4e23-8ae7-7cd526a633d0" (UID: "abcee031-4335-4e23-8ae7-7cd526a633d0"). InnerVolumeSpecName "kube-api-access-9ppdv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 03:18:52.930039 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:18:52.930013 4405 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/abcee031-4335-4e23-8ae7-7cd526a633d0-maas-api-tls" pod "abcee031-4335-4e23-8ae7-7cd526a633d0" (UID: "abcee031-4335-4e23-8ae7-7cd526a633d0"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 04 03:18:52.949626 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:18:52.949601 4405 scope.go:122] "RemoveContainer" containerID="5dae8546b8234161cd91bab1f9efb2fb6f241050981e8cd284615e06cee89a04" Nov 04 03:18:52.949985 ip-10-0-134-125 kubenswrapper[4405]: E1104 03:18:52.949965 4405 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5dae8546b8234161cd91bab1f9efb2fb6f241050981e8cd284615e06cee89a04\": container with ID starting with 5dae8546b8234161cd91bab1f9efb2fb6f241050981e8cd284615e06cee89a04 not found: ID does not exist" containerID="5dae8546b8234161cd91bab1f9efb2fb6f241050981e8cd284615e06cee89a04" Nov 04 03:18:52.950059 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:18:52.949997 4405 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5dae8546b8234161cd91bab1f9efb2fb6f241050981e8cd284615e06cee89a04"} err="failed to get container status \"5dae8546b8234161cd91bab1f9efb2fb6f241050981e8cd284615e06cee89a04\": rpc error: code = NotFound desc = could not find container \"5dae8546b8234161cd91bab1f9efb2fb6f241050981e8cd284615e06cee89a04\": container with ID starting with 5dae8546b8234161cd91bab1f9efb2fb6f241050981e8cd284615e06cee89a04 not found: ID does not exist" Nov 04 03:18:53.022783 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:18:53.022753 4405 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/abcee031-4335-4e23-8ae7-7cd526a633d0-maas-api-tls\") on node \"ip-10-0-134-125.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:18:53.022783 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:18:53.022780 4405 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/abcee031-4335-4e23-8ae7-7cd526a633d0-metrics-certs\") on node \"ip-10-0-134-125.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:18:53.022783 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:18:53.022791 4405 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9ppdv\" (UniqueName: \"kubernetes.io/projected/abcee031-4335-4e23-8ae7-7cd526a633d0-kube-api-access-9ppdv\") on node \"ip-10-0-134-125.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:18:53.339647 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:18:53.339608 4405 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-203c35-854575bcff-vvzfb"] Nov 04 03:18:53.343551 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:18:53.343515 4405 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-203c35-854575bcff-vvzfb"] Nov 04 03:18:53.699550 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:18:53.699512 4405 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="abcee031-4335-4e23-8ae7-7cd526a633d0" path="/var/lib/kubelet/pods/abcee031-4335-4e23-8ae7-7cd526a633d0/volumes" Nov 04 03:18:58.641380 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:18:58.641342 4405 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-bl5mg"] Nov 04 03:19:06.882365 ip-10-0-134-125 kubenswrapper[4405]: E1104 03:19:06.882322 4405 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5f2vb" containerName="extractor" Nov 04 03:19:36.882533 ip-10-0-134-125 kubenswrapper[4405]: E1104 03:19:36.882496 4405 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5f2vb" containerName="extractor" Nov 04 03:19:43.750152 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:19:43.750111 4405 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-m7vks"] Nov 04 03:19:43.750731 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:19:43.750465 4405 cpu_manager.go:414] "containerMap: removing container" podUID="18ffb552-4917-4c03-a3b5-aa9c6a89f0e7" containerName="istio-proxy" Nov 04 03:19:43.750731 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:19:43.750476 4405 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="18ffb552-4917-4c03-a3b5-aa9c6a89f0e7" containerName="istio-proxy" Nov 04 03:19:43.750731 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:19:43.750485 4405 cpu_manager.go:414] "containerMap: removing container" podUID="d0362f05-c556-4c71-9b54-b360072d0758" containerName="istio-proxy" Nov 04 03:19:43.750731 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:19:43.750491 4405 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d0362f05-c556-4c71-9b54-b360072d0758" containerName="istio-proxy" Nov 04 03:19:43.750731 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:19:43.750505 4405 cpu_manager.go:414] "containerMap: removing container" podUID="317d0b2a-1ab0-4278-add3-cfaee93aa69c" containerName="payload-pre-processing" Nov 04 03:19:43.750731 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:19:43.750511 4405 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="317d0b2a-1ab0-4278-add3-cfaee93aa69c" containerName="payload-pre-processing" Nov 04 03:19:43.750731 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:19:43.750519 4405 cpu_manager.go:414] "containerMap: removing container" podUID="18ffb552-4917-4c03-a3b5-aa9c6a89f0e7" containerName="istio-proxy" Nov 04 03:19:43.750731 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:19:43.750524 4405 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="18ffb552-4917-4c03-a3b5-aa9c6a89f0e7" containerName="istio-proxy" Nov 04 03:19:43.750731 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:19:43.750532 4405 cpu_manager.go:414] "containerMap: removing container" podUID="c75d3a55-98c5-4af1-96ff-596d556597b9" containerName="maas-api" Nov 04 03:19:43.750731 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:19:43.750537 4405 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c75d3a55-98c5-4af1-96ff-596d556597b9" containerName="maas-api" Nov 04 03:19:43.750731 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:19:43.750543 4405 cpu_manager.go:414] "containerMap: removing container" podUID="652d018b-8d07-49ae-b9fb-a87cfc70b0c8" containerName="istio-proxy" Nov 04 03:19:43.750731 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:19:43.750547 4405 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="652d018b-8d07-49ae-b9fb-a87cfc70b0c8" containerName="istio-proxy" Nov 04 03:19:43.750731 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:19:43.750561 4405 cpu_manager.go:414] "containerMap: removing container" podUID="abcee031-4335-4e23-8ae7-7cd526a633d0" containerName="maas-api" Nov 04 03:19:43.750731 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:19:43.750565 4405 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="abcee031-4335-4e23-8ae7-7cd526a633d0" containerName="maas-api" Nov 04 03:19:43.750731 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:19:43.750575 4405 cpu_manager.go:414] "containerMap: removing container" podUID="eb0d8bdd-6808-4fd0-b9b5-567dff3e7ca5" containerName="main" Nov 04 03:19:43.750731 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:19:43.750579 4405 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="eb0d8bdd-6808-4fd0-b9b5-567dff3e7ca5" containerName="main" Nov 04 03:19:43.750731 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:19:43.750585 4405 cpu_manager.go:414] "containerMap: removing container" podUID="652d018b-8d07-49ae-b9fb-a87cfc70b0c8" containerName="istio-proxy" Nov 04 03:19:43.750731 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:19:43.750589 4405 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="652d018b-8d07-49ae-b9fb-a87cfc70b0c8" containerName="istio-proxy" Nov 04 03:19:43.750731 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:19:43.750595 4405 cpu_manager.go:414] "containerMap: removing container" podUID="8d7f28e3-d62b-4813-81e3-364d4700983a" containerName="payload-processing" Nov 04 03:19:43.750731 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:19:43.750600 4405 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8d7f28e3-d62b-4813-81e3-364d4700983a" containerName="payload-processing" Nov 04 03:19:43.750731 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:19:43.750655 4405 memory_manager.go:365] "RemoveStaleState removing state" podUID="317d0b2a-1ab0-4278-add3-cfaee93aa69c" containerName="payload-pre-processing" Nov 04 03:19:43.750731 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:19:43.750678 4405 memory_manager.go:365] "RemoveStaleState removing state" podUID="18ffb552-4917-4c03-a3b5-aa9c6a89f0e7" containerName="istio-proxy" Nov 04 03:19:43.750731 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:19:43.750685 4405 memory_manager.go:365] "RemoveStaleState removing state" podUID="c75d3a55-98c5-4af1-96ff-596d556597b9" containerName="maas-api" Nov 04 03:19:43.750731 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:19:43.750691 4405 memory_manager.go:365] "RemoveStaleState removing state" podUID="652d018b-8d07-49ae-b9fb-a87cfc70b0c8" containerName="istio-proxy" Nov 04 03:19:43.750731 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:19:43.750698 4405 memory_manager.go:365] "RemoveStaleState removing state" podUID="abcee031-4335-4e23-8ae7-7cd526a633d0" containerName="maas-api" Nov 04 03:19:43.750731 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:19:43.750705 4405 memory_manager.go:365] "RemoveStaleState removing state" podUID="eb0d8bdd-6808-4fd0-b9b5-567dff3e7ca5" containerName="main" Nov 04 03:19:43.750731 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:19:43.750710 4405 memory_manager.go:365] "RemoveStaleState removing state" podUID="652d018b-8d07-49ae-b9fb-a87cfc70b0c8" containerName="istio-proxy" Nov 04 03:19:43.750731 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:19:43.750715 4405 memory_manager.go:365] "RemoveStaleState removing state" podUID="8d7f28e3-d62b-4813-81e3-364d4700983a" containerName="payload-processing" Nov 04 03:19:43.750731 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:19:43.750721 4405 memory_manager.go:365] "RemoveStaleState removing state" podUID="d0362f05-c556-4c71-9b54-b360072d0758" containerName="istio-proxy" Nov 04 03:19:43.754769 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:19:43.754751 4405 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-m7vks" Nov 04 03:19:43.798729 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:19:43.798686 4405 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bmgll\" (UniqueName: \"kubernetes.io/projected/88f03036-1fd4-4370-a32a-b5c3d316fcad-kube-api-access-bmgll\") pod \"maas-api-revoke-sub-85237e45-m7vks\" (UID: \"88f03036-1fd4-4370-a32a-b5c3d316fcad\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-m7vks" Nov 04 03:19:43.798890 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:19:43.798857 4405 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/88f03036-1fd4-4370-a32a-b5c3d316fcad-maas-api-xks-ca\") pod \"maas-api-revoke-sub-85237e45-m7vks\" (UID: \"88f03036-1fd4-4370-a32a-b5c3d316fcad\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-m7vks" Nov 04 03:19:43.798890 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:19:43.798886 4405 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/88f03036-1fd4-4370-a32a-b5c3d316fcad-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-85237e45-m7vks\" (UID: \"88f03036-1fd4-4370-a32a-b5c3d316fcad\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-m7vks" Nov 04 03:19:43.858839 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:19:43.858802 4405 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 04 03:19:43.859042 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:19:43.858846 4405 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 04 03:19:43.859042 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:19:43.858851 4405 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 04 03:19:43.859042 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:19:43.858928 4405 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-slbq8\"" Nov 04 03:19:43.860548 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:19:43.860527 4405 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-m7vks"] Nov 04 03:19:43.900005 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:19:43.899977 4405 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bmgll\" (UniqueName: \"kubernetes.io/projected/88f03036-1fd4-4370-a32a-b5c3d316fcad-kube-api-access-bmgll\") pod \"maas-api-revoke-sub-85237e45-m7vks\" (UID: \"88f03036-1fd4-4370-a32a-b5c3d316fcad\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-m7vks" Nov 04 03:19:43.900172 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:19:43.900044 4405 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/88f03036-1fd4-4370-a32a-b5c3d316fcad-maas-api-xks-ca\") pod \"maas-api-revoke-sub-85237e45-m7vks\" (UID: \"88f03036-1fd4-4370-a32a-b5c3d316fcad\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-m7vks" Nov 04 03:19:43.900172 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:19:43.900062 4405 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/88f03036-1fd4-4370-a32a-b5c3d316fcad-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-85237e45-m7vks\" (UID: \"88f03036-1fd4-4370-a32a-b5c3d316fcad\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-m7vks" Nov 04 03:19:43.900659 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:19:43.900643 4405 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/88f03036-1fd4-4370-a32a-b5c3d316fcad-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-85237e45-m7vks\" (UID: \"88f03036-1fd4-4370-a32a-b5c3d316fcad\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-m7vks" Nov 04 03:19:43.902628 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:19:43.902605 4405 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/88f03036-1fd4-4370-a32a-b5c3d316fcad-maas-api-xks-ca\") pod \"maas-api-revoke-sub-85237e45-m7vks\" (UID: \"88f03036-1fd4-4370-a32a-b5c3d316fcad\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-m7vks" Nov 04 03:19:43.961349 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:19:43.961312 4405 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bmgll\" (UniqueName: \"kubernetes.io/projected/88f03036-1fd4-4370-a32a-b5c3d316fcad-kube-api-access-bmgll\") pod \"maas-api-revoke-sub-85237e45-m7vks\" (UID: \"88f03036-1fd4-4370-a32a-b5c3d316fcad\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-m7vks" Nov 04 03:19:44.068227 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:19:44.068136 4405 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-m7vks" Nov 04 03:19:44.442478 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:19:44.442362 4405 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-m7vks"] Nov 04 03:19:44.445506 ip-10-0-134-125 kubenswrapper[4405]: W1104 03:19:44.445476 4405 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod88f03036_1fd4_4370_a32a_b5c3d316fcad.slice/crio-0fd28b167c181828fc9a391a6aab8cfc03f10004768964229142c7af144e40a4 WatchSource:0}: Error finding container 0fd28b167c181828fc9a391a6aab8cfc03f10004768964229142c7af144e40a4: Status 404 returned error can't find the container with id 0fd28b167c181828fc9a391a6aab8cfc03f10004768964229142c7af144e40a4 Nov 04 03:19:45.162962 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:19:45.162876 4405 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-m7vks" event={"ID":"88f03036-1fd4-4370-a32a-b5c3d316fcad","Type":"ContainerStarted","Data":"7ca16528c075b47dc5cabc4e420d8d8867c1312606b728cc04be8a3f819d3eb5"} Nov 04 03:19:45.162962 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:19:45.162935 4405 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-m7vks" event={"ID":"88f03036-1fd4-4370-a32a-b5c3d316fcad","Type":"ContainerStarted","Data":"0fd28b167c181828fc9a391a6aab8cfc03f10004768964229142c7af144e40a4"} Nov 04 03:19:45.279331 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:19:45.279274 4405 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-m7vks" podStartSLOduration=2.279258682 podStartE2EDuration="2.279258682s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 03:19:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 03:19:45.278229713 +0000 UTC m=+2831.708929835" watchObservedRunningTime="2026-11-04 03:19:45.279258682 +0000 UTC m=+2831.709958798" Nov 04 03:19:46.169228 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:19:46.169191 4405 generic.go:356] "Generic (PLEG): container finished" podID="88f03036-1fd4-4370-a32a-b5c3d316fcad" containerID="7ca16528c075b47dc5cabc4e420d8d8867c1312606b728cc04be8a3f819d3eb5" exitCode=0 Nov 04 03:19:46.169625 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:19:46.169235 4405 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-m7vks" event={"ID":"88f03036-1fd4-4370-a32a-b5c3d316fcad","Type":"ContainerDied","Data":"7ca16528c075b47dc5cabc4e420d8d8867c1312606b728cc04be8a3f819d3eb5"} Nov 04 03:19:47.042487 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:19:47.042457 4405 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-76597d6855-xjqqw"] Nov 04 03:19:47.042925 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:19:47.042910 4405 memory_manager.go:365] "RemoveStaleState removing state" podUID="18ffb552-4917-4c03-a3b5-aa9c6a89f0e7" containerName="istio-proxy" Nov 04 03:19:47.045875 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:19:47.045860 4405 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-xjqqw" Nov 04 03:19:47.128150 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:19:47.128113 4405 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c97cb26c-6fc7-45db-9984-aa459ddb54df-workload-socket\") pod \"maas-default-gateway-openshift-default-76597d6855-xjqqw\" (UID: \"c97cb26c-6fc7-45db-9984-aa459ddb54df\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-xjqqw" Nov 04 03:19:47.128150 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:19:47.128153 4405 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jkgwn\" (UniqueName: \"kubernetes.io/projected/c97cb26c-6fc7-45db-9984-aa459ddb54df-kube-api-access-jkgwn\") pod \"maas-default-gateway-openshift-default-76597d6855-xjqqw\" (UID: \"c97cb26c-6fc7-45db-9984-aa459ddb54df\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-xjqqw" Nov 04 03:19:47.128356 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:19:47.128234 4405 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c97cb26c-6fc7-45db-9984-aa459ddb54df-istio-token\") pod \"maas-default-gateway-openshift-default-76597d6855-xjqqw\" (UID: \"c97cb26c-6fc7-45db-9984-aa459ddb54df\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-xjqqw" Nov 04 03:19:47.128356 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:19:47.128275 4405 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c97cb26c-6fc7-45db-9984-aa459ddb54df-workload-certs\") pod \"maas-default-gateway-openshift-default-76597d6855-xjqqw\" (UID: \"c97cb26c-6fc7-45db-9984-aa459ddb54df\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-xjqqw" Nov 04 03:19:47.128356 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:19:47.128305 4405 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c97cb26c-6fc7-45db-9984-aa459ddb54df-istio-data\") pod \"maas-default-gateway-openshift-default-76597d6855-xjqqw\" (UID: \"c97cb26c-6fc7-45db-9984-aa459ddb54df\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-xjqqw" Nov 04 03:19:47.128461 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:19:47.128422 4405 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c97cb26c-6fc7-45db-9984-aa459ddb54df-istio-podinfo\") pod \"maas-default-gateway-openshift-default-76597d6855-xjqqw\" (UID: \"c97cb26c-6fc7-45db-9984-aa459ddb54df\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-xjqqw" Nov 04 03:19:47.128496 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:19:47.128471 4405 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c97cb26c-6fc7-45db-9984-aa459ddb54df-istio-envoy\") pod \"maas-default-gateway-openshift-default-76597d6855-xjqqw\" (UID: \"c97cb26c-6fc7-45db-9984-aa459ddb54df\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-xjqqw" Nov 04 03:19:47.128532 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:19:47.128517 4405 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c97cb26c-6fc7-45db-9984-aa459ddb54df-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-76597d6855-xjqqw\" (UID: \"c97cb26c-6fc7-45db-9984-aa459ddb54df\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-xjqqw" Nov 04 03:19:47.128567 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:19:47.128548 4405 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c97cb26c-6fc7-45db-9984-aa459ddb54df-credential-socket\") pod \"maas-default-gateway-openshift-default-76597d6855-xjqqw\" (UID: \"c97cb26c-6fc7-45db-9984-aa459ddb54df\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-xjqqw" Nov 04 03:19:47.229497 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:19:47.229462 4405 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c97cb26c-6fc7-45db-9984-aa459ddb54df-workload-certs\") pod \"maas-default-gateway-openshift-default-76597d6855-xjqqw\" (UID: \"c97cb26c-6fc7-45db-9984-aa459ddb54df\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-xjqqw" Nov 04 03:19:47.229949 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:19:47.229514 4405 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c97cb26c-6fc7-45db-9984-aa459ddb54df-istio-data\") pod \"maas-default-gateway-openshift-default-76597d6855-xjqqw\" (UID: \"c97cb26c-6fc7-45db-9984-aa459ddb54df\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-xjqqw" Nov 04 03:19:47.229949 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:19:47.229579 4405 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c97cb26c-6fc7-45db-9984-aa459ddb54df-istio-podinfo\") pod \"maas-default-gateway-openshift-default-76597d6855-xjqqw\" (UID: \"c97cb26c-6fc7-45db-9984-aa459ddb54df\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-xjqqw" Nov 04 03:19:47.229949 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:19:47.229631 4405 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c97cb26c-6fc7-45db-9984-aa459ddb54df-istio-envoy\") pod \"maas-default-gateway-openshift-default-76597d6855-xjqqw\" (UID: \"c97cb26c-6fc7-45db-9984-aa459ddb54df\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-xjqqw" Nov 04 03:19:47.229949 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:19:47.229691 4405 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c97cb26c-6fc7-45db-9984-aa459ddb54df-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-76597d6855-xjqqw\" (UID: \"c97cb26c-6fc7-45db-9984-aa459ddb54df\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-xjqqw" Nov 04 03:19:47.230192 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:19:47.230167 4405 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c97cb26c-6fc7-45db-9984-aa459ddb54df-workload-certs\") pod \"maas-default-gateway-openshift-default-76597d6855-xjqqw\" (UID: \"c97cb26c-6fc7-45db-9984-aa459ddb54df\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-xjqqw" Nov 04 03:19:47.230297 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:19:47.230280 4405 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c97cb26c-6fc7-45db-9984-aa459ddb54df-credential-socket\") pod \"maas-default-gateway-openshift-default-76597d6855-xjqqw\" (UID: \"c97cb26c-6fc7-45db-9984-aa459ddb54df\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-xjqqw" Nov 04 03:19:47.230366 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:19:47.230327 4405 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c97cb26c-6fc7-45db-9984-aa459ddb54df-workload-socket\") pod \"maas-default-gateway-openshift-default-76597d6855-xjqqw\" (UID: \"c97cb26c-6fc7-45db-9984-aa459ddb54df\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-xjqqw" Nov 04 03:19:47.230366 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:19:47.230356 4405 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jkgwn\" (UniqueName: \"kubernetes.io/projected/c97cb26c-6fc7-45db-9984-aa459ddb54df-kube-api-access-jkgwn\") pod \"maas-default-gateway-openshift-default-76597d6855-xjqqw\" (UID: \"c97cb26c-6fc7-45db-9984-aa459ddb54df\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-xjqqw" Nov 04 03:19:47.230469 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:19:47.230413 4405 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c97cb26c-6fc7-45db-9984-aa459ddb54df-istio-token\") pod \"maas-default-gateway-openshift-default-76597d6855-xjqqw\" (UID: \"c97cb26c-6fc7-45db-9984-aa459ddb54df\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-xjqqw" Nov 04 03:19:47.230938 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:19:47.230914 4405 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c97cb26c-6fc7-45db-9984-aa459ddb54df-istio-data\") pod \"maas-default-gateway-openshift-default-76597d6855-xjqqw\" (UID: \"c97cb26c-6fc7-45db-9984-aa459ddb54df\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-xjqqw" Nov 04 03:19:47.233787 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:19:47.232956 4405 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c97cb26c-6fc7-45db-9984-aa459ddb54df-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-76597d6855-xjqqw\" (UID: \"c97cb26c-6fc7-45db-9984-aa459ddb54df\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-xjqqw" Nov 04 03:19:47.233787 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:19:47.233251 4405 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c97cb26c-6fc7-45db-9984-aa459ddb54df-credential-socket\") pod \"maas-default-gateway-openshift-default-76597d6855-xjqqw\" (UID: \"c97cb26c-6fc7-45db-9984-aa459ddb54df\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-xjqqw" Nov 04 03:19:47.233787 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:19:47.233483 4405 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c97cb26c-6fc7-45db-9984-aa459ddb54df-workload-socket\") pod \"maas-default-gateway-openshift-default-76597d6855-xjqqw\" (UID: \"c97cb26c-6fc7-45db-9984-aa459ddb54df\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-xjqqw" Nov 04 03:19:47.234713 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:19:47.234649 4405 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c97cb26c-6fc7-45db-9984-aa459ddb54df-istio-envoy\") pod \"maas-default-gateway-openshift-default-76597d6855-xjqqw\" (UID: \"c97cb26c-6fc7-45db-9984-aa459ddb54df\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-xjqqw" Nov 04 03:19:47.236162 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:19:47.236140 4405 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c97cb26c-6fc7-45db-9984-aa459ddb54df-istio-podinfo\") pod \"maas-default-gateway-openshift-default-76597d6855-xjqqw\" (UID: \"c97cb26c-6fc7-45db-9984-aa459ddb54df\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-xjqqw" Nov 04 03:19:47.257909 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:19:47.257814 4405 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-76597d6855-xjqqw"] Nov 04 03:19:47.291400 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:19:47.291335 4405 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c97cb26c-6fc7-45db-9984-aa459ddb54df-istio-token\") pod \"maas-default-gateway-openshift-default-76597d6855-xjqqw\" (UID: \"c97cb26c-6fc7-45db-9984-aa459ddb54df\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-xjqqw" Nov 04 03:19:47.293744 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:19:47.293638 4405 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jkgwn\" (UniqueName: \"kubernetes.io/projected/c97cb26c-6fc7-45db-9984-aa459ddb54df-kube-api-access-jkgwn\") pod \"maas-default-gateway-openshift-default-76597d6855-xjqqw\" (UID: \"c97cb26c-6fc7-45db-9984-aa459ddb54df\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-xjqqw" Nov 04 03:19:47.316235 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:19:47.316207 4405 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-m7vks" Nov 04 03:19:47.360075 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:19:47.360040 4405 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-xjqqw" Nov 04 03:19:47.433579 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:19:47.432133 4405 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/88f03036-1fd4-4370-a32a-b5c3d316fcad-kube-api-access-bmgll\" (UniqueName: \"kubernetes.io/projected/88f03036-1fd4-4370-a32a-b5c3d316fcad-kube-api-access-bmgll\") pod \"88f03036-1fd4-4370-a32a-b5c3d316fcad\" (UID: \"88f03036-1fd4-4370-a32a-b5c3d316fcad\") " Nov 04 03:19:47.433579 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:19:47.432227 4405 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/88f03036-1fd4-4370-a32a-b5c3d316fcad-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/88f03036-1fd4-4370-a32a-b5c3d316fcad-maas-api-openshift-ca\") pod \"88f03036-1fd4-4370-a32a-b5c3d316fcad\" (UID: \"88f03036-1fd4-4370-a32a-b5c3d316fcad\") " Nov 04 03:19:47.433579 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:19:47.432368 4405 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/88f03036-1fd4-4370-a32a-b5c3d316fcad-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/88f03036-1fd4-4370-a32a-b5c3d316fcad-maas-api-xks-ca\") pod \"88f03036-1fd4-4370-a32a-b5c3d316fcad\" (UID: \"88f03036-1fd4-4370-a32a-b5c3d316fcad\") " Nov 04 03:19:47.433579 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:19:47.433532 4405 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/88f03036-1fd4-4370-a32a-b5c3d316fcad-maas-api-openshift-ca" pod "88f03036-1fd4-4370-a32a-b5c3d316fcad" (UID: "88f03036-1fd4-4370-a32a-b5c3d316fcad"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 03:19:47.445221 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:19:47.445173 4405 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/88f03036-1fd4-4370-a32a-b5c3d316fcad-maas-api-xks-ca" pod "88f03036-1fd4-4370-a32a-b5c3d316fcad" (UID: "88f03036-1fd4-4370-a32a-b5c3d316fcad"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 04 03:19:47.447599 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:19:47.447565 4405 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/88f03036-1fd4-4370-a32a-b5c3d316fcad-kube-api-access-bmgll" pod "88f03036-1fd4-4370-a32a-b5c3d316fcad" (UID: "88f03036-1fd4-4370-a32a-b5c3d316fcad"). InnerVolumeSpecName "kube-api-access-bmgll". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 03:19:47.534058 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:19:47.534015 4405 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/88f03036-1fd4-4370-a32a-b5c3d316fcad-maas-api-xks-ca\") on node \"ip-10-0-134-125.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:19:47.534058 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:19:47.534055 4405 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bmgll\" (UniqueName: \"kubernetes.io/projected/88f03036-1fd4-4370-a32a-b5c3d316fcad-kube-api-access-bmgll\") on node \"ip-10-0-134-125.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:19:47.534058 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:19:47.534069 4405 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/88f03036-1fd4-4370-a32a-b5c3d316fcad-maas-api-openshift-ca\") on node \"ip-10-0-134-125.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:19:47.795057 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:19:47.795023 4405 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-76597d6855-xjqqw"] Nov 04 03:19:47.798199 ip-10-0-134-125 kubenswrapper[4405]: W1104 03:19:47.798172 4405 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc97cb26c_6fc7_45db_9984_aa459ddb54df.slice/crio-535194e6654e41e78cb03f7810fe4eff1d14841e5fb724ee14f14be52e843f2c WatchSource:0}: Error finding container 535194e6654e41e78cb03f7810fe4eff1d14841e5fb724ee14f14be52e843f2c: Status 404 returned error can't find the container with id 535194e6654e41e78cb03f7810fe4eff1d14841e5fb724ee14f14be52e843f2c Nov 04 03:19:47.800464 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:19:47.800431 4405 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 04 03:19:47.800556 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:19:47.800496 4405 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 04 03:19:47.800556 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:19:47.800525 4405 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 04 03:19:48.179679 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:19:48.179647 4405 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-m7vks" Nov 04 03:19:48.179872 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:19:48.179690 4405 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-m7vks" event={"ID":"88f03036-1fd4-4370-a32a-b5c3d316fcad","Type":"ContainerDied","Data":"0fd28b167c181828fc9a391a6aab8cfc03f10004768964229142c7af144e40a4"} Nov 04 03:19:48.179872 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:19:48.179752 4405 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0fd28b167c181828fc9a391a6aab8cfc03f10004768964229142c7af144e40a4" Nov 04 03:19:48.181382 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:19:48.181356 4405 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-xjqqw" event={"ID":"c97cb26c-6fc7-45db-9984-aa459ddb54df","Type":"ContainerStarted","Data":"290310fe3aa4f285216bf55b8179e936197ac9279ae4c7f8c8ee60994946023f"} Nov 04 03:19:48.181507 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:19:48.181399 4405 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-xjqqw" event={"ID":"c97cb26c-6fc7-45db-9984-aa459ddb54df","Type":"ContainerStarted","Data":"535194e6654e41e78cb03f7810fe4eff1d14841e5fb724ee14f14be52e843f2c"} Nov 04 03:19:48.361171 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:19:48.361136 4405 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-xjqqw" Nov 04 03:19:48.362756 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:19:48.362731 4405 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76597d6855-xjqqw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.138:15021/healthz/ready\": dial tcp 10.134.0.138:15021: connect: connection refused" start-of-body= Nov 04 03:19:48.362865 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:19:48.362791 4405 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-xjqqw" podUID="c97cb26c-6fc7-45db-9984-aa459ddb54df" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.138:15021/healthz/ready\": dial tcp 10.134.0.138:15021: connect: connection refused" Nov 04 03:19:48.368171 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:19:48.368124 4405 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-xjqqw" podStartSLOduration=2.36811124 podStartE2EDuration="2.36811124s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 03:19:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 03:19:48.351827326 +0000 UTC m=+2834.782527447" watchObservedRunningTime="2026-11-04 03:19:48.36811124 +0000 UTC m=+2834.798811419" Nov 04 03:19:49.360800 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:19:49.360748 4405 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76597d6855-xjqqw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.138:15021/healthz/ready\": dial tcp 10.134.0.138:15021: connect: connection refused" start-of-body= Nov 04 03:19:49.361039 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:19:49.360828 4405 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-xjqqw" podUID="c97cb26c-6fc7-45db-9984-aa459ddb54df" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.138:15021/healthz/ready\": dial tcp 10.134.0.138:15021: connect: connection refused" Nov 04 03:19:50.360836 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:19:50.360798 4405 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76597d6855-xjqqw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.138:15021/healthz/ready\": dial tcp 10.134.0.138:15021: connect: connection refused" start-of-body= Nov 04 03:19:50.361260 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:19:50.360862 4405 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-xjqqw" podUID="c97cb26c-6fc7-45db-9984-aa459ddb54df" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.138:15021/healthz/ready\": dial tcp 10.134.0.138:15021: connect: connection refused" Nov 04 03:19:51.361372 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:19:51.361336 4405 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76597d6855-xjqqw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.138:15021/healthz/ready\": dial tcp 10.134.0.138:15021: connect: connection refused" start-of-body= Nov 04 03:19:51.361776 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:19:51.361394 4405 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-xjqqw" podUID="c97cb26c-6fc7-45db-9984-aa459ddb54df" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.138:15021/healthz/ready\": dial tcp 10.134.0.138:15021: connect: connection refused" Nov 04 03:19:52.360913 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:19:52.360872 4405 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76597d6855-xjqqw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.138:15021/healthz/ready\": dial tcp 10.134.0.138:15021: connect: connection refused" start-of-body= Nov 04 03:19:52.361113 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:19:52.360933 4405 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-xjqqw" podUID="c97cb26c-6fc7-45db-9984-aa459ddb54df" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.138:15021/healthz/ready\": dial tcp 10.134.0.138:15021: connect: connection refused" Nov 04 03:19:53.361375 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:19:53.361334 4405 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76597d6855-xjqqw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.138:15021/healthz/ready\": dial tcp 10.134.0.138:15021: connect: connection refused" start-of-body= Nov 04 03:19:53.361847 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:19:53.361399 4405 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-xjqqw" podUID="c97cb26c-6fc7-45db-9984-aa459ddb54df" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.138:15021/healthz/ready\": dial tcp 10.134.0.138:15021: connect: connection refused" Nov 04 03:19:54.361022 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:19:54.360972 4405 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76597d6855-xjqqw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.138:15021/healthz/ready\": dial tcp 10.134.0.138:15021: connect: connection refused" start-of-body= Nov 04 03:19:54.361253 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:19:54.361028 4405 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-xjqqw" podUID="c97cb26c-6fc7-45db-9984-aa459ddb54df" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.138:15021/healthz/ready\": dial tcp 10.134.0.138:15021: connect: connection refused" Nov 04 03:19:55.361529 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:19:55.361490 4405 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76597d6855-xjqqw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.138:15021/healthz/ready\": dial tcp 10.134.0.138:15021: connect: connection refused" start-of-body= Nov 04 03:19:55.361986 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:19:55.361553 4405 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-xjqqw" podUID="c97cb26c-6fc7-45db-9984-aa459ddb54df" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.138:15021/healthz/ready\": dial tcp 10.134.0.138:15021: connect: connection refused" Nov 04 03:19:57.360708 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:19:57.360634 4405 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76597d6855-xjqqw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.138:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 04 03:19:57.361142 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:19:57.360736 4405 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-xjqqw" podUID="c97cb26c-6fc7-45db-9984-aa459ddb54df" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.138:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 04 03:19:58.401904 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:19:58.401865 4405 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76597d6855-xjqqw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.138:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 04 03:19:58.402344 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:19:58.401949 4405 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-xjqqw" podUID="c97cb26c-6fc7-45db-9984-aa459ddb54df" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.138:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 04 03:19:58.954030 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:19:58.953998 4405 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-xjqqw" Nov 04 03:19:58.954313 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:19:58.954288 4405 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-xjqqw" Nov 04 03:19:58.954994 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:19:58.954965 4405 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-xjqqw" Nov 04 03:20:02.145538 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:20:02.145499 4405 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-76597d6855-dqn2b"] Nov 04 03:20:02.145953 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:20:02.145909 4405 cpu_manager.go:414] "containerMap: removing container" podUID="88f03036-1fd4-4370-a32a-b5c3d316fcad" containerName="revoke-keys" Nov 04 03:20:02.145953 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:20:02.145921 4405 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="88f03036-1fd4-4370-a32a-b5c3d316fcad" containerName="revoke-keys" Nov 04 03:20:02.146046 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:20:02.145988 4405 memory_manager.go:365] "RemoveStaleState removing state" podUID="88f03036-1fd4-4370-a32a-b5c3d316fcad" containerName="revoke-keys" Nov 04 03:20:02.149342 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:20:02.149327 4405 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-dqn2b" Nov 04 03:20:02.254430 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:20:02.254392 4405 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-76597d6855-dqn2b"] Nov 04 03:20:02.298246 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:20:02.298203 4405 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/415bafea-577e-4fba-925e-eb2c492e3cca-istio-envoy\") pod \"maas-default-gateway-openshift-default-76597d6855-dqn2b\" (UID: \"415bafea-577e-4fba-925e-eb2c492e3cca\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-dqn2b" Nov 04 03:20:02.298400 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:20:02.298262 4405 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/415bafea-577e-4fba-925e-eb2c492e3cca-istio-token\") pod \"maas-default-gateway-openshift-default-76597d6855-dqn2b\" (UID: \"415bafea-577e-4fba-925e-eb2c492e3cca\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-dqn2b" Nov 04 03:20:02.298468 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:20:02.298394 4405 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/415bafea-577e-4fba-925e-eb2c492e3cca-istio-data\") pod \"maas-default-gateway-openshift-default-76597d6855-dqn2b\" (UID: \"415bafea-577e-4fba-925e-eb2c492e3cca\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-dqn2b" Nov 04 03:20:02.298468 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:20:02.298440 4405 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/415bafea-577e-4fba-925e-eb2c492e3cca-istio-podinfo\") pod \"maas-default-gateway-openshift-default-76597d6855-dqn2b\" (UID: \"415bafea-577e-4fba-925e-eb2c492e3cca\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-dqn2b" Nov 04 03:20:02.298468 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:20:02.298460 4405 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/415bafea-577e-4fba-925e-eb2c492e3cca-credential-socket\") pod \"maas-default-gateway-openshift-default-76597d6855-dqn2b\" (UID: \"415bafea-577e-4fba-925e-eb2c492e3cca\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-dqn2b" Nov 04 03:20:02.298622 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:20:02.298492 4405 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/415bafea-577e-4fba-925e-eb2c492e3cca-workload-certs\") pod \"maas-default-gateway-openshift-default-76597d6855-dqn2b\" (UID: \"415bafea-577e-4fba-925e-eb2c492e3cca\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-dqn2b" Nov 04 03:20:02.298622 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:20:02.298545 4405 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rvwh2\" (UniqueName: \"kubernetes.io/projected/415bafea-577e-4fba-925e-eb2c492e3cca-kube-api-access-rvwh2\") pod \"maas-default-gateway-openshift-default-76597d6855-dqn2b\" (UID: \"415bafea-577e-4fba-925e-eb2c492e3cca\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-dqn2b" Nov 04 03:20:02.298622 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:20:02.298596 4405 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/415bafea-577e-4fba-925e-eb2c492e3cca-workload-socket\") pod \"maas-default-gateway-openshift-default-76597d6855-dqn2b\" (UID: \"415bafea-577e-4fba-925e-eb2c492e3cca\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-dqn2b" Nov 04 03:20:02.298763 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:20:02.298717 4405 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/415bafea-577e-4fba-925e-eb2c492e3cca-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-76597d6855-dqn2b\" (UID: \"415bafea-577e-4fba-925e-eb2c492e3cca\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-dqn2b" Nov 04 03:20:02.399512 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:20:02.399414 4405 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/415bafea-577e-4fba-925e-eb2c492e3cca-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-76597d6855-dqn2b\" (UID: \"415bafea-577e-4fba-925e-eb2c492e3cca\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-dqn2b" Nov 04 03:20:02.399512 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:20:02.399468 4405 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/415bafea-577e-4fba-925e-eb2c492e3cca-istio-envoy\") pod \"maas-default-gateway-openshift-default-76597d6855-dqn2b\" (UID: \"415bafea-577e-4fba-925e-eb2c492e3cca\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-dqn2b" Nov 04 03:20:02.399512 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:20:02.399506 4405 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/415bafea-577e-4fba-925e-eb2c492e3cca-istio-token\") pod \"maas-default-gateway-openshift-default-76597d6855-dqn2b\" (UID: \"415bafea-577e-4fba-925e-eb2c492e3cca\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-dqn2b" Nov 04 03:20:02.399812 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:20:02.399536 4405 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/415bafea-577e-4fba-925e-eb2c492e3cca-istio-data\") pod \"maas-default-gateway-openshift-default-76597d6855-dqn2b\" (UID: \"415bafea-577e-4fba-925e-eb2c492e3cca\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-dqn2b" Nov 04 03:20:02.399812 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:20:02.399736 4405 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/415bafea-577e-4fba-925e-eb2c492e3cca-istio-podinfo\") pod \"maas-default-gateway-openshift-default-76597d6855-dqn2b\" (UID: \"415bafea-577e-4fba-925e-eb2c492e3cca\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-dqn2b" Nov 04 03:20:02.399812 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:20:02.399791 4405 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/415bafea-577e-4fba-925e-eb2c492e3cca-credential-socket\") pod \"maas-default-gateway-openshift-default-76597d6855-dqn2b\" (UID: \"415bafea-577e-4fba-925e-eb2c492e3cca\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-dqn2b" Nov 04 03:20:02.399981 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:20:02.399816 4405 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/415bafea-577e-4fba-925e-eb2c492e3cca-workload-certs\") pod \"maas-default-gateway-openshift-default-76597d6855-dqn2b\" (UID: \"415bafea-577e-4fba-925e-eb2c492e3cca\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-dqn2b" Nov 04 03:20:02.399981 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:20:02.399846 4405 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rvwh2\" (UniqueName: \"kubernetes.io/projected/415bafea-577e-4fba-925e-eb2c492e3cca-kube-api-access-rvwh2\") pod \"maas-default-gateway-openshift-default-76597d6855-dqn2b\" (UID: \"415bafea-577e-4fba-925e-eb2c492e3cca\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-dqn2b" Nov 04 03:20:02.399981 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:20:02.399886 4405 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/415bafea-577e-4fba-925e-eb2c492e3cca-workload-socket\") pod \"maas-default-gateway-openshift-default-76597d6855-dqn2b\" (UID: \"415bafea-577e-4fba-925e-eb2c492e3cca\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-dqn2b" Nov 04 03:20:02.400137 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:20:02.400010 4405 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/415bafea-577e-4fba-925e-eb2c492e3cca-istio-data\") pod \"maas-default-gateway-openshift-default-76597d6855-dqn2b\" (UID: \"415bafea-577e-4fba-925e-eb2c492e3cca\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-dqn2b" Nov 04 03:20:02.400315 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:20:02.400288 4405 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/415bafea-577e-4fba-925e-eb2c492e3cca-workload-certs\") pod \"maas-default-gateway-openshift-default-76597d6855-dqn2b\" (UID: \"415bafea-577e-4fba-925e-eb2c492e3cca\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-dqn2b" Nov 04 03:20:02.400685 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:20:02.400318 4405 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/415bafea-577e-4fba-925e-eb2c492e3cca-workload-socket\") pod \"maas-default-gateway-openshift-default-76597d6855-dqn2b\" (UID: \"415bafea-577e-4fba-925e-eb2c492e3cca\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-dqn2b" Nov 04 03:20:02.400790 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:20:02.400321 4405 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/415bafea-577e-4fba-925e-eb2c492e3cca-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-76597d6855-dqn2b\" (UID: \"415bafea-577e-4fba-925e-eb2c492e3cca\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-dqn2b" Nov 04 03:20:02.400790 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:20:02.400391 4405 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/415bafea-577e-4fba-925e-eb2c492e3cca-credential-socket\") pod \"maas-default-gateway-openshift-default-76597d6855-dqn2b\" (UID: \"415bafea-577e-4fba-925e-eb2c492e3cca\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-dqn2b" Nov 04 03:20:02.401988 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:20:02.401963 4405 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/415bafea-577e-4fba-925e-eb2c492e3cca-istio-envoy\") pod \"maas-default-gateway-openshift-default-76597d6855-dqn2b\" (UID: \"415bafea-577e-4fba-925e-eb2c492e3cca\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-dqn2b" Nov 04 03:20:02.402337 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:20:02.402317 4405 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/415bafea-577e-4fba-925e-eb2c492e3cca-istio-podinfo\") pod \"maas-default-gateway-openshift-default-76597d6855-dqn2b\" (UID: \"415bafea-577e-4fba-925e-eb2c492e3cca\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-dqn2b" Nov 04 03:20:02.475968 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:20:02.475933 4405 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/415bafea-577e-4fba-925e-eb2c492e3cca-istio-token\") pod \"maas-default-gateway-openshift-default-76597d6855-dqn2b\" (UID: \"415bafea-577e-4fba-925e-eb2c492e3cca\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-dqn2b" Nov 04 03:20:02.476126 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:20:02.476000 4405 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rvwh2\" (UniqueName: \"kubernetes.io/projected/415bafea-577e-4fba-925e-eb2c492e3cca-kube-api-access-rvwh2\") pod \"maas-default-gateway-openshift-default-76597d6855-dqn2b\" (UID: \"415bafea-577e-4fba-925e-eb2c492e3cca\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-dqn2b" Nov 04 03:20:02.762891 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:20:02.762791 4405 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-dqn2b" Nov 04 03:20:03.188778 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:20:03.188749 4405 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-76597d6855-dqn2b"] Nov 04 03:20:03.191228 ip-10-0-134-125 kubenswrapper[4405]: W1104 03:20:03.191193 4405 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod415bafea_577e_4fba_925e_eb2c492e3cca.slice/crio-53830d509efd1c3c4e6a65e931a6d4e8a237e03479b96bd0775f31939e0bb655 WatchSource:0}: Error finding container 53830d509efd1c3c4e6a65e931a6d4e8a237e03479b96bd0775f31939e0bb655: Status 404 returned error can't find the container with id 53830d509efd1c3c4e6a65e931a6d4e8a237e03479b96bd0775f31939e0bb655 Nov 04 03:20:03.193573 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:20:03.193543 4405 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 04 03:20:03.193639 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:20:03.193610 4405 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 04 03:20:03.193695 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:20:03.193638 4405 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 04 03:20:03.277538 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:20:03.277497 4405 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-dqn2b" event={"ID":"415bafea-577e-4fba-925e-eb2c492e3cca","Type":"ContainerStarted","Data":"82348cec47f50e4bba8d3dee48daae29bee6f823ff8a9b66e6979a2e1c7930c5"} Nov 04 03:20:03.277757 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:20:03.277557 4405 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-dqn2b" event={"ID":"415bafea-577e-4fba-925e-eb2c492e3cca","Type":"ContainerStarted","Data":"53830d509efd1c3c4e6a65e931a6d4e8a237e03479b96bd0775f31939e0bb655"} Nov 04 03:20:03.423176 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:20:03.423115 4405 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-dqn2b" podStartSLOduration=1.423096978 podStartE2EDuration="1.423096978s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 03:20:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 03:20:03.422218431 +0000 UTC m=+2849.852918557" watchObservedRunningTime="2026-11-04 03:20:03.423096978 +0000 UTC m=+2849.853797099" Nov 04 03:20:03.762938 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:20:03.762896 4405 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-dqn2b" Nov 04 03:20:04.764782 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:20:04.764745 4405 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76597d6855-dqn2b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.139:15021/healthz/ready\": dial tcp 10.134.0.139:15021: connect: connection refused" start-of-body= Nov 04 03:20:04.765212 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:20:04.764798 4405 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-dqn2b" podUID="415bafea-577e-4fba-925e-eb2c492e3cca" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.139:15021/healthz/ready\": dial tcp 10.134.0.139:15021: connect: connection refused" Nov 04 03:20:05.763224 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:20:05.763188 4405 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76597d6855-dqn2b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.139:15021/healthz/ready\": dial tcp 10.134.0.139:15021: connect: connection refused" start-of-body= Nov 04 03:20:05.763446 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:20:05.763251 4405 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-dqn2b" podUID="415bafea-577e-4fba-925e-eb2c492e3cca" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.139:15021/healthz/ready\": dial tcp 10.134.0.139:15021: connect: connection refused" Nov 04 03:20:06.763604 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:20:06.763562 4405 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76597d6855-dqn2b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.139:15021/healthz/ready\": dial tcp 10.134.0.139:15021: connect: connection refused" start-of-body= Nov 04 03:20:06.764061 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:20:06.763622 4405 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-dqn2b" podUID="415bafea-577e-4fba-925e-eb2c492e3cca" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.139:15021/healthz/ready\": dial tcp 10.134.0.139:15021: connect: connection refused" Nov 04 03:20:06.882915 ip-10-0-134-125 kubenswrapper[4405]: E1104 03:20:06.882879 4405 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5f2vb" containerName="extractor" Nov 04 03:20:07.763597 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:20:07.763552 4405 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76597d6855-dqn2b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.139:15021/healthz/ready\": dial tcp 10.134.0.139:15021: connect: connection refused" start-of-body= Nov 04 03:20:07.763803 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:20:07.763613 4405 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-dqn2b" podUID="415bafea-577e-4fba-925e-eb2c492e3cca" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.139:15021/healthz/ready\": dial tcp 10.134.0.139:15021: connect: connection refused" Nov 04 03:20:08.764168 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:20:08.764127 4405 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76597d6855-dqn2b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.139:15021/healthz/ready\": dial tcp 10.134.0.139:15021: connect: connection refused" start-of-body= Nov 04 03:20:08.764606 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:20:08.764188 4405 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-dqn2b" podUID="415bafea-577e-4fba-925e-eb2c492e3cca" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.139:15021/healthz/ready\": dial tcp 10.134.0.139:15021: connect: connection refused" Nov 04 03:20:09.763645 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:20:09.763604 4405 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76597d6855-dqn2b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.139:15021/healthz/ready\": dial tcp 10.134.0.139:15021: connect: connection refused" start-of-body= Nov 04 03:20:09.763882 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:20:09.763706 4405 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-dqn2b" podUID="415bafea-577e-4fba-925e-eb2c492e3cca" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.139:15021/healthz/ready\": dial tcp 10.134.0.139:15021: connect: connection refused" Nov 04 03:20:10.764196 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:20:10.764155 4405 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76597d6855-dqn2b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.139:15021/healthz/ready\": dial tcp 10.134.0.139:15021: connect: connection refused" start-of-body= Nov 04 03:20:10.764644 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:20:10.764240 4405 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-dqn2b" podUID="415bafea-577e-4fba-925e-eb2c492e3cca" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.139:15021/healthz/ready\": dial tcp 10.134.0.139:15021: connect: connection refused" Nov 04 03:20:11.923631 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:20:11.923597 4405 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-dqn2b" Nov 04 03:20:11.924475 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:20:11.924447 4405 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-dqn2b" Nov 04 03:20:11.925244 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:20:11.925227 4405 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-dqn2b" Nov 04 03:20:36.883733 ip-10-0-134-125 kubenswrapper[4405]: E1104 03:20:36.883697 4405 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5f2vb" containerName="extractor" Nov 04 03:20:41.737889 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:20:41.737844 4405 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-bl5mg"] Nov 04 03:20:45.003694 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:20:45.000815 4405 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76597d6855-xjqqw container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.138:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 04 03:20:45.003694 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:20:45.000924 4405 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-xjqqw" podUID="c97cb26c-6fc7-45db-9984-aa459ddb54df" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.138:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 04 03:20:47.762791 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:20:47.762750 4405 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-7f6b7"] Nov 04 03:20:47.765475 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:20:47.765450 4405 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-7f6b7" Nov 04 03:20:47.867117 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:20:47.867084 4405 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 04 03:20:47.867324 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:20:47.867155 4405 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 04 03:20:47.867324 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:20:47.867154 4405 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 04 03:20:47.867324 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:20:47.867166 4405 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-slbq8\"" Nov 04 03:20:47.869053 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:20:47.869034 4405 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-7f6b7"] Nov 04 03:20:47.878940 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:20:47.878795 4405 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1bdb533b-b3cd-4f44-9a0c-eac035fbc90b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c1e271a9-7f6b7\" (UID: \"1bdb533b-b3cd-4f44-9a0c-eac035fbc90b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-7f6b7" Nov 04 03:20:47.878940 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:20:47.878876 4405 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8hffs\" (UniqueName: \"kubernetes.io/projected/1bdb533b-b3cd-4f44-9a0c-eac035fbc90b-kube-api-access-8hffs\") pod \"maas-api-revoke-sub-c1e271a9-7f6b7\" (UID: \"1bdb533b-b3cd-4f44-9a0c-eac035fbc90b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-7f6b7" Nov 04 03:20:47.879172 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:20:47.879025 4405 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1bdb533b-b3cd-4f44-9a0c-eac035fbc90b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c1e271a9-7f6b7\" (UID: \"1bdb533b-b3cd-4f44-9a0c-eac035fbc90b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-7f6b7" Nov 04 03:20:47.980695 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:20:47.980631 4405 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1bdb533b-b3cd-4f44-9a0c-eac035fbc90b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c1e271a9-7f6b7\" (UID: \"1bdb533b-b3cd-4f44-9a0c-eac035fbc90b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-7f6b7" Nov 04 03:20:47.980888 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:20:47.980735 4405 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8hffs\" (UniqueName: \"kubernetes.io/projected/1bdb533b-b3cd-4f44-9a0c-eac035fbc90b-kube-api-access-8hffs\") pod \"maas-api-revoke-sub-c1e271a9-7f6b7\" (UID: \"1bdb533b-b3cd-4f44-9a0c-eac035fbc90b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-7f6b7" Nov 04 03:20:47.980888 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:20:47.980803 4405 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1bdb533b-b3cd-4f44-9a0c-eac035fbc90b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c1e271a9-7f6b7\" (UID: \"1bdb533b-b3cd-4f44-9a0c-eac035fbc90b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-7f6b7" Nov 04 03:20:47.981481 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:20:47.981455 4405 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1bdb533b-b3cd-4f44-9a0c-eac035fbc90b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c1e271a9-7f6b7\" (UID: \"1bdb533b-b3cd-4f44-9a0c-eac035fbc90b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-7f6b7" Nov 04 03:20:47.983332 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:20:47.983308 4405 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1bdb533b-b3cd-4f44-9a0c-eac035fbc90b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c1e271a9-7f6b7\" (UID: \"1bdb533b-b3cd-4f44-9a0c-eac035fbc90b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-7f6b7" Nov 04 03:20:48.062477 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:20:48.062410 4405 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8hffs\" (UniqueName: \"kubernetes.io/projected/1bdb533b-b3cd-4f44-9a0c-eac035fbc90b-kube-api-access-8hffs\") pod \"maas-api-revoke-sub-c1e271a9-7f6b7\" (UID: \"1bdb533b-b3cd-4f44-9a0c-eac035fbc90b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-7f6b7" Nov 04 03:20:48.077842 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:20:48.077808 4405 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-7f6b7" Nov 04 03:20:48.451956 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:20:48.451933 4405 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-7f6b7"] Nov 04 03:20:48.454410 ip-10-0-134-125 kubenswrapper[4405]: W1104 03:20:48.454382 4405 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1bdb533b_b3cd_4f44_9a0c_eac035fbc90b.slice/crio-d5ccb7452eb90f6d323f352bf83c28bf51735e907cbd554e1c37e7c15984dfaa WatchSource:0}: Error finding container d5ccb7452eb90f6d323f352bf83c28bf51735e907cbd554e1c37e7c15984dfaa: Status 404 returned error can't find the container with id d5ccb7452eb90f6d323f352bf83c28bf51735e907cbd554e1c37e7c15984dfaa Nov 04 03:20:48.509030 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:20:48.508991 4405 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-7f6b7" event={"ID":"1bdb533b-b3cd-4f44-9a0c-eac035fbc90b","Type":"ContainerStarted","Data":"d5ccb7452eb90f6d323f352bf83c28bf51735e907cbd554e1c37e7c15984dfaa"} Nov 04 03:20:49.514998 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:20:49.514963 4405 generic.go:356] "Generic (PLEG): container finished" podID="1bdb533b-b3cd-4f44-9a0c-eac035fbc90b" containerID="d85bf0a3fb627a23278d8c0be24df4f7f8d39df6b5e89f6c6b588148a5367b81" exitCode=0 Nov 04 03:20:49.515382 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:20:49.515081 4405 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-7f6b7" event={"ID":"1bdb533b-b3cd-4f44-9a0c-eac035fbc90b","Type":"ContainerDied","Data":"d85bf0a3fb627a23278d8c0be24df4f7f8d39df6b5e89f6c6b588148a5367b81"} Nov 04 03:20:50.656962 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:20:50.656928 4405 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-7f6b7" Nov 04 03:20:50.807831 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:20:50.807731 4405 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1bdb533b-b3cd-4f44-9a0c-eac035fbc90b-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1bdb533b-b3cd-4f44-9a0c-eac035fbc90b-maas-api-xks-ca\") pod \"1bdb533b-b3cd-4f44-9a0c-eac035fbc90b\" (UID: \"1bdb533b-b3cd-4f44-9a0c-eac035fbc90b\") " Nov 04 03:20:50.807831 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:20:50.807800 4405 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1bdb533b-b3cd-4f44-9a0c-eac035fbc90b-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1bdb533b-b3cd-4f44-9a0c-eac035fbc90b-maas-api-openshift-ca\") pod \"1bdb533b-b3cd-4f44-9a0c-eac035fbc90b\" (UID: \"1bdb533b-b3cd-4f44-9a0c-eac035fbc90b\") " Nov 04 03:20:50.808139 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:20:50.807844 4405 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1bdb533b-b3cd-4f44-9a0c-eac035fbc90b-kube-api-access-8hffs\" (UniqueName: \"kubernetes.io/projected/1bdb533b-b3cd-4f44-9a0c-eac035fbc90b-kube-api-access-8hffs\") pod \"1bdb533b-b3cd-4f44-9a0c-eac035fbc90b\" (UID: \"1bdb533b-b3cd-4f44-9a0c-eac035fbc90b\") " Nov 04 03:20:50.808202 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:20:50.808174 4405 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1bdb533b-b3cd-4f44-9a0c-eac035fbc90b-maas-api-openshift-ca" pod "1bdb533b-b3cd-4f44-9a0c-eac035fbc90b" (UID: "1bdb533b-b3cd-4f44-9a0c-eac035fbc90b"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 03:20:50.810111 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:20:50.810086 4405 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1bdb533b-b3cd-4f44-9a0c-eac035fbc90b-maas-api-xks-ca" pod "1bdb533b-b3cd-4f44-9a0c-eac035fbc90b" (UID: "1bdb533b-b3cd-4f44-9a0c-eac035fbc90b"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 04 03:20:50.810193 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:20:50.810126 4405 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1bdb533b-b3cd-4f44-9a0c-eac035fbc90b-kube-api-access-8hffs" pod "1bdb533b-b3cd-4f44-9a0c-eac035fbc90b" (UID: "1bdb533b-b3cd-4f44-9a0c-eac035fbc90b"). InnerVolumeSpecName "kube-api-access-8hffs". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 03:20:50.909279 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:20:50.909223 4405 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1bdb533b-b3cd-4f44-9a0c-eac035fbc90b-maas-api-xks-ca\") on node \"ip-10-0-134-125.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:20:50.909279 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:20:50.909274 4405 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1bdb533b-b3cd-4f44-9a0c-eac035fbc90b-maas-api-openshift-ca\") on node \"ip-10-0-134-125.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:20:50.909279 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:20:50.909287 4405 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8hffs\" (UniqueName: \"kubernetes.io/projected/1bdb533b-b3cd-4f44-9a0c-eac035fbc90b-kube-api-access-8hffs\") on node \"ip-10-0-134-125.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:20:51.526080 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:20:51.526050 4405 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-7f6b7" Nov 04 03:20:51.526295 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:20:51.526048 4405 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-7f6b7" event={"ID":"1bdb533b-b3cd-4f44-9a0c-eac035fbc90b","Type":"ContainerDied","Data":"d5ccb7452eb90f6d323f352bf83c28bf51735e907cbd554e1c37e7c15984dfaa"} Nov 04 03:20:51.526295 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:20:51.526183 4405 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d5ccb7452eb90f6d323f352bf83c28bf51735e907cbd554e1c37e7c15984dfaa" Nov 04 03:21:00.641481 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:21:00.641433 4405 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-bl5mg"] Nov 04 03:21:06.897389 ip-10-0-134-125 kubenswrapper[4405]: E1104 03:21:06.897287 4405 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5f2vb" containerName="extractor" Nov 04 03:21:11.939347 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:21:11.939307 4405 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-bl5mg"] Nov 04 03:21:13.310773 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:21:13.310734 4405 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-jk2tn"] Nov 04 03:21:13.311187 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:21:13.311083 4405 cpu_manager.go:414] "containerMap: removing container" podUID="1bdb533b-b3cd-4f44-9a0c-eac035fbc90b" containerName="revoke-keys" Nov 04 03:21:13.311187 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:21:13.311093 4405 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1bdb533b-b3cd-4f44-9a0c-eac035fbc90b" containerName="revoke-keys" Nov 04 03:21:13.311187 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:21:13.311171 4405 memory_manager.go:365] "RemoveStaleState removing state" podUID="1bdb533b-b3cd-4f44-9a0c-eac035fbc90b" containerName="revoke-keys" Nov 04 03:21:13.314300 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:21:13.314283 4405 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-jk2tn" Nov 04 03:21:13.366840 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:21:13.366809 4405 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 04 03:21:13.416262 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:21:13.416232 4405 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 04 03:21:13.416446 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:21:13.416298 4405 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 04 03:21:13.416655 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:21:13.416638 4405 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-slbq8\"" Nov 04 03:21:13.418401 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:21:13.418379 4405 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-jk2tn"] Nov 04 03:21:13.426107 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:21:13.426086 4405 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ba6e018c-ab91-4734-8b38-572dd172093d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2e1f5b5f-jk2tn\" (UID: \"ba6e018c-ab91-4734-8b38-572dd172093d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-jk2tn" Nov 04 03:21:13.426234 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:21:13.426215 4405 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ba6e018c-ab91-4734-8b38-572dd172093d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2e1f5b5f-jk2tn\" (UID: \"ba6e018c-ab91-4734-8b38-572dd172093d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-jk2tn" Nov 04 03:21:13.426303 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:21:13.426286 4405 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cmrzs\" (UniqueName: \"kubernetes.io/projected/ba6e018c-ab91-4734-8b38-572dd172093d-kube-api-access-cmrzs\") pod \"maas-api-revoke-sub-2e1f5b5f-jk2tn\" (UID: \"ba6e018c-ab91-4734-8b38-572dd172093d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-jk2tn" Nov 04 03:21:13.527765 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:21:13.527724 4405 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ba6e018c-ab91-4734-8b38-572dd172093d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2e1f5b5f-jk2tn\" (UID: \"ba6e018c-ab91-4734-8b38-572dd172093d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-jk2tn" Nov 04 03:21:13.527954 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:21:13.527806 4405 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cmrzs\" (UniqueName: \"kubernetes.io/projected/ba6e018c-ab91-4734-8b38-572dd172093d-kube-api-access-cmrzs\") pod \"maas-api-revoke-sub-2e1f5b5f-jk2tn\" (UID: \"ba6e018c-ab91-4734-8b38-572dd172093d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-jk2tn" Nov 04 03:21:13.527954 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:21:13.527872 4405 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ba6e018c-ab91-4734-8b38-572dd172093d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2e1f5b5f-jk2tn\" (UID: \"ba6e018c-ab91-4734-8b38-572dd172093d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-jk2tn" Nov 04 03:21:13.528501 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:21:13.528480 4405 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ba6e018c-ab91-4734-8b38-572dd172093d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2e1f5b5f-jk2tn\" (UID: \"ba6e018c-ab91-4734-8b38-572dd172093d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-jk2tn" Nov 04 03:21:13.530319 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:21:13.530300 4405 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ba6e018c-ab91-4734-8b38-572dd172093d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2e1f5b5f-jk2tn\" (UID: \"ba6e018c-ab91-4734-8b38-572dd172093d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-jk2tn" Nov 04 03:21:13.589118 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:21:13.589090 4405 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cmrzs\" (UniqueName: \"kubernetes.io/projected/ba6e018c-ab91-4734-8b38-572dd172093d-kube-api-access-cmrzs\") pod \"maas-api-revoke-sub-2e1f5b5f-jk2tn\" (UID: \"ba6e018c-ab91-4734-8b38-572dd172093d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-jk2tn" Nov 04 03:21:13.626780 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:21:13.626746 4405 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-jk2tn" Nov 04 03:21:14.004864 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:21:14.004841 4405 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-jk2tn"] Nov 04 03:21:14.007057 ip-10-0-134-125 kubenswrapper[4405]: W1104 03:21:14.007015 4405 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podba6e018c_ab91_4734_8b38_572dd172093d.slice/crio-87bae85e8cbe481ac0f2a567552ae545be73552cf138679001e820ed41282087 WatchSource:0}: Error finding container 87bae85e8cbe481ac0f2a567552ae545be73552cf138679001e820ed41282087: Status 404 returned error can't find the container with id 87bae85e8cbe481ac0f2a567552ae545be73552cf138679001e820ed41282087 Nov 04 03:21:14.650010 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:21:14.649973 4405 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-jk2tn" event={"ID":"ba6e018c-ab91-4734-8b38-572dd172093d","Type":"ContainerStarted","Data":"0ec0750c29bad6a50f632b25f0556a599197fb8ece20f2dfe21f1faf48d96358"} Nov 04 03:21:14.650518 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:21:14.650028 4405 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-jk2tn" event={"ID":"ba6e018c-ab91-4734-8b38-572dd172093d","Type":"ContainerStarted","Data":"87bae85e8cbe481ac0f2a567552ae545be73552cf138679001e820ed41282087"} Nov 04 03:21:14.766505 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:21:14.766446 4405 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-jk2tn" podStartSLOduration=1.766429557 podStartE2EDuration="1.766429557s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 03:21:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 03:21:14.765032456 +0000 UTC m=+2921.195732577" watchObservedRunningTime="2026-11-04 03:21:14.766429557 +0000 UTC m=+2921.197129677" Nov 04 03:21:15.661287 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:21:15.661249 4405 generic.go:356] "Generic (PLEG): container finished" podID="ba6e018c-ab91-4734-8b38-572dd172093d" containerID="0ec0750c29bad6a50f632b25f0556a599197fb8ece20f2dfe21f1faf48d96358" exitCode=0 Nov 04 03:21:15.661829 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:21:15.661331 4405 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-jk2tn" event={"ID":"ba6e018c-ab91-4734-8b38-572dd172093d","Type":"ContainerDied","Data":"0ec0750c29bad6a50f632b25f0556a599197fb8ece20f2dfe21f1faf48d96358"} Nov 04 03:21:16.827285 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:21:16.827260 4405 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-jk2tn" Nov 04 03:21:16.962550 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:21:16.962460 4405 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ba6e018c-ab91-4734-8b38-572dd172093d-kube-api-access-cmrzs\" (UniqueName: \"kubernetes.io/projected/ba6e018c-ab91-4734-8b38-572dd172093d-kube-api-access-cmrzs\") pod \"ba6e018c-ab91-4734-8b38-572dd172093d\" (UID: \"ba6e018c-ab91-4734-8b38-572dd172093d\") " Nov 04 03:21:16.962550 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:21:16.962531 4405 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ba6e018c-ab91-4734-8b38-572dd172093d-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ba6e018c-ab91-4734-8b38-572dd172093d-maas-api-xks-ca\") pod \"ba6e018c-ab91-4734-8b38-572dd172093d\" (UID: \"ba6e018c-ab91-4734-8b38-572dd172093d\") " Nov 04 03:21:16.962866 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:21:16.962556 4405 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ba6e018c-ab91-4734-8b38-572dd172093d-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ba6e018c-ab91-4734-8b38-572dd172093d-maas-api-openshift-ca\") pod \"ba6e018c-ab91-4734-8b38-572dd172093d\" (UID: \"ba6e018c-ab91-4734-8b38-572dd172093d\") " Nov 04 03:21:16.963085 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:21:16.963052 4405 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ba6e018c-ab91-4734-8b38-572dd172093d-maas-api-openshift-ca" pod "ba6e018c-ab91-4734-8b38-572dd172093d" (UID: "ba6e018c-ab91-4734-8b38-572dd172093d"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 03:21:16.965227 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:21:16.965195 4405 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ba6e018c-ab91-4734-8b38-572dd172093d-maas-api-xks-ca" pod "ba6e018c-ab91-4734-8b38-572dd172093d" (UID: "ba6e018c-ab91-4734-8b38-572dd172093d"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 04 03:21:16.965227 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:21:16.965217 4405 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ba6e018c-ab91-4734-8b38-572dd172093d-kube-api-access-cmrzs" pod "ba6e018c-ab91-4734-8b38-572dd172093d" (UID: "ba6e018c-ab91-4734-8b38-572dd172093d"). InnerVolumeSpecName "kube-api-access-cmrzs". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 03:21:17.063323 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:21:17.063288 4405 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cmrzs\" (UniqueName: \"kubernetes.io/projected/ba6e018c-ab91-4734-8b38-572dd172093d-kube-api-access-cmrzs\") on node \"ip-10-0-134-125.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:21:17.063323 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:21:17.063314 4405 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ba6e018c-ab91-4734-8b38-572dd172093d-maas-api-xks-ca\") on node \"ip-10-0-134-125.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:21:17.063323 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:21:17.063330 4405 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ba6e018c-ab91-4734-8b38-572dd172093d-maas-api-openshift-ca\") on node \"ip-10-0-134-125.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:21:17.673091 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:21:17.673058 4405 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-jk2tn" Nov 04 03:21:17.673279 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:21:17.673087 4405 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-jk2tn" event={"ID":"ba6e018c-ab91-4734-8b38-572dd172093d","Type":"ContainerDied","Data":"87bae85e8cbe481ac0f2a567552ae545be73552cf138679001e820ed41282087"} Nov 04 03:21:17.673279 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:21:17.673135 4405 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="87bae85e8cbe481ac0f2a567552ae545be73552cf138679001e820ed41282087" Nov 04 03:21:25.813790 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:21:25.813751 4405 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-v4d8q"] Nov 04 03:21:25.814290 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:21:25.814163 4405 cpu_manager.go:414] "containerMap: removing container" podUID="ba6e018c-ab91-4734-8b38-572dd172093d" containerName="revoke-keys" Nov 04 03:21:25.814290 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:21:25.814176 4405 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ba6e018c-ab91-4734-8b38-572dd172093d" containerName="revoke-keys" Nov 04 03:21:25.814290 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:21:25.814265 4405 memory_manager.go:365] "RemoveStaleState removing state" podUID="ba6e018c-ab91-4734-8b38-572dd172093d" containerName="revoke-keys" Nov 04 03:21:25.817715 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:21:25.817693 4405 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-v4d8q" Nov 04 03:21:25.922529 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:21:25.922440 4405 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 04 03:21:25.922529 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:21:25.922491 4405 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 04 03:21:25.922850 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:21:25.922728 4405 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 04 03:21:25.922850 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:21:25.922759 4405 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-slbq8\"" Nov 04 03:21:25.925108 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:21:25.925085 4405 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-v4d8q"] Nov 04 03:21:25.953694 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:21:25.953629 4405 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/54df9890-5e4d-44a6-8cf4-329c7544c2e2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ec843018-v4d8q\" (UID: \"54df9890-5e4d-44a6-8cf4-329c7544c2e2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-v4d8q" Nov 04 03:21:25.953895 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:21:25.953863 4405 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-llttr\" (UniqueName: \"kubernetes.io/projected/54df9890-5e4d-44a6-8cf4-329c7544c2e2-kube-api-access-llttr\") pod \"maas-api-revoke-sub-ec843018-v4d8q\" (UID: \"54df9890-5e4d-44a6-8cf4-329c7544c2e2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-v4d8q" Nov 04 03:21:25.953965 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:21:25.953931 4405 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/54df9890-5e4d-44a6-8cf4-329c7544c2e2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ec843018-v4d8q\" (UID: \"54df9890-5e4d-44a6-8cf4-329c7544c2e2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-v4d8q" Nov 04 03:21:26.054713 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:21:26.054631 4405 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-llttr\" (UniqueName: \"kubernetes.io/projected/54df9890-5e4d-44a6-8cf4-329c7544c2e2-kube-api-access-llttr\") pod \"maas-api-revoke-sub-ec843018-v4d8q\" (UID: \"54df9890-5e4d-44a6-8cf4-329c7544c2e2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-v4d8q" Nov 04 03:21:26.054950 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:21:26.054753 4405 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/54df9890-5e4d-44a6-8cf4-329c7544c2e2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ec843018-v4d8q\" (UID: \"54df9890-5e4d-44a6-8cf4-329c7544c2e2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-v4d8q" Nov 04 03:21:26.054950 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:21:26.054799 4405 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/54df9890-5e4d-44a6-8cf4-329c7544c2e2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ec843018-v4d8q\" (UID: \"54df9890-5e4d-44a6-8cf4-329c7544c2e2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-v4d8q" Nov 04 03:21:26.056172 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:21:26.056135 4405 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/54df9890-5e4d-44a6-8cf4-329c7544c2e2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ec843018-v4d8q\" (UID: \"54df9890-5e4d-44a6-8cf4-329c7544c2e2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-v4d8q" Nov 04 03:21:26.058479 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:21:26.058442 4405 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/54df9890-5e4d-44a6-8cf4-329c7544c2e2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ec843018-v4d8q\" (UID: \"54df9890-5e4d-44a6-8cf4-329c7544c2e2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-v4d8q" Nov 04 03:21:26.115831 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:21:26.115793 4405 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-llttr\" (UniqueName: \"kubernetes.io/projected/54df9890-5e4d-44a6-8cf4-329c7544c2e2-kube-api-access-llttr\") pod \"maas-api-revoke-sub-ec843018-v4d8q\" (UID: \"54df9890-5e4d-44a6-8cf4-329c7544c2e2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-v4d8q" Nov 04 03:21:26.134819 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:21:26.134789 4405 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-v4d8q" Nov 04 03:21:26.253035 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:21:26.252997 4405 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-bl5mg"] Nov 04 03:21:26.509141 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:21:26.509041 4405 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-v4d8q"] Nov 04 03:21:26.511388 ip-10-0-134-125 kubenswrapper[4405]: W1104 03:21:26.511355 4405 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod54df9890_5e4d_44a6_8cf4_329c7544c2e2.slice/crio-f6715b35962a21b4b50176626e6245e95c2df345d9705a6e88da4b063fc32479 WatchSource:0}: Error finding container f6715b35962a21b4b50176626e6245e95c2df345d9705a6e88da4b063fc32479: Status 404 returned error can't find the container with id f6715b35962a21b4b50176626e6245e95c2df345d9705a6e88da4b063fc32479 Nov 04 03:21:26.715793 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:21:26.715758 4405 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-v4d8q" event={"ID":"54df9890-5e4d-44a6-8cf4-329c7544c2e2","Type":"ContainerStarted","Data":"f6715b35962a21b4b50176626e6245e95c2df345d9705a6e88da4b063fc32479"} Nov 04 03:21:27.722428 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:21:27.722328 4405 generic.go:356] "Generic (PLEG): container finished" podID="54df9890-5e4d-44a6-8cf4-329c7544c2e2" containerID="7cf5c464c48a4969e75bbe372e5f8e38658be9036ef98cdf56718330104d2c54" exitCode=0 Nov 04 03:21:27.722428 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:21:27.722367 4405 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-v4d8q" event={"ID":"54df9890-5e4d-44a6-8cf4-329c7544c2e2","Type":"ContainerDied","Data":"7cf5c464c48a4969e75bbe372e5f8e38658be9036ef98cdf56718330104d2c54"} Nov 04 03:21:28.862262 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:21:28.862233 4405 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-v4d8q" Nov 04 03:21:28.984585 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:21:28.984486 4405 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/54df9890-5e4d-44a6-8cf4-329c7544c2e2-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/54df9890-5e4d-44a6-8cf4-329c7544c2e2-maas-api-openshift-ca\") pod \"54df9890-5e4d-44a6-8cf4-329c7544c2e2\" (UID: \"54df9890-5e4d-44a6-8cf4-329c7544c2e2\") " Nov 04 03:21:28.984585 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:21:28.984555 4405 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/54df9890-5e4d-44a6-8cf4-329c7544c2e2-kube-api-access-llttr\" (UniqueName: \"kubernetes.io/projected/54df9890-5e4d-44a6-8cf4-329c7544c2e2-kube-api-access-llttr\") pod \"54df9890-5e4d-44a6-8cf4-329c7544c2e2\" (UID: \"54df9890-5e4d-44a6-8cf4-329c7544c2e2\") " Nov 04 03:21:28.984888 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:21:28.984638 4405 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/54df9890-5e4d-44a6-8cf4-329c7544c2e2-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/54df9890-5e4d-44a6-8cf4-329c7544c2e2-maas-api-xks-ca\") pod \"54df9890-5e4d-44a6-8cf4-329c7544c2e2\" (UID: \"54df9890-5e4d-44a6-8cf4-329c7544c2e2\") " Nov 04 03:21:28.985125 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:21:28.985087 4405 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/54df9890-5e4d-44a6-8cf4-329c7544c2e2-maas-api-openshift-ca" pod "54df9890-5e4d-44a6-8cf4-329c7544c2e2" (UID: "54df9890-5e4d-44a6-8cf4-329c7544c2e2"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 03:21:28.987623 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:21:28.987584 4405 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/54df9890-5e4d-44a6-8cf4-329c7544c2e2-maas-api-xks-ca" pod "54df9890-5e4d-44a6-8cf4-329c7544c2e2" (UID: "54df9890-5e4d-44a6-8cf4-329c7544c2e2"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 04 03:21:28.988169 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:21:28.988140 4405 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/54df9890-5e4d-44a6-8cf4-329c7544c2e2-kube-api-access-llttr" pod "54df9890-5e4d-44a6-8cf4-329c7544c2e2" (UID: "54df9890-5e4d-44a6-8cf4-329c7544c2e2"). InnerVolumeSpecName "kube-api-access-llttr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 03:21:29.086069 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:21:29.086031 4405 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/54df9890-5e4d-44a6-8cf4-329c7544c2e2-maas-api-openshift-ca\") on node \"ip-10-0-134-125.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:21:29.086069 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:21:29.086065 4405 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-llttr\" (UniqueName: \"kubernetes.io/projected/54df9890-5e4d-44a6-8cf4-329c7544c2e2-kube-api-access-llttr\") on node \"ip-10-0-134-125.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:21:29.086335 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:21:29.086079 4405 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/54df9890-5e4d-44a6-8cf4-329c7544c2e2-maas-api-xks-ca\") on node \"ip-10-0-134-125.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:21:29.732395 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:21:29.732360 4405 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-v4d8q" event={"ID":"54df9890-5e4d-44a6-8cf4-329c7544c2e2","Type":"ContainerDied","Data":"f6715b35962a21b4b50176626e6245e95c2df345d9705a6e88da4b063fc32479"} Nov 04 03:21:29.732606 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:21:29.732419 4405 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f6715b35962a21b4b50176626e6245e95c2df345d9705a6e88da4b063fc32479" Nov 04 03:21:29.732606 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:21:29.732373 4405 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-v4d8q" Nov 04 03:21:35.235120 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:21:35.235088 4405 scope.go:122] "RemoveContainer" containerID="e8d1a5666eb7317340bf5fce328af705481496dfdeffbcb56490374b3b7e1181" Nov 04 03:21:36.884050 ip-10-0-134-125 kubenswrapper[4405]: E1104 03:21:36.884018 4405 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5f2vb" containerName="extractor" Nov 04 03:21:43.148452 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:21:43.148417 4405 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-bl5mg"] Nov 04 03:22:01.520743 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:22:01.520709 4405 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-g9srh"] Nov 04 03:22:01.521148 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:22:01.521123 4405 cpu_manager.go:414] "containerMap: removing container" podUID="54df9890-5e4d-44a6-8cf4-329c7544c2e2" containerName="revoke-keys" Nov 04 03:22:01.521148 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:22:01.521139 4405 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="54df9890-5e4d-44a6-8cf4-329c7544c2e2" containerName="revoke-keys" Nov 04 03:22:01.521232 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:22:01.521226 4405 memory_manager.go:365] "RemoveStaleState removing state" podUID="54df9890-5e4d-44a6-8cf4-329c7544c2e2" containerName="revoke-keys" Nov 04 03:22:01.525680 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:22:01.525644 4405 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-g9srh" Nov 04 03:22:01.577913 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:22:01.577877 4405 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 04 03:22:01.627455 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:22:01.627423 4405 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 04 03:22:01.627455 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:22:01.627447 4405 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 04 03:22:01.627694 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:22:01.627463 4405 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-slbq8\"" Nov 04 03:22:01.629927 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:22:01.629902 4405 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/14d1b082-b0f7-4795-ab6e-cfc82d67ae8f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-046e1343-g9srh\" (UID: \"14d1b082-b0f7-4795-ab6e-cfc82d67ae8f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-g9srh" Nov 04 03:22:01.630093 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:22:01.629979 4405 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xdgcd\" (UniqueName: \"kubernetes.io/projected/14d1b082-b0f7-4795-ab6e-cfc82d67ae8f-kube-api-access-xdgcd\") pod \"maas-api-revoke-sub-046e1343-g9srh\" (UID: \"14d1b082-b0f7-4795-ab6e-cfc82d67ae8f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-g9srh" Nov 04 03:22:01.630093 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:22:01.630070 4405 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-g9srh"] Nov 04 03:22:01.630217 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:22:01.630092 4405 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/14d1b082-b0f7-4795-ab6e-cfc82d67ae8f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-046e1343-g9srh\" (UID: \"14d1b082-b0f7-4795-ab6e-cfc82d67ae8f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-g9srh" Nov 04 03:22:01.730995 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:22:01.730961 4405 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/14d1b082-b0f7-4795-ab6e-cfc82d67ae8f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-046e1343-g9srh\" (UID: \"14d1b082-b0f7-4795-ab6e-cfc82d67ae8f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-g9srh" Nov 04 03:22:01.731169 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:22:01.731010 4405 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/14d1b082-b0f7-4795-ab6e-cfc82d67ae8f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-046e1343-g9srh\" (UID: \"14d1b082-b0f7-4795-ab6e-cfc82d67ae8f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-g9srh" Nov 04 03:22:01.731169 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:22:01.731064 4405 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xdgcd\" (UniqueName: \"kubernetes.io/projected/14d1b082-b0f7-4795-ab6e-cfc82d67ae8f-kube-api-access-xdgcd\") pod \"maas-api-revoke-sub-046e1343-g9srh\" (UID: \"14d1b082-b0f7-4795-ab6e-cfc82d67ae8f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-g9srh" Nov 04 03:22:01.731694 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:22:01.731650 4405 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/14d1b082-b0f7-4795-ab6e-cfc82d67ae8f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-046e1343-g9srh\" (UID: \"14d1b082-b0f7-4795-ab6e-cfc82d67ae8f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-g9srh" Nov 04 03:22:01.733630 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:22:01.733612 4405 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/14d1b082-b0f7-4795-ab6e-cfc82d67ae8f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-046e1343-g9srh\" (UID: \"14d1b082-b0f7-4795-ab6e-cfc82d67ae8f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-g9srh" Nov 04 03:22:01.792571 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:22:01.792505 4405 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xdgcd\" (UniqueName: \"kubernetes.io/projected/14d1b082-b0f7-4795-ab6e-cfc82d67ae8f-kube-api-access-xdgcd\") pod \"maas-api-revoke-sub-046e1343-g9srh\" (UID: \"14d1b082-b0f7-4795-ab6e-cfc82d67ae8f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-g9srh" Nov 04 03:22:01.839444 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:22:01.839405 4405 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-g9srh" Nov 04 03:22:02.212616 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:22:02.212582 4405 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-g9srh"] Nov 04 03:22:02.215058 ip-10-0-134-125 kubenswrapper[4405]: W1104 03:22:02.215028 4405 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod14d1b082_b0f7_4795_ab6e_cfc82d67ae8f.slice/crio-3064bd1647f53a5a668d2005383eacc1109f70632895d2d8703ce8c775f24d5e WatchSource:0}: Error finding container 3064bd1647f53a5a668d2005383eacc1109f70632895d2d8703ce8c775f24d5e: Status 404 returned error can't find the container with id 3064bd1647f53a5a668d2005383eacc1109f70632895d2d8703ce8c775f24d5e Nov 04 03:22:02.907227 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:22:02.907184 4405 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-g9srh" event={"ID":"14d1b082-b0f7-4795-ab6e-cfc82d67ae8f","Type":"ContainerStarted","Data":"034a5fa4b16710a4bca4ac24b226b29558472d7c002b4059fc4e0c12a44ff46d"} Nov 04 03:22:02.907696 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:22:02.907249 4405 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-g9srh" event={"ID":"14d1b082-b0f7-4795-ab6e-cfc82d67ae8f","Type":"ContainerStarted","Data":"3064bd1647f53a5a668d2005383eacc1109f70632895d2d8703ce8c775f24d5e"} Nov 04 03:22:03.023526 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:22:03.023463 4405 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-g9srh" podStartSLOduration=2.023447938 podStartE2EDuration="2.023447938s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 03:22:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 03:22:03.022513812 +0000 UTC m=+2969.453213944" watchObservedRunningTime="2026-11-04 03:22:03.023447938 +0000 UTC m=+2969.454148058" Nov 04 03:22:03.913945 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:22:03.913907 4405 generic.go:356] "Generic (PLEG): container finished" podID="14d1b082-b0f7-4795-ab6e-cfc82d67ae8f" containerID="034a5fa4b16710a4bca4ac24b226b29558472d7c002b4059fc4e0c12a44ff46d" exitCode=0 Nov 04 03:22:03.914372 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:22:03.913992 4405 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-g9srh" event={"ID":"14d1b082-b0f7-4795-ab6e-cfc82d67ae8f","Type":"ContainerDied","Data":"034a5fa4b16710a4bca4ac24b226b29558472d7c002b4059fc4e0c12a44ff46d"} Nov 04 03:22:05.062415 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:22:05.062387 4405 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-g9srh" Nov 04 03:22:05.157457 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:22:05.157419 4405 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/14d1b082-b0f7-4795-ab6e-cfc82d67ae8f-kube-api-access-xdgcd\" (UniqueName: \"kubernetes.io/projected/14d1b082-b0f7-4795-ab6e-cfc82d67ae8f-kube-api-access-xdgcd\") pod \"14d1b082-b0f7-4795-ab6e-cfc82d67ae8f\" (UID: \"14d1b082-b0f7-4795-ab6e-cfc82d67ae8f\") " Nov 04 03:22:05.157651 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:22:05.157522 4405 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/14d1b082-b0f7-4795-ab6e-cfc82d67ae8f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/14d1b082-b0f7-4795-ab6e-cfc82d67ae8f-maas-api-xks-ca\") pod \"14d1b082-b0f7-4795-ab6e-cfc82d67ae8f\" (UID: \"14d1b082-b0f7-4795-ab6e-cfc82d67ae8f\") " Nov 04 03:22:05.157651 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:22:05.157577 4405 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/14d1b082-b0f7-4795-ab6e-cfc82d67ae8f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/14d1b082-b0f7-4795-ab6e-cfc82d67ae8f-maas-api-openshift-ca\") pod \"14d1b082-b0f7-4795-ab6e-cfc82d67ae8f\" (UID: \"14d1b082-b0f7-4795-ab6e-cfc82d67ae8f\") " Nov 04 03:22:05.157982 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:22:05.157957 4405 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/14d1b082-b0f7-4795-ab6e-cfc82d67ae8f-maas-api-openshift-ca" pod "14d1b082-b0f7-4795-ab6e-cfc82d67ae8f" (UID: "14d1b082-b0f7-4795-ab6e-cfc82d67ae8f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 03:22:05.159821 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:22:05.159793 4405 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/14d1b082-b0f7-4795-ab6e-cfc82d67ae8f-maas-api-xks-ca" pod "14d1b082-b0f7-4795-ab6e-cfc82d67ae8f" (UID: "14d1b082-b0f7-4795-ab6e-cfc82d67ae8f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 04 03:22:05.160332 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:22:05.160307 4405 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/14d1b082-b0f7-4795-ab6e-cfc82d67ae8f-kube-api-access-xdgcd" pod "14d1b082-b0f7-4795-ab6e-cfc82d67ae8f" (UID: "14d1b082-b0f7-4795-ab6e-cfc82d67ae8f"). InnerVolumeSpecName "kube-api-access-xdgcd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 03:22:05.258887 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:22:05.258790 4405 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/14d1b082-b0f7-4795-ab6e-cfc82d67ae8f-maas-api-xks-ca\") on node \"ip-10-0-134-125.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:22:05.258887 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:22:05.258831 4405 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/14d1b082-b0f7-4795-ab6e-cfc82d67ae8f-maas-api-openshift-ca\") on node \"ip-10-0-134-125.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:22:05.258887 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:22:05.258841 4405 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xdgcd\" (UniqueName: \"kubernetes.io/projected/14d1b082-b0f7-4795-ab6e-cfc82d67ae8f-kube-api-access-xdgcd\") on node \"ip-10-0-134-125.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:22:05.928826 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:22:05.928790 4405 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-g9srh" event={"ID":"14d1b082-b0f7-4795-ab6e-cfc82d67ae8f","Type":"ContainerDied","Data":"3064bd1647f53a5a668d2005383eacc1109f70632895d2d8703ce8c775f24d5e"} Nov 04 03:22:05.928826 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:22:05.928823 4405 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-g9srh" Nov 04 03:22:05.929050 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:22:05.928839 4405 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3064bd1647f53a5a668d2005383eacc1109f70632895d2d8703ce8c775f24d5e" Nov 04 03:22:06.882996 ip-10-0-134-125 kubenswrapper[4405]: E1104 03:22:06.882959 4405 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5f2vb" containerName="extractor" Nov 04 03:22:12.305822 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:22:12.305788 4405 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-jdmvl"] Nov 04 03:22:12.306368 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:22:12.306346 4405 cpu_manager.go:414] "containerMap: removing container" podUID="14d1b082-b0f7-4795-ab6e-cfc82d67ae8f" containerName="revoke-keys" Nov 04 03:22:12.306457 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:22:12.306371 4405 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="14d1b082-b0f7-4795-ab6e-cfc82d67ae8f" containerName="revoke-keys" Nov 04 03:22:12.306516 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:22:12.306477 4405 memory_manager.go:365] "RemoveStaleState removing state" podUID="14d1b082-b0f7-4795-ab6e-cfc82d67ae8f" containerName="revoke-keys" Nov 04 03:22:12.309728 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:22:12.309706 4405 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-jdmvl" Nov 04 03:22:12.367433 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:22:12.367341 4405 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-jdmvl"] Nov 04 03:22:12.413990 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:22:12.413931 4405 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 04 03:22:12.413990 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:22:12.413942 4405 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 04 03:22:12.414235 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:22:12.413967 4405 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 04 03:22:12.414235 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:22:12.414069 4405 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-slbq8\"" Nov 04 03:22:12.429513 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:22:12.429481 4405 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kr8rr\" (UniqueName: \"kubernetes.io/projected/2108b065-456e-48e6-9bc9-78dc33327f20-kube-api-access-kr8rr\") pod \"maas-api-revoke-sub-d3fba056-jdmvl\" (UID: \"2108b065-456e-48e6-9bc9-78dc33327f20\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-jdmvl" Nov 04 03:22:12.429692 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:22:12.429529 4405 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2108b065-456e-48e6-9bc9-78dc33327f20-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d3fba056-jdmvl\" (UID: \"2108b065-456e-48e6-9bc9-78dc33327f20\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-jdmvl" Nov 04 03:22:12.429692 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:22:12.429548 4405 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2108b065-456e-48e6-9bc9-78dc33327f20-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d3fba056-jdmvl\" (UID: \"2108b065-456e-48e6-9bc9-78dc33327f20\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-jdmvl" Nov 04 03:22:12.531067 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:22:12.531031 4405 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kr8rr\" (UniqueName: \"kubernetes.io/projected/2108b065-456e-48e6-9bc9-78dc33327f20-kube-api-access-kr8rr\") pod \"maas-api-revoke-sub-d3fba056-jdmvl\" (UID: \"2108b065-456e-48e6-9bc9-78dc33327f20\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-jdmvl" Nov 04 03:22:12.531254 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:22:12.531091 4405 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2108b065-456e-48e6-9bc9-78dc33327f20-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d3fba056-jdmvl\" (UID: \"2108b065-456e-48e6-9bc9-78dc33327f20\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-jdmvl" Nov 04 03:22:12.531254 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:22:12.531147 4405 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2108b065-456e-48e6-9bc9-78dc33327f20-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d3fba056-jdmvl\" (UID: \"2108b065-456e-48e6-9bc9-78dc33327f20\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-jdmvl" Nov 04 03:22:12.531997 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:22:12.531949 4405 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2108b065-456e-48e6-9bc9-78dc33327f20-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d3fba056-jdmvl\" (UID: \"2108b065-456e-48e6-9bc9-78dc33327f20\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-jdmvl" Nov 04 03:22:12.533730 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:22:12.533704 4405 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2108b065-456e-48e6-9bc9-78dc33327f20-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d3fba056-jdmvl\" (UID: \"2108b065-456e-48e6-9bc9-78dc33327f20\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-jdmvl" Nov 04 03:22:12.592124 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:22:12.592082 4405 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kr8rr\" (UniqueName: \"kubernetes.io/projected/2108b065-456e-48e6-9bc9-78dc33327f20-kube-api-access-kr8rr\") pod \"maas-api-revoke-sub-d3fba056-jdmvl\" (UID: \"2108b065-456e-48e6-9bc9-78dc33327f20\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-jdmvl" Nov 04 03:22:12.622778 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:22:12.622748 4405 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-jdmvl" Nov 04 03:22:13.001925 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:22:13.001896 4405 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-jdmvl"] Nov 04 03:22:13.004320 ip-10-0-134-125 kubenswrapper[4405]: W1104 03:22:13.004290 4405 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2108b065_456e_48e6_9bc9_78dc33327f20.slice/crio-efc59d6b06af18cacf2c90b1e4e93b7567e6b02a44215a8e61c9738acd375b95 WatchSource:0}: Error finding container efc59d6b06af18cacf2c90b1e4e93b7567e6b02a44215a8e61c9738acd375b95: Status 404 returned error can't find the container with id efc59d6b06af18cacf2c90b1e4e93b7567e6b02a44215a8e61c9738acd375b95 Nov 04 03:22:13.976146 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:22:13.976109 4405 generic.go:356] "Generic (PLEG): container finished" podID="2108b065-456e-48e6-9bc9-78dc33327f20" containerID="093db2af3c688c900cd8e89831c23df16d07ced00cab05013b3a807cb4c5d141" exitCode=0 Nov 04 03:22:13.976572 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:22:13.976189 4405 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-jdmvl" event={"ID":"2108b065-456e-48e6-9bc9-78dc33327f20","Type":"ContainerDied","Data":"093db2af3c688c900cd8e89831c23df16d07ced00cab05013b3a807cb4c5d141"} Nov 04 03:22:13.976572 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:22:13.976243 4405 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-jdmvl" event={"ID":"2108b065-456e-48e6-9bc9-78dc33327f20","Type":"ContainerStarted","Data":"efc59d6b06af18cacf2c90b1e4e93b7567e6b02a44215a8e61c9738acd375b95"} Nov 04 03:22:14.536047 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:22:14.536013 4405 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-bl5mg"] Nov 04 03:22:15.155577 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:22:15.155544 4405 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-jdmvl" Nov 04 03:22:15.260337 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:22:15.260227 4405 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2108b065-456e-48e6-9bc9-78dc33327f20-kube-api-access-kr8rr\" (UniqueName: \"kubernetes.io/projected/2108b065-456e-48e6-9bc9-78dc33327f20-kube-api-access-kr8rr\") pod \"2108b065-456e-48e6-9bc9-78dc33327f20\" (UID: \"2108b065-456e-48e6-9bc9-78dc33327f20\") " Nov 04 03:22:15.260337 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:22:15.260294 4405 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2108b065-456e-48e6-9bc9-78dc33327f20-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2108b065-456e-48e6-9bc9-78dc33327f20-maas-api-xks-ca\") pod \"2108b065-456e-48e6-9bc9-78dc33327f20\" (UID: \"2108b065-456e-48e6-9bc9-78dc33327f20\") " Nov 04 03:22:15.260609 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:22:15.260410 4405 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2108b065-456e-48e6-9bc9-78dc33327f20-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2108b065-456e-48e6-9bc9-78dc33327f20-maas-api-openshift-ca\") pod \"2108b065-456e-48e6-9bc9-78dc33327f20\" (UID: \"2108b065-456e-48e6-9bc9-78dc33327f20\") " Nov 04 03:22:15.261172 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:22:15.261035 4405 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2108b065-456e-48e6-9bc9-78dc33327f20-maas-api-openshift-ca" pod "2108b065-456e-48e6-9bc9-78dc33327f20" (UID: "2108b065-456e-48e6-9bc9-78dc33327f20"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 03:22:15.262639 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:22:15.262609 4405 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2108b065-456e-48e6-9bc9-78dc33327f20-maas-api-xks-ca" pod "2108b065-456e-48e6-9bc9-78dc33327f20" (UID: "2108b065-456e-48e6-9bc9-78dc33327f20"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 04 03:22:15.263132 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:22:15.263109 4405 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2108b065-456e-48e6-9bc9-78dc33327f20-kube-api-access-kr8rr" pod "2108b065-456e-48e6-9bc9-78dc33327f20" (UID: "2108b065-456e-48e6-9bc9-78dc33327f20"). InnerVolumeSpecName "kube-api-access-kr8rr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 03:22:15.361511 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:22:15.361461 4405 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2108b065-456e-48e6-9bc9-78dc33327f20-maas-api-xks-ca\") on node \"ip-10-0-134-125.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:22:15.361511 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:22:15.361505 4405 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2108b065-456e-48e6-9bc9-78dc33327f20-maas-api-openshift-ca\") on node \"ip-10-0-134-125.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:22:15.361511 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:22:15.361520 4405 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kr8rr\" (UniqueName: \"kubernetes.io/projected/2108b065-456e-48e6-9bc9-78dc33327f20-kube-api-access-kr8rr\") on node \"ip-10-0-134-125.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:22:15.990899 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:22:15.989837 4405 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-jdmvl" event={"ID":"2108b065-456e-48e6-9bc9-78dc33327f20","Type":"ContainerDied","Data":"efc59d6b06af18cacf2c90b1e4e93b7567e6b02a44215a8e61c9738acd375b95"} Nov 04 03:22:15.990899 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:22:15.989901 4405 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="efc59d6b06af18cacf2c90b1e4e93b7567e6b02a44215a8e61c9738acd375b95" Nov 04 03:22:15.990899 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:22:15.989970 4405 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-jdmvl" Nov 04 03:22:25.132446 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:22:25.132409 4405 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-bl5mg"] Nov 04 03:22:34.177758 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:22:34.177725 4405 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-vwcx8_ffbb2d83-5a93-4a5b-bce3-e40afb8f58ea/ovn-acl-logging/0.log" Nov 04 03:22:34.186694 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:22:34.186637 4405 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-vwcx8_ffbb2d83-5a93-4a5b-bce3-e40afb8f58ea/ovn-acl-logging/0.log" Nov 04 03:22:36.884060 ip-10-0-134-125 kubenswrapper[4405]: E1104 03:22:36.884019 4405 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5f2vb" containerName="extractor" Nov 04 03:22:37.953879 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:22:37.953846 4405 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-bl5mg"] Nov 04 03:22:47.055299 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:22:47.055210 4405 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-8stcp"] Nov 04 03:22:47.055764 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:22:47.055567 4405 cpu_manager.go:414] "containerMap: removing container" podUID="2108b065-456e-48e6-9bc9-78dc33327f20" containerName="revoke-keys" Nov 04 03:22:47.055764 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:22:47.055577 4405 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2108b065-456e-48e6-9bc9-78dc33327f20" containerName="revoke-keys" Nov 04 03:22:47.055764 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:22:47.055654 4405 memory_manager.go:365] "RemoveStaleState removing state" podUID="2108b065-456e-48e6-9bc9-78dc33327f20" containerName="revoke-keys" Nov 04 03:22:47.058936 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:22:47.058917 4405 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-8stcp" Nov 04 03:22:47.116278 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:22:47.116241 4405 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 04 03:22:47.165399 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:22:47.165358 4405 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 04 03:22:47.165586 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:22:47.165437 4405 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-slbq8\"" Nov 04 03:22:47.165586 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:22:47.165455 4405 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 04 03:22:47.167424 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:22:47.167403 4405 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-8stcp"] Nov 04 03:22:47.209044 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:22:47.209011 4405 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e182532f-6946-4215-a1f6-9015142edcbf-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ae594979-8stcp\" (UID: \"e182532f-6946-4215-a1f6-9015142edcbf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-8stcp" Nov 04 03:22:47.209211 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:22:47.209091 4405 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e182532f-6946-4215-a1f6-9015142edcbf-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ae594979-8stcp\" (UID: \"e182532f-6946-4215-a1f6-9015142edcbf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-8stcp" Nov 04 03:22:47.209211 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:22:47.209158 4405 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d5xgh\" (UniqueName: \"kubernetes.io/projected/e182532f-6946-4215-a1f6-9015142edcbf-kube-api-access-d5xgh\") pod \"maas-api-revoke-sub-ae594979-8stcp\" (UID: \"e182532f-6946-4215-a1f6-9015142edcbf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-8stcp" Nov 04 03:22:47.310555 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:22:47.310454 4405 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e182532f-6946-4215-a1f6-9015142edcbf-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ae594979-8stcp\" (UID: \"e182532f-6946-4215-a1f6-9015142edcbf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-8stcp" Nov 04 03:22:47.310555 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:22:47.310500 4405 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d5xgh\" (UniqueName: \"kubernetes.io/projected/e182532f-6946-4215-a1f6-9015142edcbf-kube-api-access-d5xgh\") pod \"maas-api-revoke-sub-ae594979-8stcp\" (UID: \"e182532f-6946-4215-a1f6-9015142edcbf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-8stcp" Nov 04 03:22:47.310818 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:22:47.310577 4405 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e182532f-6946-4215-a1f6-9015142edcbf-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ae594979-8stcp\" (UID: \"e182532f-6946-4215-a1f6-9015142edcbf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-8stcp" Nov 04 03:22:47.311416 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:22:47.311394 4405 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e182532f-6946-4215-a1f6-9015142edcbf-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ae594979-8stcp\" (UID: \"e182532f-6946-4215-a1f6-9015142edcbf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-8stcp" Nov 04 03:22:47.313141 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:22:47.313108 4405 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e182532f-6946-4215-a1f6-9015142edcbf-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ae594979-8stcp\" (UID: \"e182532f-6946-4215-a1f6-9015142edcbf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-8stcp" Nov 04 03:22:47.375127 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:22:47.375085 4405 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d5xgh\" (UniqueName: \"kubernetes.io/projected/e182532f-6946-4215-a1f6-9015142edcbf-kube-api-access-d5xgh\") pod \"maas-api-revoke-sub-ae594979-8stcp\" (UID: \"e182532f-6946-4215-a1f6-9015142edcbf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-8stcp" Nov 04 03:22:47.671810 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:22:47.671770 4405 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-8stcp" Nov 04 03:22:48.050294 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:22:48.050265 4405 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-8stcp"] Nov 04 03:22:48.052597 ip-10-0-134-125 kubenswrapper[4405]: W1104 03:22:48.052565 4405 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode182532f_6946_4215_a1f6_9015142edcbf.slice/crio-c9d039c03ce6e7f0316ce72bf6713f01732dec2485da364218a86a988484c931 WatchSource:0}: Error finding container c9d039c03ce6e7f0316ce72bf6713f01732dec2485da364218a86a988484c931: Status 404 returned error can't find the container with id c9d039c03ce6e7f0316ce72bf6713f01732dec2485da364218a86a988484c931 Nov 04 03:22:48.155185 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:22:48.155056 4405 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-8stcp" event={"ID":"e182532f-6946-4215-a1f6-9015142edcbf","Type":"ContainerStarted","Data":"c9d039c03ce6e7f0316ce72bf6713f01732dec2485da364218a86a988484c931"} Nov 04 03:22:49.161820 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:22:49.161728 4405 generic.go:356] "Generic (PLEG): container finished" podID="e182532f-6946-4215-a1f6-9015142edcbf" containerID="78bc6f432dc5e7c3db66269e82669a8c9c70e34e4db04db0678a85160ef81a50" exitCode=0 Nov 04 03:22:49.162281 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:22:49.161811 4405 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-8stcp" event={"ID":"e182532f-6946-4215-a1f6-9015142edcbf","Type":"ContainerDied","Data":"78bc6f432dc5e7c3db66269e82669a8c9c70e34e4db04db0678a85160ef81a50"} Nov 04 03:22:50.302281 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:22:50.302255 4405 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-8stcp" Nov 04 03:22:50.442227 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:22:50.442130 4405 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e182532f-6946-4215-a1f6-9015142edcbf-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e182532f-6946-4215-a1f6-9015142edcbf-maas-api-openshift-ca\") pod \"e182532f-6946-4215-a1f6-9015142edcbf\" (UID: \"e182532f-6946-4215-a1f6-9015142edcbf\") " Nov 04 03:22:50.442415 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:22:50.442317 4405 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e182532f-6946-4215-a1f6-9015142edcbf-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e182532f-6946-4215-a1f6-9015142edcbf-maas-api-xks-ca\") pod \"e182532f-6946-4215-a1f6-9015142edcbf\" (UID: \"e182532f-6946-4215-a1f6-9015142edcbf\") " Nov 04 03:22:50.442415 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:22:50.442383 4405 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e182532f-6946-4215-a1f6-9015142edcbf-kube-api-access-d5xgh\" (UniqueName: \"kubernetes.io/projected/e182532f-6946-4215-a1f6-9015142edcbf-kube-api-access-d5xgh\") pod \"e182532f-6946-4215-a1f6-9015142edcbf\" (UID: \"e182532f-6946-4215-a1f6-9015142edcbf\") " Nov 04 03:22:50.442541 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:22:50.442497 4405 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e182532f-6946-4215-a1f6-9015142edcbf-maas-api-openshift-ca" pod "e182532f-6946-4215-a1f6-9015142edcbf" (UID: "e182532f-6946-4215-a1f6-9015142edcbf"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 03:22:50.442756 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:22:50.442736 4405 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e182532f-6946-4215-a1f6-9015142edcbf-maas-api-openshift-ca\") on node \"ip-10-0-134-125.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:22:50.444789 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:22:50.444753 4405 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e182532f-6946-4215-a1f6-9015142edcbf-maas-api-xks-ca" pod "e182532f-6946-4215-a1f6-9015142edcbf" (UID: "e182532f-6946-4215-a1f6-9015142edcbf"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 04 03:22:50.444895 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:22:50.444760 4405 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e182532f-6946-4215-a1f6-9015142edcbf-kube-api-access-d5xgh" pod "e182532f-6946-4215-a1f6-9015142edcbf" (UID: "e182532f-6946-4215-a1f6-9015142edcbf"). InnerVolumeSpecName "kube-api-access-d5xgh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 03:22:50.544028 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:22:50.543985 4405 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e182532f-6946-4215-a1f6-9015142edcbf-maas-api-xks-ca\") on node \"ip-10-0-134-125.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:22:50.544028 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:22:50.544019 4405 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d5xgh\" (UniqueName: \"kubernetes.io/projected/e182532f-6946-4215-a1f6-9015142edcbf-kube-api-access-d5xgh\") on node \"ip-10-0-134-125.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:22:51.172248 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:22:51.172210 4405 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-8stcp" event={"ID":"e182532f-6946-4215-a1f6-9015142edcbf","Type":"ContainerDied","Data":"c9d039c03ce6e7f0316ce72bf6713f01732dec2485da364218a86a988484c931"} Nov 04 03:22:51.172248 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:22:51.172245 4405 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-8stcp" Nov 04 03:22:51.172482 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:22:51.172264 4405 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c9d039c03ce6e7f0316ce72bf6713f01732dec2485da364218a86a988484c931" Nov 04 03:23:00.023474 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:23:00.023442 4405 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-bl5mg"] Nov 04 03:23:06.890075 ip-10-0-134-125 kubenswrapper[4405]: E1104 03:23:06.890026 4405 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5f2vb" containerName="extractor" Nov 04 03:23:09.100051 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:23:09.100018 4405 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-vnrfp"] Nov 04 03:23:09.100653 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:23:09.100581 4405 cpu_manager.go:414] "containerMap: removing container" podUID="e182532f-6946-4215-a1f6-9015142edcbf" containerName="revoke-keys" Nov 04 03:23:09.100653 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:23:09.100612 4405 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e182532f-6946-4215-a1f6-9015142edcbf" containerName="revoke-keys" Nov 04 03:23:09.100819 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:23:09.100780 4405 memory_manager.go:365] "RemoveStaleState removing state" podUID="e182532f-6946-4215-a1f6-9015142edcbf" containerName="revoke-keys" Nov 04 03:23:09.103885 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:23:09.103866 4405 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-vnrfp" Nov 04 03:23:09.177560 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:23:09.177523 4405 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ca88b883-b203-4513-b004-be51b52827f3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a4aef7d7-vnrfp\" (UID: \"ca88b883-b203-4513-b004-be51b52827f3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-vnrfp" Nov 04 03:23:09.177754 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:23:09.177581 4405 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ca88b883-b203-4513-b004-be51b52827f3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a4aef7d7-vnrfp\" (UID: \"ca88b883-b203-4513-b004-be51b52827f3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-vnrfp" Nov 04 03:23:09.177815 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:23:09.177751 4405 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-snkss\" (UniqueName: \"kubernetes.io/projected/ca88b883-b203-4513-b004-be51b52827f3-kube-api-access-snkss\") pod \"maas-api-revoke-sub-a4aef7d7-vnrfp\" (UID: \"ca88b883-b203-4513-b004-be51b52827f3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-vnrfp" Nov 04 03:23:09.208722 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:23:09.208651 4405 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 04 03:23:09.208722 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:23:09.208710 4405 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 04 03:23:09.208928 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:23:09.208733 4405 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-slbq8\"" Nov 04 03:23:09.208928 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:23:09.208778 4405 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 04 03:23:09.210644 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:23:09.210625 4405 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-vnrfp"] Nov 04 03:23:09.278881 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:23:09.278852 4405 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ca88b883-b203-4513-b004-be51b52827f3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a4aef7d7-vnrfp\" (UID: \"ca88b883-b203-4513-b004-be51b52827f3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-vnrfp" Nov 04 03:23:09.279077 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:23:09.278911 4405 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-snkss\" (UniqueName: \"kubernetes.io/projected/ca88b883-b203-4513-b004-be51b52827f3-kube-api-access-snkss\") pod \"maas-api-revoke-sub-a4aef7d7-vnrfp\" (UID: \"ca88b883-b203-4513-b004-be51b52827f3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-vnrfp" Nov 04 03:23:09.279077 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:23:09.278973 4405 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ca88b883-b203-4513-b004-be51b52827f3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a4aef7d7-vnrfp\" (UID: \"ca88b883-b203-4513-b004-be51b52827f3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-vnrfp" Nov 04 03:23:09.279737 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:23:09.279709 4405 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ca88b883-b203-4513-b004-be51b52827f3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a4aef7d7-vnrfp\" (UID: \"ca88b883-b203-4513-b004-be51b52827f3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-vnrfp" Nov 04 03:23:09.281561 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:23:09.281530 4405 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ca88b883-b203-4513-b004-be51b52827f3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a4aef7d7-vnrfp\" (UID: \"ca88b883-b203-4513-b004-be51b52827f3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-vnrfp" Nov 04 03:23:09.340414 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:23:09.340376 4405 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-snkss\" (UniqueName: \"kubernetes.io/projected/ca88b883-b203-4513-b004-be51b52827f3-kube-api-access-snkss\") pod \"maas-api-revoke-sub-a4aef7d7-vnrfp\" (UID: \"ca88b883-b203-4513-b004-be51b52827f3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-vnrfp" Nov 04 03:23:09.417415 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:23:09.417331 4405 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-vnrfp" Nov 04 03:23:09.797514 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:23:09.797491 4405 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-vnrfp"] Nov 04 03:23:09.799612 ip-10-0-134-125 kubenswrapper[4405]: W1104 03:23:09.799575 4405 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podca88b883_b203_4513_b004_be51b52827f3.slice/crio-e3a9b2d0134e40686493e3e7f813e60cd676546dfea411437c43a6ee86c3febb WatchSource:0}: Error finding container e3a9b2d0134e40686493e3e7f813e60cd676546dfea411437c43a6ee86c3febb: Status 404 returned error can't find the container with id e3a9b2d0134e40686493e3e7f813e60cd676546dfea411437c43a6ee86c3febb Nov 04 03:23:09.814486 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:23:09.814454 4405 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-bl5mg"] Nov 04 03:23:10.285312 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:23:10.285267 4405 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-vnrfp" event={"ID":"ca88b883-b203-4513-b004-be51b52827f3","Type":"ContainerStarted","Data":"50c71ad3dd2fdaee87a29f92529284365e9b51f6b210c320fb850679e5913258"} Nov 04 03:23:10.285749 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:23:10.285329 4405 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-vnrfp" event={"ID":"ca88b883-b203-4513-b004-be51b52827f3","Type":"ContainerStarted","Data":"e3a9b2d0134e40686493e3e7f813e60cd676546dfea411437c43a6ee86c3febb"} Nov 04 03:23:10.401598 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:23:10.401493 4405 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-vnrfp" podStartSLOduration=1.401478534 podStartE2EDuration="1.401478534s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 03:23:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 03:23:10.39962686 +0000 UTC m=+3036.830326987" watchObservedRunningTime="2026-11-04 03:23:10.401478534 +0000 UTC m=+3036.832178686" Nov 04 03:23:11.292576 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:23:11.292540 4405 generic.go:356] "Generic (PLEG): container finished" podID="ca88b883-b203-4513-b004-be51b52827f3" containerID="50c71ad3dd2fdaee87a29f92529284365e9b51f6b210c320fb850679e5913258" exitCode=0 Nov 04 03:23:11.292999 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:23:11.292639 4405 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-vnrfp" event={"ID":"ca88b883-b203-4513-b004-be51b52827f3","Type":"ContainerDied","Data":"50c71ad3dd2fdaee87a29f92529284365e9b51f6b210c320fb850679e5913258"} Nov 04 03:23:12.441706 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:23:12.441680 4405 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-vnrfp" Nov 04 03:23:12.610870 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:23:12.610833 4405 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ca88b883-b203-4513-b004-be51b52827f3-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ca88b883-b203-4513-b004-be51b52827f3-maas-api-openshift-ca\") pod \"ca88b883-b203-4513-b004-be51b52827f3\" (UID: \"ca88b883-b203-4513-b004-be51b52827f3\") " Nov 04 03:23:12.611121 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:23:12.610900 4405 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ca88b883-b203-4513-b004-be51b52827f3-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ca88b883-b203-4513-b004-be51b52827f3-maas-api-xks-ca\") pod \"ca88b883-b203-4513-b004-be51b52827f3\" (UID: \"ca88b883-b203-4513-b004-be51b52827f3\") " Nov 04 03:23:12.611121 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:23:12.611007 4405 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ca88b883-b203-4513-b004-be51b52827f3-kube-api-access-snkss\" (UniqueName: \"kubernetes.io/projected/ca88b883-b203-4513-b004-be51b52827f3-kube-api-access-snkss\") pod \"ca88b883-b203-4513-b004-be51b52827f3\" (UID: \"ca88b883-b203-4513-b004-be51b52827f3\") " Nov 04 03:23:12.611333 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:23:12.611305 4405 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ca88b883-b203-4513-b004-be51b52827f3-maas-api-openshift-ca" pod "ca88b883-b203-4513-b004-be51b52827f3" (UID: "ca88b883-b203-4513-b004-be51b52827f3"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 03:23:12.613293 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:23:12.613269 4405 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ca88b883-b203-4513-b004-be51b52827f3-maas-api-xks-ca" pod "ca88b883-b203-4513-b004-be51b52827f3" (UID: "ca88b883-b203-4513-b004-be51b52827f3"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 04 03:23:12.613293 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:23:12.613278 4405 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ca88b883-b203-4513-b004-be51b52827f3-kube-api-access-snkss" pod "ca88b883-b203-4513-b004-be51b52827f3" (UID: "ca88b883-b203-4513-b004-be51b52827f3"). InnerVolumeSpecName "kube-api-access-snkss". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 03:23:12.712331 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:23:12.712294 4405 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ca88b883-b203-4513-b004-be51b52827f3-maas-api-openshift-ca\") on node \"ip-10-0-134-125.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:23:12.712331 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:23:12.712334 4405 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ca88b883-b203-4513-b004-be51b52827f3-maas-api-xks-ca\") on node \"ip-10-0-134-125.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:23:12.712564 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:23:12.712350 4405 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-snkss\" (UniqueName: \"kubernetes.io/projected/ca88b883-b203-4513-b004-be51b52827f3-kube-api-access-snkss\") on node \"ip-10-0-134-125.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:23:13.305590 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:23:13.305549 4405 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-vnrfp" event={"ID":"ca88b883-b203-4513-b004-be51b52827f3","Type":"ContainerDied","Data":"e3a9b2d0134e40686493e3e7f813e60cd676546dfea411437c43a6ee86c3febb"} Nov 04 03:23:13.305590 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:23:13.305589 4405 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-vnrfp" Nov 04 03:23:13.305845 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:23:13.305613 4405 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e3a9b2d0134e40686493e3e7f813e60cd676546dfea411437c43a6ee86c3febb" Nov 04 03:23:18.423940 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:23:18.423901 4405 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-bl5mg"] Nov 04 03:23:27.625180 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:23:27.625145 4405 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-bl5mg"] Nov 04 03:23:30.254280 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:23:30.254230 4405 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-cpcwh"] Nov 04 03:23:30.282107 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:23:30.282059 4405 cpu_manager.go:414] "containerMap: removing container" podUID="ca88b883-b203-4513-b004-be51b52827f3" containerName="revoke-keys" Nov 04 03:23:30.282367 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:23:30.282346 4405 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ca88b883-b203-4513-b004-be51b52827f3" containerName="revoke-keys" Nov 04 03:23:30.282839 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:23:30.282814 4405 memory_manager.go:365] "RemoveStaleState removing state" podUID="ca88b883-b203-4513-b004-be51b52827f3" containerName="revoke-keys" Nov 04 03:23:30.289253 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:23:30.289226 4405 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-cpcwh" Nov 04 03:23:30.340695 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:23:30.338406 4405 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-cpcwh"] Nov 04 03:23:30.384956 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:23:30.384922 4405 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 04 03:23:30.385184 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:23:30.385161 4405 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 04 03:23:30.385304 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:23:30.385289 4405 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 04 03:23:30.385457 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:23:30.385427 4405 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-slbq8\"" Nov 04 03:23:30.387279 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:23:30.387254 4405 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4cfd1631-3c74-44ee-a5b2-5af21c02f74b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-79805ca9-cpcwh\" (UID: \"4cfd1631-3c74-44ee-a5b2-5af21c02f74b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-cpcwh" Nov 04 03:23:30.387432 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:23:30.387415 4405 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4kpws\" (UniqueName: \"kubernetes.io/projected/4cfd1631-3c74-44ee-a5b2-5af21c02f74b-kube-api-access-4kpws\") pod \"maas-api-revoke-sub-79805ca9-cpcwh\" (UID: \"4cfd1631-3c74-44ee-a5b2-5af21c02f74b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-cpcwh" Nov 04 03:23:30.387632 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:23:30.387459 4405 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4cfd1631-3c74-44ee-a5b2-5af21c02f74b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-79805ca9-cpcwh\" (UID: \"4cfd1631-3c74-44ee-a5b2-5af21c02f74b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-cpcwh" Nov 04 03:23:30.488940 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:23:30.488906 4405 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4kpws\" (UniqueName: \"kubernetes.io/projected/4cfd1631-3c74-44ee-a5b2-5af21c02f74b-kube-api-access-4kpws\") pod \"maas-api-revoke-sub-79805ca9-cpcwh\" (UID: \"4cfd1631-3c74-44ee-a5b2-5af21c02f74b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-cpcwh" Nov 04 03:23:30.489111 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:23:30.488948 4405 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4cfd1631-3c74-44ee-a5b2-5af21c02f74b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-79805ca9-cpcwh\" (UID: \"4cfd1631-3c74-44ee-a5b2-5af21c02f74b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-cpcwh" Nov 04 03:23:30.489111 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:23:30.488982 4405 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4cfd1631-3c74-44ee-a5b2-5af21c02f74b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-79805ca9-cpcwh\" (UID: \"4cfd1631-3c74-44ee-a5b2-5af21c02f74b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-cpcwh" Nov 04 03:23:30.490729 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:23:30.490565 4405 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4cfd1631-3c74-44ee-a5b2-5af21c02f74b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-79805ca9-cpcwh\" (UID: \"4cfd1631-3c74-44ee-a5b2-5af21c02f74b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-cpcwh" Nov 04 03:23:30.492116 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:23:30.492091 4405 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4cfd1631-3c74-44ee-a5b2-5af21c02f74b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-79805ca9-cpcwh\" (UID: \"4cfd1631-3c74-44ee-a5b2-5af21c02f74b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-cpcwh" Nov 04 03:23:30.561742 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:23:30.561630 4405 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4kpws\" (UniqueName: \"kubernetes.io/projected/4cfd1631-3c74-44ee-a5b2-5af21c02f74b-kube-api-access-4kpws\") pod \"maas-api-revoke-sub-79805ca9-cpcwh\" (UID: \"4cfd1631-3c74-44ee-a5b2-5af21c02f74b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-cpcwh" Nov 04 03:23:30.679035 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:23:30.678990 4405 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-cpcwh" Nov 04 03:23:31.211641 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:23:31.211498 4405 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-cpcwh"] Nov 04 03:23:31.215283 ip-10-0-134-125 kubenswrapper[4405]: W1104 03:23:31.215251 4405 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4cfd1631_3c74_44ee_a5b2_5af21c02f74b.slice/crio-ae9a7b1b059771d17e715917500e61e8579129bf25b0c2606187d1433137b6d7 WatchSource:0}: Error finding container ae9a7b1b059771d17e715917500e61e8579129bf25b0c2606187d1433137b6d7: Status 404 returned error can't find the container with id ae9a7b1b059771d17e715917500e61e8579129bf25b0c2606187d1433137b6d7 Nov 04 03:23:31.396365 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:23:31.396322 4405 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-cpcwh" event={"ID":"4cfd1631-3c74-44ee-a5b2-5af21c02f74b","Type":"ContainerStarted","Data":"ae9a7b1b059771d17e715917500e61e8579129bf25b0c2606187d1433137b6d7"} Nov 04 03:23:32.403006 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:23:32.402919 4405 generic.go:356] "Generic (PLEG): container finished" podID="4cfd1631-3c74-44ee-a5b2-5af21c02f74b" containerID="dce3edf67cda61f1c2f37ed58cf318a842444277ae95cc9241f1de480b71cf53" exitCode=0 Nov 04 03:23:32.403006 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:23:32.402974 4405 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-cpcwh" event={"ID":"4cfd1631-3c74-44ee-a5b2-5af21c02f74b","Type":"ContainerDied","Data":"dce3edf67cda61f1c2f37ed58cf318a842444277ae95cc9241f1de480b71cf53"} Nov 04 03:23:33.547869 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:23:33.547844 4405 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-cpcwh" Nov 04 03:23:33.644527 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:23:33.644495 4405 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4cfd1631-3c74-44ee-a5b2-5af21c02f74b-kube-api-access-4kpws\" (UniqueName: \"kubernetes.io/projected/4cfd1631-3c74-44ee-a5b2-5af21c02f74b-kube-api-access-4kpws\") pod \"4cfd1631-3c74-44ee-a5b2-5af21c02f74b\" (UID: \"4cfd1631-3c74-44ee-a5b2-5af21c02f74b\") " Nov 04 03:23:33.644760 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:23:33.644538 4405 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4cfd1631-3c74-44ee-a5b2-5af21c02f74b-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4cfd1631-3c74-44ee-a5b2-5af21c02f74b-maas-api-openshift-ca\") pod \"4cfd1631-3c74-44ee-a5b2-5af21c02f74b\" (UID: \"4cfd1631-3c74-44ee-a5b2-5af21c02f74b\") " Nov 04 03:23:33.644760 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:23:33.644610 4405 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4cfd1631-3c74-44ee-a5b2-5af21c02f74b-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4cfd1631-3c74-44ee-a5b2-5af21c02f74b-maas-api-xks-ca\") pod \"4cfd1631-3c74-44ee-a5b2-5af21c02f74b\" (UID: \"4cfd1631-3c74-44ee-a5b2-5af21c02f74b\") " Nov 04 03:23:33.645010 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:23:33.644984 4405 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4cfd1631-3c74-44ee-a5b2-5af21c02f74b-maas-api-openshift-ca" pod "4cfd1631-3c74-44ee-a5b2-5af21c02f74b" (UID: "4cfd1631-3c74-44ee-a5b2-5af21c02f74b"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 03:23:33.646908 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:23:33.646877 4405 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4cfd1631-3c74-44ee-a5b2-5af21c02f74b-maas-api-xks-ca" pod "4cfd1631-3c74-44ee-a5b2-5af21c02f74b" (UID: "4cfd1631-3c74-44ee-a5b2-5af21c02f74b"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 04 03:23:33.647015 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:23:33.646942 4405 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4cfd1631-3c74-44ee-a5b2-5af21c02f74b-kube-api-access-4kpws" pod "4cfd1631-3c74-44ee-a5b2-5af21c02f74b" (UID: "4cfd1631-3c74-44ee-a5b2-5af21c02f74b"). InnerVolumeSpecName "kube-api-access-4kpws". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 03:23:33.745593 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:23:33.745562 4405 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4cfd1631-3c74-44ee-a5b2-5af21c02f74b-maas-api-openshift-ca\") on node \"ip-10-0-134-125.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:23:33.745593 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:23:33.745593 4405 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4cfd1631-3c74-44ee-a5b2-5af21c02f74b-maas-api-xks-ca\") on node \"ip-10-0-134-125.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:23:33.745857 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:23:33.745603 4405 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4kpws\" (UniqueName: \"kubernetes.io/projected/4cfd1631-3c74-44ee-a5b2-5af21c02f74b-kube-api-access-4kpws\") on node \"ip-10-0-134-125.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:23:34.413967 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:23:34.413929 4405 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-cpcwh" event={"ID":"4cfd1631-3c74-44ee-a5b2-5af21c02f74b","Type":"ContainerDied","Data":"ae9a7b1b059771d17e715917500e61e8579129bf25b0c2606187d1433137b6d7"} Nov 04 03:23:34.413967 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:23:34.413970 4405 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-cpcwh" Nov 04 03:23:34.414249 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:23:34.413981 4405 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ae9a7b1b059771d17e715917500e61e8579129bf25b0c2606187d1433137b6d7" Nov 04 03:23:36.883212 ip-10-0-134-125 kubenswrapper[4405]: E1104 03:23:36.883178 4405 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5f2vb" containerName="extractor" Nov 04 03:23:38.450727 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:23:38.450691 4405 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-bl5mg"] Nov 04 03:23:47.721448 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:23:47.721410 4405 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-7dh59"] Nov 04 03:23:47.722723 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:23:47.722696 4405 cpu_manager.go:414] "containerMap: removing container" podUID="4cfd1631-3c74-44ee-a5b2-5af21c02f74b" containerName="revoke-keys" Nov 04 03:23:47.722920 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:23:47.722905 4405 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4cfd1631-3c74-44ee-a5b2-5af21c02f74b" containerName="revoke-keys" Nov 04 03:23:47.723137 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:23:47.723122 4405 memory_manager.go:365] "RemoveStaleState removing state" podUID="4cfd1631-3c74-44ee-a5b2-5af21c02f74b" containerName="revoke-keys" Nov 04 03:23:47.746693 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:23:47.744128 4405 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-7dh59" Nov 04 03:23:47.785307 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:23:47.785259 4405 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-7dh59"] Nov 04 03:23:47.851263 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:23:47.851227 4405 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 04 03:23:47.851645 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:23:47.851625 4405 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-slbq8\"" Nov 04 03:23:47.852482 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:23:47.852461 4405 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 04 03:23:47.885374 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:23:47.885339 4405 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 04 03:23:47.886844 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:23:47.886746 4405 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/23d11cb3-9270-41ad-999e-54f71f4d310d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0b7e8edb-7dh59\" (UID: \"23d11cb3-9270-41ad-999e-54f71f4d310d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-7dh59" Nov 04 03:23:47.887108 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:23:47.887085 4405 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/23d11cb3-9270-41ad-999e-54f71f4d310d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0b7e8edb-7dh59\" (UID: \"23d11cb3-9270-41ad-999e-54f71f4d310d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-7dh59" Nov 04 03:23:47.887441 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:23:47.887421 4405 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gvp6m\" (UniqueName: \"kubernetes.io/projected/23d11cb3-9270-41ad-999e-54f71f4d310d-kube-api-access-gvp6m\") pod \"maas-api-revoke-sub-0b7e8edb-7dh59\" (UID: \"23d11cb3-9270-41ad-999e-54f71f4d310d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-7dh59" Nov 04 03:23:47.988930 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:23:47.988827 4405 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/23d11cb3-9270-41ad-999e-54f71f4d310d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0b7e8edb-7dh59\" (UID: \"23d11cb3-9270-41ad-999e-54f71f4d310d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-7dh59" Nov 04 03:23:47.989131 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:23:47.988929 4405 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/23d11cb3-9270-41ad-999e-54f71f4d310d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0b7e8edb-7dh59\" (UID: \"23d11cb3-9270-41ad-999e-54f71f4d310d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-7dh59" Nov 04 03:23:47.989131 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:23:47.989027 4405 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gvp6m\" (UniqueName: \"kubernetes.io/projected/23d11cb3-9270-41ad-999e-54f71f4d310d-kube-api-access-gvp6m\") pod \"maas-api-revoke-sub-0b7e8edb-7dh59\" (UID: \"23d11cb3-9270-41ad-999e-54f71f4d310d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-7dh59" Nov 04 03:23:47.989753 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:23:47.989726 4405 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/23d11cb3-9270-41ad-999e-54f71f4d310d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0b7e8edb-7dh59\" (UID: \"23d11cb3-9270-41ad-999e-54f71f4d310d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-7dh59" Nov 04 03:23:47.991766 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:23:47.991742 4405 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/23d11cb3-9270-41ad-999e-54f71f4d310d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0b7e8edb-7dh59\" (UID: \"23d11cb3-9270-41ad-999e-54f71f4d310d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-7dh59" Nov 04 03:23:48.005692 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:23:48.004760 4405 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76597d6855-7c95j container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.81:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 04 03:23:48.005692 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:23:48.004816 4405 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-7c95j" podUID="76df1601-21fb-40ab-a5ee-433f367453c7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.81:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 04 03:23:48.053399 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:23:48.053364 4405 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gvp6m\" (UniqueName: \"kubernetes.io/projected/23d11cb3-9270-41ad-999e-54f71f4d310d-kube-api-access-gvp6m\") pod \"maas-api-revoke-sub-0b7e8edb-7dh59\" (UID: \"23d11cb3-9270-41ad-999e-54f71f4d310d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-7dh59" Nov 04 03:23:48.066813 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:23:48.066786 4405 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-7dh59" Nov 04 03:23:48.600690 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:23:48.598548 4405 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-7dh59"] Nov 04 03:23:49.500871 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:23:49.500834 4405 generic.go:356] "Generic (PLEG): container finished" podID="23d11cb3-9270-41ad-999e-54f71f4d310d" containerID="e6becfeac434fad4041e4c03d37445de2908da100c1a1790fcc6ca3366cb9463" exitCode=0 Nov 04 03:23:49.501294 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:23:49.500919 4405 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-7dh59" event={"ID":"23d11cb3-9270-41ad-999e-54f71f4d310d","Type":"ContainerDied","Data":"e6becfeac434fad4041e4c03d37445de2908da100c1a1790fcc6ca3366cb9463"} Nov 04 03:23:49.501294 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:23:49.500988 4405 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-7dh59" event={"ID":"23d11cb3-9270-41ad-999e-54f71f4d310d","Type":"ContainerStarted","Data":"8ca78cade8fb74f2417bb50c2994cd040cdc853357697981e31fa8144885742e"} Nov 04 03:23:50.641330 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:23:50.641305 4405 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-7dh59" Nov 04 03:23:50.710785 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:23:50.710751 4405 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/23d11cb3-9270-41ad-999e-54f71f4d310d-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/23d11cb3-9270-41ad-999e-54f71f4d310d-maas-api-xks-ca\") pod \"23d11cb3-9270-41ad-999e-54f71f4d310d\" (UID: \"23d11cb3-9270-41ad-999e-54f71f4d310d\") " Nov 04 03:23:50.710979 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:23:50.710875 4405 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/23d11cb3-9270-41ad-999e-54f71f4d310d-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/23d11cb3-9270-41ad-999e-54f71f4d310d-maas-api-openshift-ca\") pod \"23d11cb3-9270-41ad-999e-54f71f4d310d\" (UID: \"23d11cb3-9270-41ad-999e-54f71f4d310d\") " Nov 04 03:23:50.711055 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:23:50.710975 4405 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/23d11cb3-9270-41ad-999e-54f71f4d310d-kube-api-access-gvp6m\" (UniqueName: \"kubernetes.io/projected/23d11cb3-9270-41ad-999e-54f71f4d310d-kube-api-access-gvp6m\") pod \"23d11cb3-9270-41ad-999e-54f71f4d310d\" (UID: \"23d11cb3-9270-41ad-999e-54f71f4d310d\") " Nov 04 03:23:50.711244 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:23:50.711223 4405 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/23d11cb3-9270-41ad-999e-54f71f4d310d-maas-api-openshift-ca" pod "23d11cb3-9270-41ad-999e-54f71f4d310d" (UID: "23d11cb3-9270-41ad-999e-54f71f4d310d"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 03:23:50.713146 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:23:50.713123 4405 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/23d11cb3-9270-41ad-999e-54f71f4d310d-maas-api-xks-ca" pod "23d11cb3-9270-41ad-999e-54f71f4d310d" (UID: "23d11cb3-9270-41ad-999e-54f71f4d310d"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 04 03:23:50.713251 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:23:50.713148 4405 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/23d11cb3-9270-41ad-999e-54f71f4d310d-kube-api-access-gvp6m" pod "23d11cb3-9270-41ad-999e-54f71f4d310d" (UID: "23d11cb3-9270-41ad-999e-54f71f4d310d"). InnerVolumeSpecName "kube-api-access-gvp6m". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 03:23:50.812464 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:23:50.812376 4405 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/23d11cb3-9270-41ad-999e-54f71f4d310d-maas-api-xks-ca\") on node \"ip-10-0-134-125.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:23:50.812464 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:23:50.812409 4405 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/23d11cb3-9270-41ad-999e-54f71f4d310d-maas-api-openshift-ca\") on node \"ip-10-0-134-125.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:23:50.812464 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:23:50.812423 4405 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gvp6m\" (UniqueName: \"kubernetes.io/projected/23d11cb3-9270-41ad-999e-54f71f4d310d-kube-api-access-gvp6m\") on node \"ip-10-0-134-125.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:23:51.408527 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:23:51.408479 4405 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-bl5mg"] Nov 04 03:23:51.511057 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:23:51.511022 4405 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-7dh59" Nov 04 03:23:51.511057 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:23:51.511049 4405 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-7dh59" event={"ID":"23d11cb3-9270-41ad-999e-54f71f4d310d","Type":"ContainerDied","Data":"8ca78cade8fb74f2417bb50c2994cd040cdc853357697981e31fa8144885742e"} Nov 04 03:23:51.511275 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:23:51.511092 4405 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8ca78cade8fb74f2417bb50c2994cd040cdc853357697981e31fa8144885742e" Nov 04 03:23:59.012048 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:23:59.012012 4405 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-9kmfd"] Nov 04 03:23:59.012437 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:23:59.012412 4405 cpu_manager.go:414] "containerMap: removing container" podUID="23d11cb3-9270-41ad-999e-54f71f4d310d" containerName="revoke-keys" Nov 04 03:23:59.012437 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:23:59.012423 4405 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="23d11cb3-9270-41ad-999e-54f71f4d310d" containerName="revoke-keys" Nov 04 03:23:59.012515 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:23:59.012481 4405 memory_manager.go:365] "RemoveStaleState removing state" podUID="23d11cb3-9270-41ad-999e-54f71f4d310d" containerName="revoke-keys" Nov 04 03:23:59.015773 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:23:59.015750 4405 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-9kmfd" Nov 04 03:23:59.073573 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:23:59.073538 4405 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-9kmfd"] Nov 04 03:23:59.093466 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:23:59.093432 4405 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qbhkj\" (UniqueName: \"kubernetes.io/projected/7f0b79eb-f570-4c96-a6c3-be2fa446539c-kube-api-access-qbhkj\") pod \"maas-api-revoke-sub-3d4156f1-9kmfd\" (UID: \"7f0b79eb-f570-4c96-a6c3-be2fa446539c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-9kmfd" Nov 04 03:23:59.093635 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:23:59.093566 4405 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7f0b79eb-f570-4c96-a6c3-be2fa446539c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-9kmfd\" (UID: \"7f0b79eb-f570-4c96-a6c3-be2fa446539c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-9kmfd" Nov 04 03:23:59.093635 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:23:59.093622 4405 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7f0b79eb-f570-4c96-a6c3-be2fa446539c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-9kmfd\" (UID: \"7f0b79eb-f570-4c96-a6c3-be2fa446539c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-9kmfd" Nov 04 03:23:59.119661 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:23:59.119633 4405 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 04 03:23:59.119954 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:23:59.119808 4405 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 04 03:23:59.119954 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:23:59.119869 4405 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 04 03:23:59.119954 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:23:59.119913 4405 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-slbq8\"" Nov 04 03:23:59.194621 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:23:59.194587 4405 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7f0b79eb-f570-4c96-a6c3-be2fa446539c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-9kmfd\" (UID: \"7f0b79eb-f570-4c96-a6c3-be2fa446539c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-9kmfd" Nov 04 03:23:59.194843 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:23:59.194646 4405 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7f0b79eb-f570-4c96-a6c3-be2fa446539c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-9kmfd\" (UID: \"7f0b79eb-f570-4c96-a6c3-be2fa446539c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-9kmfd" Nov 04 03:23:59.194843 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:23:59.194707 4405 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qbhkj\" (UniqueName: \"kubernetes.io/projected/7f0b79eb-f570-4c96-a6c3-be2fa446539c-kube-api-access-qbhkj\") pod \"maas-api-revoke-sub-3d4156f1-9kmfd\" (UID: \"7f0b79eb-f570-4c96-a6c3-be2fa446539c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-9kmfd" Nov 04 03:23:59.195272 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:23:59.195248 4405 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7f0b79eb-f570-4c96-a6c3-be2fa446539c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-9kmfd\" (UID: \"7f0b79eb-f570-4c96-a6c3-be2fa446539c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-9kmfd" Nov 04 03:23:59.197103 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:23:59.197082 4405 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7f0b79eb-f570-4c96-a6c3-be2fa446539c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-9kmfd\" (UID: \"7f0b79eb-f570-4c96-a6c3-be2fa446539c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-9kmfd" Nov 04 03:23:59.256323 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:23:59.256287 4405 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qbhkj\" (UniqueName: \"kubernetes.io/projected/7f0b79eb-f570-4c96-a6c3-be2fa446539c-kube-api-access-qbhkj\") pod \"maas-api-revoke-sub-3d4156f1-9kmfd\" (UID: \"7f0b79eb-f570-4c96-a6c3-be2fa446539c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-9kmfd" Nov 04 03:23:59.334108 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:23:59.334064 4405 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-9kmfd" Nov 04 03:23:59.713428 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:23:59.713404 4405 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-9kmfd"] Nov 04 03:23:59.716019 ip-10-0-134-125 kubenswrapper[4405]: W1104 03:23:59.715988 4405 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7f0b79eb_f570_4c96_a6c3_be2fa446539c.slice/crio-faecb19f01d2a031a715127a09a04f6f428b325a5b30b033fcda6e2bc3e7e963 WatchSource:0}: Error finding container faecb19f01d2a031a715127a09a04f6f428b325a5b30b033fcda6e2bc3e7e963: Status 404 returned error can't find the container with id faecb19f01d2a031a715127a09a04f6f428b325a5b30b033fcda6e2bc3e7e963 Nov 04 03:23:59.935944 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:23:59.935895 4405 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-bl5mg"] Nov 04 03:24:00.557523 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:24:00.557481 4405 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-9kmfd" event={"ID":"7f0b79eb-f570-4c96-a6c3-be2fa446539c","Type":"ContainerStarted","Data":"dc56731eb209dd2bec6c7a1591522e0ba02f09356abde992d7a252e1ed352fe5"} Nov 04 03:24:00.557987 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:24:00.557550 4405 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-9kmfd" event={"ID":"7f0b79eb-f570-4c96-a6c3-be2fa446539c","Type":"ContainerStarted","Data":"faecb19f01d2a031a715127a09a04f6f428b325a5b30b033fcda6e2bc3e7e963"} Nov 04 03:24:00.675337 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:24:00.675283 4405 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-9kmfd" podStartSLOduration=2.675268651 podStartE2EDuration="2.675268651s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 03:23:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 03:24:00.67339946 +0000 UTC m=+3087.104099581" watchObservedRunningTime="2026-11-04 03:24:00.675268651 +0000 UTC m=+3087.105968770" Nov 04 03:24:01.564963 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:24:01.564921 4405 generic.go:356] "Generic (PLEG): container finished" podID="7f0b79eb-f570-4c96-a6c3-be2fa446539c" containerID="dc56731eb209dd2bec6c7a1591522e0ba02f09356abde992d7a252e1ed352fe5" exitCode=0 Nov 04 03:24:01.565417 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:24:01.564991 4405 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-9kmfd" event={"ID":"7f0b79eb-f570-4c96-a6c3-be2fa446539c","Type":"ContainerDied","Data":"dc56731eb209dd2bec6c7a1591522e0ba02f09356abde992d7a252e1ed352fe5"} Nov 04 03:24:02.718155 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:24:02.718120 4405 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-9kmfd" Nov 04 03:24:02.831030 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:24:02.830937 4405 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7f0b79eb-f570-4c96-a6c3-be2fa446539c-kube-api-access-qbhkj\" (UniqueName: \"kubernetes.io/projected/7f0b79eb-f570-4c96-a6c3-be2fa446539c-kube-api-access-qbhkj\") pod \"7f0b79eb-f570-4c96-a6c3-be2fa446539c\" (UID: \"7f0b79eb-f570-4c96-a6c3-be2fa446539c\") " Nov 04 03:24:02.831216 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:24:02.831067 4405 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7f0b79eb-f570-4c96-a6c3-be2fa446539c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7f0b79eb-f570-4c96-a6c3-be2fa446539c-maas-api-xks-ca\") pod \"7f0b79eb-f570-4c96-a6c3-be2fa446539c\" (UID: \"7f0b79eb-f570-4c96-a6c3-be2fa446539c\") " Nov 04 03:24:02.831216 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:24:02.831107 4405 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7f0b79eb-f570-4c96-a6c3-be2fa446539c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7f0b79eb-f570-4c96-a6c3-be2fa446539c-maas-api-openshift-ca\") pod \"7f0b79eb-f570-4c96-a6c3-be2fa446539c\" (UID: \"7f0b79eb-f570-4c96-a6c3-be2fa446539c\") " Nov 04 03:24:02.831521 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:24:02.831483 4405 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7f0b79eb-f570-4c96-a6c3-be2fa446539c-maas-api-openshift-ca" pod "7f0b79eb-f570-4c96-a6c3-be2fa446539c" (UID: "7f0b79eb-f570-4c96-a6c3-be2fa446539c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 03:24:02.833484 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:24:02.833428 4405 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7f0b79eb-f570-4c96-a6c3-be2fa446539c-maas-api-xks-ca" pod "7f0b79eb-f570-4c96-a6c3-be2fa446539c" (UID: "7f0b79eb-f570-4c96-a6c3-be2fa446539c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 04 03:24:02.834236 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:24:02.834207 4405 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7f0b79eb-f570-4c96-a6c3-be2fa446539c-kube-api-access-qbhkj" pod "7f0b79eb-f570-4c96-a6c3-be2fa446539c" (UID: "7f0b79eb-f570-4c96-a6c3-be2fa446539c"). InnerVolumeSpecName "kube-api-access-qbhkj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 03:24:02.932626 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:24:02.932586 4405 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7f0b79eb-f570-4c96-a6c3-be2fa446539c-maas-api-xks-ca\") on node \"ip-10-0-134-125.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:24:02.932626 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:24:02.932626 4405 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7f0b79eb-f570-4c96-a6c3-be2fa446539c-maas-api-openshift-ca\") on node \"ip-10-0-134-125.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:24:02.932870 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:24:02.932641 4405 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qbhkj\" (UniqueName: \"kubernetes.io/projected/7f0b79eb-f570-4c96-a6c3-be2fa446539c-kube-api-access-qbhkj\") on node \"ip-10-0-134-125.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:24:03.576696 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:24:03.576639 4405 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-9kmfd" Nov 04 03:24:03.576883 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:24:03.576694 4405 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-9kmfd" event={"ID":"7f0b79eb-f570-4c96-a6c3-be2fa446539c","Type":"ContainerDied","Data":"faecb19f01d2a031a715127a09a04f6f428b325a5b30b033fcda6e2bc3e7e963"} Nov 04 03:24:03.576883 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:24:03.576745 4405 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="faecb19f01d2a031a715127a09a04f6f428b325a5b30b033fcda6e2bc3e7e963" Nov 04 03:24:06.883408 ip-10-0-134-125 kubenswrapper[4405]: E1104 03:24:06.883364 4405 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5f2vb" containerName="extractor" Nov 04 03:24:13.721381 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:24:13.721344 4405 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-bl5mg"] Nov 04 03:24:24.025038 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:24:24.025001 4405 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-9kmfd"] Nov 04 03:24:24.074860 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:24:24.074823 4405 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-9kmfd"] Nov 04 03:24:24.153479 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:24:24.153447 4405 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-vswrk"] Nov 04 03:24:24.153868 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:24:24.153854 4405 cpu_manager.go:414] "containerMap: removing container" podUID="7f0b79eb-f570-4c96-a6c3-be2fa446539c" containerName="revoke-keys" Nov 04 03:24:24.153931 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:24:24.153869 4405 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7f0b79eb-f570-4c96-a6c3-be2fa446539c" containerName="revoke-keys" Nov 04 03:24:24.153988 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:24:24.153936 4405 memory_manager.go:365] "RemoveStaleState removing state" podUID="7f0b79eb-f570-4c96-a6c3-be2fa446539c" containerName="revoke-keys" Nov 04 03:24:24.157184 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:24:24.157168 4405 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-vswrk" Nov 04 03:24:24.185330 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:24:24.185300 4405 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/290211e9-9c4f-4fc2-86f2-243b8cecde7b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-vswrk\" (UID: \"290211e9-9c4f-4fc2-86f2-243b8cecde7b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-vswrk" Nov 04 03:24:24.185494 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:24:24.185348 4405 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/290211e9-9c4f-4fc2-86f2-243b8cecde7b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-vswrk\" (UID: \"290211e9-9c4f-4fc2-86f2-243b8cecde7b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-vswrk" Nov 04 03:24:24.185494 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:24:24.185408 4405 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6gv8m\" (UniqueName: \"kubernetes.io/projected/290211e9-9c4f-4fc2-86f2-243b8cecde7b-kube-api-access-6gv8m\") pod \"maas-api-revoke-sub-3d4156f1-vswrk\" (UID: \"290211e9-9c4f-4fc2-86f2-243b8cecde7b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-vswrk" Nov 04 03:24:24.212463 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:24:24.212433 4405 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 04 03:24:24.212624 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:24:24.212434 4405 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 04 03:24:24.212624 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:24:24.212463 4405 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 04 03:24:24.212754 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:24:24.212639 4405 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-slbq8\"" Nov 04 03:24:24.214903 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:24:24.214871 4405 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-vswrk"] Nov 04 03:24:24.285943 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:24:24.285856 4405 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/290211e9-9c4f-4fc2-86f2-243b8cecde7b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-vswrk\" (UID: \"290211e9-9c4f-4fc2-86f2-243b8cecde7b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-vswrk" Nov 04 03:24:24.285943 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:24:24.285900 4405 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/290211e9-9c4f-4fc2-86f2-243b8cecde7b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-vswrk\" (UID: \"290211e9-9c4f-4fc2-86f2-243b8cecde7b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-vswrk" Nov 04 03:24:24.285943 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:24:24.285920 4405 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6gv8m\" (UniqueName: \"kubernetes.io/projected/290211e9-9c4f-4fc2-86f2-243b8cecde7b-kube-api-access-6gv8m\") pod \"maas-api-revoke-sub-3d4156f1-vswrk\" (UID: \"290211e9-9c4f-4fc2-86f2-243b8cecde7b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-vswrk" Nov 04 03:24:24.286625 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:24:24.286600 4405 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/290211e9-9c4f-4fc2-86f2-243b8cecde7b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-vswrk\" (UID: \"290211e9-9c4f-4fc2-86f2-243b8cecde7b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-vswrk" Nov 04 03:24:24.288401 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:24:24.288376 4405 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/290211e9-9c4f-4fc2-86f2-243b8cecde7b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-vswrk\" (UID: \"290211e9-9c4f-4fc2-86f2-243b8cecde7b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-vswrk" Nov 04 03:24:24.346825 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:24:24.346796 4405 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6gv8m\" (UniqueName: \"kubernetes.io/projected/290211e9-9c4f-4fc2-86f2-243b8cecde7b-kube-api-access-6gv8m\") pod \"maas-api-revoke-sub-3d4156f1-vswrk\" (UID: \"290211e9-9c4f-4fc2-86f2-243b8cecde7b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-vswrk" Nov 04 03:24:24.470092 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:24:24.470054 4405 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-vswrk" Nov 04 03:24:24.852639 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:24:24.852615 4405 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-vswrk"] Nov 04 03:24:24.854877 ip-10-0-134-125 kubenswrapper[4405]: W1104 03:24:24.854847 4405 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod290211e9_9c4f_4fc2_86f2_243b8cecde7b.slice/crio-f61cace406601ac7b882f566a55778fd7f32db4e49722d94dc30c11ff3571f63 WatchSource:0}: Error finding container f61cace406601ac7b882f566a55778fd7f32db4e49722d94dc30c11ff3571f63: Status 404 returned error can't find the container with id f61cace406601ac7b882f566a55778fd7f32db4e49722d94dc30c11ff3571f63 Nov 04 03:24:25.063582 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:24:25.063545 4405 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-bl5mg"] Nov 04 03:24:25.708755 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:24:25.708724 4405 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7f0b79eb-f570-4c96-a6c3-be2fa446539c" path="/var/lib/kubelet/pods/7f0b79eb-f570-4c96-a6c3-be2fa446539c/volumes" Nov 04 03:24:25.747444 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:24:25.747412 4405 generic.go:356] "Generic (PLEG): container finished" podID="290211e9-9c4f-4fc2-86f2-243b8cecde7b" containerID="6652fb30b5bd28b13a8550f2902ffcd9e4171c4f4765d1dfc357248848426a59" exitCode=0 Nov 04 03:24:25.747609 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:24:25.747455 4405 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-vswrk" event={"ID":"290211e9-9c4f-4fc2-86f2-243b8cecde7b","Type":"ContainerDied","Data":"6652fb30b5bd28b13a8550f2902ffcd9e4171c4f4765d1dfc357248848426a59"} Nov 04 03:24:25.747609 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:24:25.747515 4405 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-vswrk" event={"ID":"290211e9-9c4f-4fc2-86f2-243b8cecde7b","Type":"ContainerStarted","Data":"f61cace406601ac7b882f566a55778fd7f32db4e49722d94dc30c11ff3571f63"} Nov 04 03:24:26.890271 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:24:26.890247 4405 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-vswrk" Nov 04 03:24:26.911095 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:24:26.911069 4405 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/290211e9-9c4f-4fc2-86f2-243b8cecde7b-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/290211e9-9c4f-4fc2-86f2-243b8cecde7b-maas-api-xks-ca\") pod \"290211e9-9c4f-4fc2-86f2-243b8cecde7b\" (UID: \"290211e9-9c4f-4fc2-86f2-243b8cecde7b\") " Nov 04 03:24:26.911240 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:24:26.911126 4405 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/290211e9-9c4f-4fc2-86f2-243b8cecde7b-kube-api-access-6gv8m\" (UniqueName: \"kubernetes.io/projected/290211e9-9c4f-4fc2-86f2-243b8cecde7b-kube-api-access-6gv8m\") pod \"290211e9-9c4f-4fc2-86f2-243b8cecde7b\" (UID: \"290211e9-9c4f-4fc2-86f2-243b8cecde7b\") " Nov 04 03:24:26.911282 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:24:26.911244 4405 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/290211e9-9c4f-4fc2-86f2-243b8cecde7b-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/290211e9-9c4f-4fc2-86f2-243b8cecde7b-maas-api-openshift-ca\") pod \"290211e9-9c4f-4fc2-86f2-243b8cecde7b\" (UID: \"290211e9-9c4f-4fc2-86f2-243b8cecde7b\") " Nov 04 03:24:26.911571 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:24:26.911548 4405 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/290211e9-9c4f-4fc2-86f2-243b8cecde7b-maas-api-openshift-ca" pod "290211e9-9c4f-4fc2-86f2-243b8cecde7b" (UID: "290211e9-9c4f-4fc2-86f2-243b8cecde7b"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 03:24:26.913294 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:24:26.913260 4405 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/290211e9-9c4f-4fc2-86f2-243b8cecde7b-kube-api-access-6gv8m" pod "290211e9-9c4f-4fc2-86f2-243b8cecde7b" (UID: "290211e9-9c4f-4fc2-86f2-243b8cecde7b"). InnerVolumeSpecName "kube-api-access-6gv8m". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 03:24:26.913294 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:24:26.913268 4405 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/290211e9-9c4f-4fc2-86f2-243b8cecde7b-maas-api-xks-ca" pod "290211e9-9c4f-4fc2-86f2-243b8cecde7b" (UID: "290211e9-9c4f-4fc2-86f2-243b8cecde7b"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 04 03:24:27.012408 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:24:27.012336 4405 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/290211e9-9c4f-4fc2-86f2-243b8cecde7b-maas-api-openshift-ca\") on node \"ip-10-0-134-125.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:24:27.012408 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:24:27.012368 4405 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/290211e9-9c4f-4fc2-86f2-243b8cecde7b-maas-api-xks-ca\") on node \"ip-10-0-134-125.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:24:27.012408 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:24:27.012382 4405 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6gv8m\" (UniqueName: \"kubernetes.io/projected/290211e9-9c4f-4fc2-86f2-243b8cecde7b-kube-api-access-6gv8m\") on node \"ip-10-0-134-125.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:24:27.758847 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:24:27.758815 4405 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-vswrk" Nov 04 03:24:27.759043 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:24:27.758856 4405 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-vswrk" event={"ID":"290211e9-9c4f-4fc2-86f2-243b8cecde7b","Type":"ContainerDied","Data":"f61cace406601ac7b882f566a55778fd7f32db4e49722d94dc30c11ff3571f63"} Nov 04 03:24:27.759043 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:24:27.758904 4405 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f61cace406601ac7b882f566a55778fd7f32db4e49722d94dc30c11ff3571f63" Nov 04 03:24:36.886539 ip-10-0-134-125 kubenswrapper[4405]: E1104 03:24:36.886489 4405 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5f2vb" containerName="extractor" Nov 04 03:24:38.904017 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:24:38.903981 4405 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-bl5mg"] Nov 04 03:24:48.059289 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:24:48.059252 4405 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-m7vks"] Nov 04 03:24:48.109022 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:24:48.108987 4405 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-m7vks"] Nov 04 03:24:49.257205 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:24:49.257166 4405 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-bl5mg"] Nov 04 03:24:49.699515 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:24:49.699481 4405 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="88f03036-1fd4-4370-a32a-b5c3d316fcad" path="/var/lib/kubelet/pods/88f03036-1fd4-4370-a32a-b5c3d316fcad/volumes" Nov 04 03:25:01.248323 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:25:01.248288 4405 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-bl5mg"] Nov 04 03:25:06.883239 ip-10-0-134-125 kubenswrapper[4405]: E1104 03:25:06.883197 4405 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5f2vb" containerName="extractor" Nov 04 03:25:09.571045 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:25:09.571009 4405 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-cq2lz"] Nov 04 03:25:09.571458 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:25:09.571413 4405 cpu_manager.go:414] "containerMap: removing container" podUID="290211e9-9c4f-4fc2-86f2-243b8cecde7b" containerName="revoke-keys" Nov 04 03:25:09.571458 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:25:09.571425 4405 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="290211e9-9c4f-4fc2-86f2-243b8cecde7b" containerName="revoke-keys" Nov 04 03:25:09.571532 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:25:09.571481 4405 memory_manager.go:365] "RemoveStaleState removing state" podUID="290211e9-9c4f-4fc2-86f2-243b8cecde7b" containerName="revoke-keys" Nov 04 03:25:09.574563 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:25:09.574546 4405 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-cq2lz" Nov 04 03:25:09.632228 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:25:09.632195 4405 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-cq2lz"] Nov 04 03:25:09.638004 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:25:09.637976 4405 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 04 03:25:09.649479 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:25:09.649449 4405 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b583181c-d65d-4b4d-93c8-b9dda58822de-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e2fc9c5b-cq2lz\" (UID: \"b583181c-d65d-4b4d-93c8-b9dda58822de\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-cq2lz" Nov 04 03:25:09.649619 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:25:09.649487 4405 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b583181c-d65d-4b4d-93c8-b9dda58822de-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e2fc9c5b-cq2lz\" (UID: \"b583181c-d65d-4b4d-93c8-b9dda58822de\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-cq2lz" Nov 04 03:25:09.649619 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:25:09.649509 4405 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nnxnr\" (UniqueName: \"kubernetes.io/projected/b583181c-d65d-4b4d-93c8-b9dda58822de-kube-api-access-nnxnr\") pod \"maas-api-revoke-sub-e2fc9c5b-cq2lz\" (UID: \"b583181c-d65d-4b4d-93c8-b9dda58822de\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-cq2lz" Nov 04 03:25:09.679030 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:25:09.678997 4405 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-slbq8\"" Nov 04 03:25:09.679190 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:25:09.679153 4405 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 04 03:25:09.679190 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:25:09.679168 4405 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 04 03:25:09.750314 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:25:09.750273 4405 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b583181c-d65d-4b4d-93c8-b9dda58822de-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e2fc9c5b-cq2lz\" (UID: \"b583181c-d65d-4b4d-93c8-b9dda58822de\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-cq2lz" Nov 04 03:25:09.750314 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:25:09.750313 4405 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b583181c-d65d-4b4d-93c8-b9dda58822de-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e2fc9c5b-cq2lz\" (UID: \"b583181c-d65d-4b4d-93c8-b9dda58822de\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-cq2lz" Nov 04 03:25:09.750562 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:25:09.750334 4405 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nnxnr\" (UniqueName: \"kubernetes.io/projected/b583181c-d65d-4b4d-93c8-b9dda58822de-kube-api-access-nnxnr\") pod \"maas-api-revoke-sub-e2fc9c5b-cq2lz\" (UID: \"b583181c-d65d-4b4d-93c8-b9dda58822de\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-cq2lz" Nov 04 03:25:09.751011 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:25:09.750989 4405 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b583181c-d65d-4b4d-93c8-b9dda58822de-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e2fc9c5b-cq2lz\" (UID: \"b583181c-d65d-4b4d-93c8-b9dda58822de\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-cq2lz" Nov 04 03:25:09.752842 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:25:09.752819 4405 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b583181c-d65d-4b4d-93c8-b9dda58822de-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e2fc9c5b-cq2lz\" (UID: \"b583181c-d65d-4b4d-93c8-b9dda58822de\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-cq2lz" Nov 04 03:25:09.811726 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:25:09.811692 4405 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nnxnr\" (UniqueName: \"kubernetes.io/projected/b583181c-d65d-4b4d-93c8-b9dda58822de-kube-api-access-nnxnr\") pod \"maas-api-revoke-sub-e2fc9c5b-cq2lz\" (UID: \"b583181c-d65d-4b4d-93c8-b9dda58822de\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-cq2lz" Nov 04 03:25:09.887407 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:25:09.887368 4405 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-cq2lz" Nov 04 03:25:10.260744 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:25:10.260705 4405 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-cq2lz"] Nov 04 03:25:10.263079 ip-10-0-134-125 kubenswrapper[4405]: W1104 03:25:10.263051 4405 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb583181c_d65d_4b4d_93c8_b9dda58822de.slice/crio-7dbd822b4fff1be19a61a48338ce3e776ad517b442ed113b38734885b80583ac WatchSource:0}: Error finding container 7dbd822b4fff1be19a61a48338ce3e776ad517b442ed113b38734885b80583ac: Status 404 returned error can't find the container with id 7dbd822b4fff1be19a61a48338ce3e776ad517b442ed113b38734885b80583ac Nov 04 03:25:11.013122 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:25:11.013019 4405 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-cq2lz" event={"ID":"b583181c-d65d-4b4d-93c8-b9dda58822de","Type":"ContainerStarted","Data":"d6daffbc70716eaea0881a6e0e1d4fd308f7688c9fb85ac8941c80de27f45624"} Nov 04 03:25:11.013122 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:25:11.013074 4405 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-cq2lz" event={"ID":"b583181c-d65d-4b4d-93c8-b9dda58822de","Type":"ContainerStarted","Data":"7dbd822b4fff1be19a61a48338ce3e776ad517b442ed113b38734885b80583ac"} Nov 04 03:25:11.130391 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:25:11.130336 4405 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-cq2lz" podStartSLOduration=2.130323938 podStartE2EDuration="2.130323938s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 03:25:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 03:25:11.128336956 +0000 UTC m=+3157.559037077" watchObservedRunningTime="2026-11-04 03:25:11.130323938 +0000 UTC m=+3157.561024060" Nov 04 03:25:12.020550 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:25:12.020512 4405 generic.go:356] "Generic (PLEG): container finished" podID="b583181c-d65d-4b4d-93c8-b9dda58822de" containerID="d6daffbc70716eaea0881a6e0e1d4fd308f7688c9fb85ac8941c80de27f45624" exitCode=0 Nov 04 03:25:12.021015 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:25:12.020593 4405 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-cq2lz" event={"ID":"b583181c-d65d-4b4d-93c8-b9dda58822de","Type":"ContainerDied","Data":"d6daffbc70716eaea0881a6e0e1d4fd308f7688c9fb85ac8941c80de27f45624"} Nov 04 03:25:13.165482 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:25:13.165453 4405 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-cq2lz" Nov 04 03:25:13.180695 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:25:13.180648 4405 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b583181c-d65d-4b4d-93c8-b9dda58822de-kube-api-access-nnxnr\" (UniqueName: \"kubernetes.io/projected/b583181c-d65d-4b4d-93c8-b9dda58822de-kube-api-access-nnxnr\") pod \"b583181c-d65d-4b4d-93c8-b9dda58822de\" (UID: \"b583181c-d65d-4b4d-93c8-b9dda58822de\") " Nov 04 03:25:13.180830 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:25:13.180783 4405 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b583181c-d65d-4b4d-93c8-b9dda58822de-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b583181c-d65d-4b4d-93c8-b9dda58822de-maas-api-openshift-ca\") pod \"b583181c-d65d-4b4d-93c8-b9dda58822de\" (UID: \"b583181c-d65d-4b4d-93c8-b9dda58822de\") " Nov 04 03:25:13.180830 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:25:13.180818 4405 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b583181c-d65d-4b4d-93c8-b9dda58822de-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b583181c-d65d-4b4d-93c8-b9dda58822de-maas-api-xks-ca\") pod \"b583181c-d65d-4b4d-93c8-b9dda58822de\" (UID: \"b583181c-d65d-4b4d-93c8-b9dda58822de\") " Nov 04 03:25:13.181163 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:25:13.181137 4405 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b583181c-d65d-4b4d-93c8-b9dda58822de-maas-api-openshift-ca" pod "b583181c-d65d-4b4d-93c8-b9dda58822de" (UID: "b583181c-d65d-4b4d-93c8-b9dda58822de"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 03:25:13.182997 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:25:13.182972 4405 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b583181c-d65d-4b4d-93c8-b9dda58822de-kube-api-access-nnxnr" pod "b583181c-d65d-4b4d-93c8-b9dda58822de" (UID: "b583181c-d65d-4b4d-93c8-b9dda58822de"). InnerVolumeSpecName "kube-api-access-nnxnr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 03:25:13.183082 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:25:13.183028 4405 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b583181c-d65d-4b4d-93c8-b9dda58822de-maas-api-xks-ca" pod "b583181c-d65d-4b4d-93c8-b9dda58822de" (UID: "b583181c-d65d-4b4d-93c8-b9dda58822de"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 04 03:25:13.282445 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:25:13.282350 4405 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nnxnr\" (UniqueName: \"kubernetes.io/projected/b583181c-d65d-4b4d-93c8-b9dda58822de-kube-api-access-nnxnr\") on node \"ip-10-0-134-125.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:25:13.282445 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:25:13.282390 4405 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b583181c-d65d-4b4d-93c8-b9dda58822de-maas-api-openshift-ca\") on node \"ip-10-0-134-125.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:25:13.282445 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:25:13.282406 4405 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b583181c-d65d-4b4d-93c8-b9dda58822de-maas-api-xks-ca\") on node \"ip-10-0-134-125.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:25:14.031354 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:25:14.031256 4405 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-cq2lz" Nov 04 03:25:14.031354 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:25:14.031269 4405 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-cq2lz" event={"ID":"b583181c-d65d-4b4d-93c8-b9dda58822de","Type":"ContainerDied","Data":"7dbd822b4fff1be19a61a48338ce3e776ad517b442ed113b38734885b80583ac"} Nov 04 03:25:14.031539 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:25:14.031362 4405 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7dbd822b4fff1be19a61a48338ce3e776ad517b442ed113b38734885b80583ac" Nov 04 03:25:20.335894 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:25:20.335859 4405 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-nb52q"] Nov 04 03:25:20.336310 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:25:20.336283 4405 cpu_manager.go:414] "containerMap: removing container" podUID="b583181c-d65d-4b4d-93c8-b9dda58822de" containerName="revoke-keys" Nov 04 03:25:20.336310 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:25:20.336295 4405 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b583181c-d65d-4b4d-93c8-b9dda58822de" containerName="revoke-keys" Nov 04 03:25:20.336397 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:25:20.336360 4405 memory_manager.go:365] "RemoveStaleState removing state" podUID="b583181c-d65d-4b4d-93c8-b9dda58822de" containerName="revoke-keys" Nov 04 03:25:20.339439 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:25:20.339397 4405 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-nb52q" Nov 04 03:25:20.397199 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:25:20.397165 4405 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-nb52q"] Nov 04 03:25:20.443342 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:25:20.443307 4405 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 04 03:25:20.443342 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:25:20.443324 4405 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 04 03:25:20.443578 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:25:20.443383 4405 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 04 03:25:20.443578 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:25:20.443454 4405 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-slbq8\"" Nov 04 03:25:20.449802 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:25:20.449774 4405 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f679ff00-5722-4370-afa1-455ae3b28127-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5a29efb3-nb52q\" (UID: \"f679ff00-5722-4370-afa1-455ae3b28127\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-nb52q" Nov 04 03:25:20.449916 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:25:20.449896 4405 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f679ff00-5722-4370-afa1-455ae3b28127-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5a29efb3-nb52q\" (UID: \"f679ff00-5722-4370-afa1-455ae3b28127\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-nb52q" Nov 04 03:25:20.449979 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:25:20.449966 4405 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4hblh\" (UniqueName: \"kubernetes.io/projected/f679ff00-5722-4370-afa1-455ae3b28127-kube-api-access-4hblh\") pod \"maas-api-revoke-sub-5a29efb3-nb52q\" (UID: \"f679ff00-5722-4370-afa1-455ae3b28127\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-nb52q" Nov 04 03:25:20.551172 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:25:20.551135 4405 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f679ff00-5722-4370-afa1-455ae3b28127-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5a29efb3-nb52q\" (UID: \"f679ff00-5722-4370-afa1-455ae3b28127\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-nb52q" Nov 04 03:25:20.551369 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:25:20.551206 4405 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4hblh\" (UniqueName: \"kubernetes.io/projected/f679ff00-5722-4370-afa1-455ae3b28127-kube-api-access-4hblh\") pod \"maas-api-revoke-sub-5a29efb3-nb52q\" (UID: \"f679ff00-5722-4370-afa1-455ae3b28127\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-nb52q" Nov 04 03:25:20.551369 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:25:20.551254 4405 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f679ff00-5722-4370-afa1-455ae3b28127-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5a29efb3-nb52q\" (UID: \"f679ff00-5722-4370-afa1-455ae3b28127\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-nb52q" Nov 04 03:25:20.551986 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:25:20.551961 4405 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f679ff00-5722-4370-afa1-455ae3b28127-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5a29efb3-nb52q\" (UID: \"f679ff00-5722-4370-afa1-455ae3b28127\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-nb52q" Nov 04 03:25:20.553874 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:25:20.553846 4405 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f679ff00-5722-4370-afa1-455ae3b28127-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5a29efb3-nb52q\" (UID: \"f679ff00-5722-4370-afa1-455ae3b28127\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-nb52q" Nov 04 03:25:20.612609 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:25:20.612525 4405 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4hblh\" (UniqueName: \"kubernetes.io/projected/f679ff00-5722-4370-afa1-455ae3b28127-kube-api-access-4hblh\") pod \"maas-api-revoke-sub-5a29efb3-nb52q\" (UID: \"f679ff00-5722-4370-afa1-455ae3b28127\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-nb52q" Nov 04 03:25:20.651621 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:25:20.651583 4405 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-nb52q" Nov 04 03:25:21.026095 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:25:21.026070 4405 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-nb52q"] Nov 04 03:25:21.027933 ip-10-0-134-125 kubenswrapper[4405]: W1104 03:25:21.027904 4405 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf679ff00_5722_4370_afa1_455ae3b28127.slice/crio-d10df3820305b633803720113e06755129e49f3781be84324fd0fa626b460fb0 WatchSource:0}: Error finding container d10df3820305b633803720113e06755129e49f3781be84324fd0fa626b460fb0: Status 404 returned error can't find the container with id d10df3820305b633803720113e06755129e49f3781be84324fd0fa626b460fb0 Nov 04 03:25:21.066528 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:25:21.066501 4405 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-nb52q" event={"ID":"f679ff00-5722-4370-afa1-455ae3b28127","Type":"ContainerStarted","Data":"d10df3820305b633803720113e06755129e49f3781be84324fd0fa626b460fb0"} Nov 04 03:25:22.072769 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:25:22.072648 4405 generic.go:356] "Generic (PLEG): container finished" podID="f679ff00-5722-4370-afa1-455ae3b28127" containerID="ef907911ce26743bb2e840171c70805ef8023f11671f560a51d71bdb15696216" exitCode=0 Nov 04 03:25:22.073149 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:25:22.072745 4405 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-nb52q" event={"ID":"f679ff00-5722-4370-afa1-455ae3b28127","Type":"ContainerDied","Data":"ef907911ce26743bb2e840171c70805ef8023f11671f560a51d71bdb15696216"} Nov 04 03:25:22.436999 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:25:22.436966 4405 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-bl5mg"] Nov 04 03:25:23.247703 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:25:23.247651 4405 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-nb52q" Nov 04 03:25:23.275657 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:25:23.275616 4405 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f679ff00-5722-4370-afa1-455ae3b28127-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f679ff00-5722-4370-afa1-455ae3b28127-maas-api-xks-ca\") pod \"f679ff00-5722-4370-afa1-455ae3b28127\" (UID: \"f679ff00-5722-4370-afa1-455ae3b28127\") " Nov 04 03:25:23.275860 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:25:23.275655 4405 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f679ff00-5722-4370-afa1-455ae3b28127-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f679ff00-5722-4370-afa1-455ae3b28127-maas-api-openshift-ca\") pod \"f679ff00-5722-4370-afa1-455ae3b28127\" (UID: \"f679ff00-5722-4370-afa1-455ae3b28127\") " Nov 04 03:25:23.275860 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:25:23.275814 4405 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f679ff00-5722-4370-afa1-455ae3b28127-kube-api-access-4hblh\" (UniqueName: \"kubernetes.io/projected/f679ff00-5722-4370-afa1-455ae3b28127-kube-api-access-4hblh\") pod \"f679ff00-5722-4370-afa1-455ae3b28127\" (UID: \"f679ff00-5722-4370-afa1-455ae3b28127\") " Nov 04 03:25:23.276096 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:25:23.276059 4405 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f679ff00-5722-4370-afa1-455ae3b28127-maas-api-openshift-ca" pod "f679ff00-5722-4370-afa1-455ae3b28127" (UID: "f679ff00-5722-4370-afa1-455ae3b28127"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 03:25:23.278211 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:25:23.278182 4405 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f679ff00-5722-4370-afa1-455ae3b28127-kube-api-access-4hblh" pod "f679ff00-5722-4370-afa1-455ae3b28127" (UID: "f679ff00-5722-4370-afa1-455ae3b28127"). InnerVolumeSpecName "kube-api-access-4hblh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 03:25:23.278420 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:25:23.278389 4405 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f679ff00-5722-4370-afa1-455ae3b28127-maas-api-xks-ca" pod "f679ff00-5722-4370-afa1-455ae3b28127" (UID: "f679ff00-5722-4370-afa1-455ae3b28127"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 04 03:25:23.377546 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:25:23.377507 4405 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4hblh\" (UniqueName: \"kubernetes.io/projected/f679ff00-5722-4370-afa1-455ae3b28127-kube-api-access-4hblh\") on node \"ip-10-0-134-125.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:25:23.377546 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:25:23.377552 4405 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f679ff00-5722-4370-afa1-455ae3b28127-maas-api-xks-ca\") on node \"ip-10-0-134-125.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:25:23.377848 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:25:23.377568 4405 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f679ff00-5722-4370-afa1-455ae3b28127-maas-api-openshift-ca\") on node \"ip-10-0-134-125.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:25:24.085618 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:25:24.085585 4405 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-nb52q" event={"ID":"f679ff00-5722-4370-afa1-455ae3b28127","Type":"ContainerDied","Data":"d10df3820305b633803720113e06755129e49f3781be84324fd0fa626b460fb0"} Nov 04 03:25:24.085805 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:25:24.085636 4405 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d10df3820305b633803720113e06755129e49f3781be84324fd0fa626b460fb0" Nov 04 03:25:24.085805 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:25:24.085633 4405 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-nb52q" Nov 04 03:25:33.217192 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:25:33.217150 4405 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-bl5mg"] Nov 04 03:25:36.883080 ip-10-0-134-125 kubenswrapper[4405]: E1104 03:25:36.883037 4405 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5f2vb" containerName="extractor" Nov 04 03:25:39.132737 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:25:39.132699 4405 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-vswrk"] Nov 04 03:25:39.182369 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:25:39.182327 4405 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-vswrk"] Nov 04 03:25:39.243921 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:25:39.243883 4405 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-lw772"] Nov 04 03:25:39.244271 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:25:39.244257 4405 cpu_manager.go:414] "containerMap: removing container" podUID="f679ff00-5722-4370-afa1-455ae3b28127" containerName="revoke-keys" Nov 04 03:25:39.244341 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:25:39.244271 4405 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f679ff00-5722-4370-afa1-455ae3b28127" containerName="revoke-keys" Nov 04 03:25:39.244408 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:25:39.244344 4405 memory_manager.go:365] "RemoveStaleState removing state" podUID="f679ff00-5722-4370-afa1-455ae3b28127" containerName="revoke-keys" Nov 04 03:25:39.247282 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:25:39.247259 4405 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-lw772" Nov 04 03:25:39.304143 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:25:39.304080 4405 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 04 03:25:39.304371 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:25:39.304121 4405 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-slbq8\"" Nov 04 03:25:39.304513 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:25:39.304490 4405 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 04 03:25:39.304700 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:25:39.304683 4405 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 04 03:25:39.307342 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:25:39.307320 4405 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-lw772"] Nov 04 03:25:39.365342 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:25:39.365275 4405 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/98ac0ac8-41cc-4be0-80b7-c9b7515aefce-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-lw772\" (UID: \"98ac0ac8-41cc-4be0-80b7-c9b7515aefce\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-lw772" Nov 04 03:25:39.365551 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:25:39.365391 4405 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/98ac0ac8-41cc-4be0-80b7-c9b7515aefce-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-lw772\" (UID: \"98ac0ac8-41cc-4be0-80b7-c9b7515aefce\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-lw772" Nov 04 03:25:39.365551 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:25:39.365427 4405 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-plr8z\" (UniqueName: \"kubernetes.io/projected/98ac0ac8-41cc-4be0-80b7-c9b7515aefce-kube-api-access-plr8z\") pod \"maas-api-revoke-sub-3d4156f1-lw772\" (UID: \"98ac0ac8-41cc-4be0-80b7-c9b7515aefce\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-lw772" Nov 04 03:25:39.466301 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:25:39.466187 4405 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/98ac0ac8-41cc-4be0-80b7-c9b7515aefce-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-lw772\" (UID: \"98ac0ac8-41cc-4be0-80b7-c9b7515aefce\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-lw772" Nov 04 03:25:39.466301 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:25:39.466253 4405 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/98ac0ac8-41cc-4be0-80b7-c9b7515aefce-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-lw772\" (UID: \"98ac0ac8-41cc-4be0-80b7-c9b7515aefce\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-lw772" Nov 04 03:25:39.466301 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:25:39.466273 4405 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-plr8z\" (UniqueName: \"kubernetes.io/projected/98ac0ac8-41cc-4be0-80b7-c9b7515aefce-kube-api-access-plr8z\") pod \"maas-api-revoke-sub-3d4156f1-lw772\" (UID: \"98ac0ac8-41cc-4be0-80b7-c9b7515aefce\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-lw772" Nov 04 03:25:39.466993 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:25:39.466963 4405 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/98ac0ac8-41cc-4be0-80b7-c9b7515aefce-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-lw772\" (UID: \"98ac0ac8-41cc-4be0-80b7-c9b7515aefce\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-lw772" Nov 04 03:25:39.468816 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:25:39.468798 4405 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/98ac0ac8-41cc-4be0-80b7-c9b7515aefce-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-lw772\" (UID: \"98ac0ac8-41cc-4be0-80b7-c9b7515aefce\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-lw772" Nov 04 03:25:39.527351 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:25:39.527322 4405 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-plr8z\" (UniqueName: \"kubernetes.io/projected/98ac0ac8-41cc-4be0-80b7-c9b7515aefce-kube-api-access-plr8z\") pod \"maas-api-revoke-sub-3d4156f1-lw772\" (UID: \"98ac0ac8-41cc-4be0-80b7-c9b7515aefce\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-lw772" Nov 04 03:25:39.561198 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:25:39.561168 4405 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-lw772" Nov 04 03:25:39.699996 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:25:39.699961 4405 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="290211e9-9c4f-4fc2-86f2-243b8cecde7b" path="/var/lib/kubelet/pods/290211e9-9c4f-4fc2-86f2-243b8cecde7b/volumes" Nov 04 03:25:39.931464 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:25:39.931437 4405 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-lw772"] Nov 04 03:25:39.934303 ip-10-0-134-125 kubenswrapper[4405]: W1104 03:25:39.934265 4405 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod98ac0ac8_41cc_4be0_80b7_c9b7515aefce.slice/crio-2abadf5afc0f1355cea87e83d72d0cd36ddf3c3d784a813214fe05080d050908 WatchSource:0}: Error finding container 2abadf5afc0f1355cea87e83d72d0cd36ddf3c3d784a813214fe05080d050908: Status 404 returned error can't find the container with id 2abadf5afc0f1355cea87e83d72d0cd36ddf3c3d784a813214fe05080d050908 Nov 04 03:25:40.176246 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:25:40.176207 4405 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-lw772" event={"ID":"98ac0ac8-41cc-4be0-80b7-c9b7515aefce","Type":"ContainerStarted","Data":"2abadf5afc0f1355cea87e83d72d0cd36ddf3c3d784a813214fe05080d050908"} Nov 04 03:25:41.182783 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:25:41.182748 4405 generic.go:356] "Generic (PLEG): container finished" podID="98ac0ac8-41cc-4be0-80b7-c9b7515aefce" containerID="b4ddd9014b68b14b9f39769deb693e041106d61935a030006da2aa71f0314210" exitCode=0 Nov 04 03:25:41.183201 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:25:41.182837 4405 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-lw772" event={"ID":"98ac0ac8-41cc-4be0-80b7-c9b7515aefce","Type":"ContainerDied","Data":"b4ddd9014b68b14b9f39769deb693e041106d61935a030006da2aa71f0314210"} Nov 04 03:25:42.338380 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:25:42.338352 4405 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-lw772" Nov 04 03:25:42.392059 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:25:42.392027 4405 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/98ac0ac8-41cc-4be0-80b7-c9b7515aefce-kube-api-access-plr8z\" (UniqueName: \"kubernetes.io/projected/98ac0ac8-41cc-4be0-80b7-c9b7515aefce-kube-api-access-plr8z\") pod \"98ac0ac8-41cc-4be0-80b7-c9b7515aefce\" (UID: \"98ac0ac8-41cc-4be0-80b7-c9b7515aefce\") " Nov 04 03:25:42.392256 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:25:42.392094 4405 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/98ac0ac8-41cc-4be0-80b7-c9b7515aefce-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/98ac0ac8-41cc-4be0-80b7-c9b7515aefce-maas-api-xks-ca\") pod \"98ac0ac8-41cc-4be0-80b7-c9b7515aefce\" (UID: \"98ac0ac8-41cc-4be0-80b7-c9b7515aefce\") " Nov 04 03:25:42.392256 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:25:42.392123 4405 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/98ac0ac8-41cc-4be0-80b7-c9b7515aefce-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/98ac0ac8-41cc-4be0-80b7-c9b7515aefce-maas-api-openshift-ca\") pod \"98ac0ac8-41cc-4be0-80b7-c9b7515aefce\" (UID: \"98ac0ac8-41cc-4be0-80b7-c9b7515aefce\") " Nov 04 03:25:42.392502 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:25:42.392473 4405 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/98ac0ac8-41cc-4be0-80b7-c9b7515aefce-maas-api-openshift-ca" pod "98ac0ac8-41cc-4be0-80b7-c9b7515aefce" (UID: "98ac0ac8-41cc-4be0-80b7-c9b7515aefce"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 03:25:42.394417 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:25:42.394388 4405 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/98ac0ac8-41cc-4be0-80b7-c9b7515aefce-maas-api-xks-ca" pod "98ac0ac8-41cc-4be0-80b7-c9b7515aefce" (UID: "98ac0ac8-41cc-4be0-80b7-c9b7515aefce"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 04 03:25:42.394417 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:25:42.394407 4405 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/98ac0ac8-41cc-4be0-80b7-c9b7515aefce-kube-api-access-plr8z" pod "98ac0ac8-41cc-4be0-80b7-c9b7515aefce" (UID: "98ac0ac8-41cc-4be0-80b7-c9b7515aefce"). InnerVolumeSpecName "kube-api-access-plr8z". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 03:25:42.493532 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:25:42.493447 4405 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-plr8z\" (UniqueName: \"kubernetes.io/projected/98ac0ac8-41cc-4be0-80b7-c9b7515aefce-kube-api-access-plr8z\") on node \"ip-10-0-134-125.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:25:42.493532 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:25:42.493480 4405 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/98ac0ac8-41cc-4be0-80b7-c9b7515aefce-maas-api-xks-ca\") on node \"ip-10-0-134-125.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:25:42.493532 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:25:42.493489 4405 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/98ac0ac8-41cc-4be0-80b7-c9b7515aefce-maas-api-openshift-ca\") on node \"ip-10-0-134-125.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:25:43.194165 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:25:43.194129 4405 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-lw772" event={"ID":"98ac0ac8-41cc-4be0-80b7-c9b7515aefce","Type":"ContainerDied","Data":"2abadf5afc0f1355cea87e83d72d0cd36ddf3c3d784a813214fe05080d050908"} Nov 04 03:25:43.194348 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:25:43.194196 4405 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2abadf5afc0f1355cea87e83d72d0cd36ddf3c3d784a813214fe05080d050908" Nov 04 03:25:43.194348 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:25:43.194141 4405 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-lw772" Nov 04 03:25:51.049249 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:25:51.049169 4405 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-7f6b7"] Nov 04 03:25:51.099409 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:25:51.099370 4405 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-7f6b7"] Nov 04 03:25:51.700186 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:25:51.700148 4405 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1bdb533b-b3cd-4f44-9a0c-eac035fbc90b" path="/var/lib/kubelet/pods/1bdb533b-b3cd-4f44-9a0c-eac035fbc90b/volumes" Nov 04 03:25:53.922106 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:25:53.922058 4405 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-bl5mg"] Nov 04 03:25:55.775763 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:25:55.775718 4405 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-single-main-07ce81-openshift-default-5446f4ff6fwvjql"] Nov 04 03:25:55.776295 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:25:55.776245 4405 cpu_manager.go:414] "containerMap: removing container" podUID="98ac0ac8-41cc-4be0-80b7-c9b7515aefce" containerName="revoke-keys" Nov 04 03:25:55.776295 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:25:55.776266 4405 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="98ac0ac8-41cc-4be0-80b7-c9b7515aefce" containerName="revoke-keys" Nov 04 03:25:55.776421 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:25:55.776401 4405 memory_manager.go:365] "RemoveStaleState removing state" podUID="98ac0ac8-41cc-4be0-80b7-c9b7515aefce" containerName="revoke-keys" Nov 04 03:25:55.780858 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:25:55.780833 4405 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-single-main-07ce81-openshift-default-5446f4ff6fwvjql" Nov 04 03:25:55.836892 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:25:55.836048 4405 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-single-main-07ce81-openshift-default-dockercfg-5d45f\"" Nov 04 03:25:55.847375 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:25:55.847325 4405 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5111ad38-b833-4be2-9a7c-e7db5c2bd133-istio-envoy\") pod \"e2e-worker-single-main-07ce81-openshift-default-5446f4ff6fwvjql\" (UID: \"5111ad38-b833-4be2-9a7c-e7db5c2bd133\") " pod="openshift-ingress/e2e-worker-single-main-07ce81-openshift-default-5446f4ff6fwvjql" Nov 04 03:25:55.847720 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:25:55.847695 4405 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5111ad38-b833-4be2-9a7c-e7db5c2bd133-istiod-ca-cert\") pod \"e2e-worker-single-main-07ce81-openshift-default-5446f4ff6fwvjql\" (UID: \"5111ad38-b833-4be2-9a7c-e7db5c2bd133\") " pod="openshift-ingress/e2e-worker-single-main-07ce81-openshift-default-5446f4ff6fwvjql" Nov 04 03:25:55.847926 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:25:55.847894 4405 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5111ad38-b833-4be2-9a7c-e7db5c2bd133-workload-socket\") pod \"e2e-worker-single-main-07ce81-openshift-default-5446f4ff6fwvjql\" (UID: \"5111ad38-b833-4be2-9a7c-e7db5c2bd133\") " pod="openshift-ingress/e2e-worker-single-main-07ce81-openshift-default-5446f4ff6fwvjql" Nov 04 03:25:55.848196 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:25:55.848162 4405 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v85pq\" (UniqueName: \"kubernetes.io/projected/5111ad38-b833-4be2-9a7c-e7db5c2bd133-kube-api-access-v85pq\") pod \"e2e-worker-single-main-07ce81-openshift-default-5446f4ff6fwvjql\" (UID: \"5111ad38-b833-4be2-9a7c-e7db5c2bd133\") " pod="openshift-ingress/e2e-worker-single-main-07ce81-openshift-default-5446f4ff6fwvjql" Nov 04 03:25:55.848457 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:25:55.848436 4405 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5111ad38-b833-4be2-9a7c-e7db5c2bd133-istio-token\") pod \"e2e-worker-single-main-07ce81-openshift-default-5446f4ff6fwvjql\" (UID: \"5111ad38-b833-4be2-9a7c-e7db5c2bd133\") " pod="openshift-ingress/e2e-worker-single-main-07ce81-openshift-default-5446f4ff6fwvjql" Nov 04 03:25:55.848656 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:25:55.848624 4405 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5111ad38-b833-4be2-9a7c-e7db5c2bd133-workload-certs\") pod \"e2e-worker-single-main-07ce81-openshift-default-5446f4ff6fwvjql\" (UID: \"5111ad38-b833-4be2-9a7c-e7db5c2bd133\") " pod="openshift-ingress/e2e-worker-single-main-07ce81-openshift-default-5446f4ff6fwvjql" Nov 04 03:25:55.848895 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:25:55.848872 4405 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5111ad38-b833-4be2-9a7c-e7db5c2bd133-istio-data\") pod \"e2e-worker-single-main-07ce81-openshift-default-5446f4ff6fwvjql\" (UID: \"5111ad38-b833-4be2-9a7c-e7db5c2bd133\") " pod="openshift-ingress/e2e-worker-single-main-07ce81-openshift-default-5446f4ff6fwvjql" Nov 04 03:25:55.849088 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:25:55.849053 4405 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5111ad38-b833-4be2-9a7c-e7db5c2bd133-credential-socket\") pod \"e2e-worker-single-main-07ce81-openshift-default-5446f4ff6fwvjql\" (UID: \"5111ad38-b833-4be2-9a7c-e7db5c2bd133\") " pod="openshift-ingress/e2e-worker-single-main-07ce81-openshift-default-5446f4ff6fwvjql" Nov 04 03:25:55.853911 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:25:55.853746 4405 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5111ad38-b833-4be2-9a7c-e7db5c2bd133-istio-podinfo\") pod \"e2e-worker-single-main-07ce81-openshift-default-5446f4ff6fwvjql\" (UID: \"5111ad38-b833-4be2-9a7c-e7db5c2bd133\") " pod="openshift-ingress/e2e-worker-single-main-07ce81-openshift-default-5446f4ff6fwvjql" Nov 04 03:25:55.889615 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:25:55.889581 4405 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-single-main-07ce81-openshift-default-5446f4ff6fwvjql"] Nov 04 03:25:55.989424 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:25:55.989386 4405 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5111ad38-b833-4be2-9a7c-e7db5c2bd133-istiod-ca-cert\") pod \"e2e-worker-single-main-07ce81-openshift-default-5446f4ff6fwvjql\" (UID: \"5111ad38-b833-4be2-9a7c-e7db5c2bd133\") " pod="openshift-ingress/e2e-worker-single-main-07ce81-openshift-default-5446f4ff6fwvjql" Nov 04 03:25:55.989761 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:25:55.989735 4405 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5111ad38-b833-4be2-9a7c-e7db5c2bd133-workload-socket\") pod \"e2e-worker-single-main-07ce81-openshift-default-5446f4ff6fwvjql\" (UID: \"5111ad38-b833-4be2-9a7c-e7db5c2bd133\") " pod="openshift-ingress/e2e-worker-single-main-07ce81-openshift-default-5446f4ff6fwvjql" Nov 04 03:25:55.989947 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:25:55.989928 4405 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v85pq\" (UniqueName: \"kubernetes.io/projected/5111ad38-b833-4be2-9a7c-e7db5c2bd133-kube-api-access-v85pq\") pod \"e2e-worker-single-main-07ce81-openshift-default-5446f4ff6fwvjql\" (UID: \"5111ad38-b833-4be2-9a7c-e7db5c2bd133\") " pod="openshift-ingress/e2e-worker-single-main-07ce81-openshift-default-5446f4ff6fwvjql" Nov 04 03:25:55.990116 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:25:55.990099 4405 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5111ad38-b833-4be2-9a7c-e7db5c2bd133-istio-token\") pod \"e2e-worker-single-main-07ce81-openshift-default-5446f4ff6fwvjql\" (UID: \"5111ad38-b833-4be2-9a7c-e7db5c2bd133\") " pod="openshift-ingress/e2e-worker-single-main-07ce81-openshift-default-5446f4ff6fwvjql" Nov 04 03:25:55.990238 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:25:55.990223 4405 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5111ad38-b833-4be2-9a7c-e7db5c2bd133-workload-certs\") pod \"e2e-worker-single-main-07ce81-openshift-default-5446f4ff6fwvjql\" (UID: \"5111ad38-b833-4be2-9a7c-e7db5c2bd133\") " pod="openshift-ingress/e2e-worker-single-main-07ce81-openshift-default-5446f4ff6fwvjql" Nov 04 03:25:55.990355 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:25:55.990337 4405 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5111ad38-b833-4be2-9a7c-e7db5c2bd133-istio-data\") pod \"e2e-worker-single-main-07ce81-openshift-default-5446f4ff6fwvjql\" (UID: \"5111ad38-b833-4be2-9a7c-e7db5c2bd133\") " pod="openshift-ingress/e2e-worker-single-main-07ce81-openshift-default-5446f4ff6fwvjql" Nov 04 03:25:55.990492 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:25:55.990477 4405 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5111ad38-b833-4be2-9a7c-e7db5c2bd133-credential-socket\") pod \"e2e-worker-single-main-07ce81-openshift-default-5446f4ff6fwvjql\" (UID: \"5111ad38-b833-4be2-9a7c-e7db5c2bd133\") " pod="openshift-ingress/e2e-worker-single-main-07ce81-openshift-default-5446f4ff6fwvjql" Nov 04 03:25:55.990604 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:25:55.990588 4405 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5111ad38-b833-4be2-9a7c-e7db5c2bd133-istio-podinfo\") pod \"e2e-worker-single-main-07ce81-openshift-default-5446f4ff6fwvjql\" (UID: \"5111ad38-b833-4be2-9a7c-e7db5c2bd133\") " pod="openshift-ingress/e2e-worker-single-main-07ce81-openshift-default-5446f4ff6fwvjql" Nov 04 03:25:55.990782 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:25:55.990764 4405 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5111ad38-b833-4be2-9a7c-e7db5c2bd133-istio-envoy\") pod \"e2e-worker-single-main-07ce81-openshift-default-5446f4ff6fwvjql\" (UID: \"5111ad38-b833-4be2-9a7c-e7db5c2bd133\") " pod="openshift-ingress/e2e-worker-single-main-07ce81-openshift-default-5446f4ff6fwvjql" Nov 04 03:25:55.991194 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:25:55.990152 4405 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5111ad38-b833-4be2-9a7c-e7db5c2bd133-istiod-ca-cert\") pod \"e2e-worker-single-main-07ce81-openshift-default-5446f4ff6fwvjql\" (UID: \"5111ad38-b833-4be2-9a7c-e7db5c2bd133\") " pod="openshift-ingress/e2e-worker-single-main-07ce81-openshift-default-5446f4ff6fwvjql" Nov 04 03:25:55.991289 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:25:55.991208 4405 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5111ad38-b833-4be2-9a7c-e7db5c2bd133-istio-data\") pod \"e2e-worker-single-main-07ce81-openshift-default-5446f4ff6fwvjql\" (UID: \"5111ad38-b833-4be2-9a7c-e7db5c2bd133\") " pod="openshift-ingress/e2e-worker-single-main-07ce81-openshift-default-5446f4ff6fwvjql" Nov 04 03:25:55.991534 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:25:55.991506 4405 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5111ad38-b833-4be2-9a7c-e7db5c2bd133-workload-certs\") pod \"e2e-worker-single-main-07ce81-openshift-default-5446f4ff6fwvjql\" (UID: \"5111ad38-b833-4be2-9a7c-e7db5c2bd133\") " pod="openshift-ingress/e2e-worker-single-main-07ce81-openshift-default-5446f4ff6fwvjql" Nov 04 03:25:55.991534 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:25:55.991523 4405 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5111ad38-b833-4be2-9a7c-e7db5c2bd133-credential-socket\") pod \"e2e-worker-single-main-07ce81-openshift-default-5446f4ff6fwvjql\" (UID: \"5111ad38-b833-4be2-9a7c-e7db5c2bd133\") " pod="openshift-ingress/e2e-worker-single-main-07ce81-openshift-default-5446f4ff6fwvjql" Nov 04 03:25:55.991704 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:25:55.990417 4405 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5111ad38-b833-4be2-9a7c-e7db5c2bd133-workload-socket\") pod \"e2e-worker-single-main-07ce81-openshift-default-5446f4ff6fwvjql\" (UID: \"5111ad38-b833-4be2-9a7c-e7db5c2bd133\") " pod="openshift-ingress/e2e-worker-single-main-07ce81-openshift-default-5446f4ff6fwvjql" Nov 04 03:25:56.046075 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:25:56.045314 4405 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5111ad38-b833-4be2-9a7c-e7db5c2bd133-istio-envoy\") pod \"e2e-worker-single-main-07ce81-openshift-default-5446f4ff6fwvjql\" (UID: \"5111ad38-b833-4be2-9a7c-e7db5c2bd133\") " pod="openshift-ingress/e2e-worker-single-main-07ce81-openshift-default-5446f4ff6fwvjql" Nov 04 03:25:56.054729 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:25:56.048588 4405 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5111ad38-b833-4be2-9a7c-e7db5c2bd133-istio-podinfo\") pod \"e2e-worker-single-main-07ce81-openshift-default-5446f4ff6fwvjql\" (UID: \"5111ad38-b833-4be2-9a7c-e7db5c2bd133\") " pod="openshift-ingress/e2e-worker-single-main-07ce81-openshift-default-5446f4ff6fwvjql" Nov 04 03:25:56.082347 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:25:56.082321 4405 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5111ad38-b833-4be2-9a7c-e7db5c2bd133-istio-token\") pod \"e2e-worker-single-main-07ce81-openshift-default-5446f4ff6fwvjql\" (UID: \"5111ad38-b833-4be2-9a7c-e7db5c2bd133\") " pod="openshift-ingress/e2e-worker-single-main-07ce81-openshift-default-5446f4ff6fwvjql" Nov 04 03:25:56.088876 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:25:56.088839 4405 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v85pq\" (UniqueName: \"kubernetes.io/projected/5111ad38-b833-4be2-9a7c-e7db5c2bd133-kube-api-access-v85pq\") pod \"e2e-worker-single-main-07ce81-openshift-default-5446f4ff6fwvjql\" (UID: \"5111ad38-b833-4be2-9a7c-e7db5c2bd133\") " pod="openshift-ingress/e2e-worker-single-main-07ce81-openshift-default-5446f4ff6fwvjql" Nov 04 03:25:56.145722 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:25:56.136960 4405 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-single-main-07ce81-openshift-default-5446f4ff6fwvjql" Nov 04 03:25:56.749843 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:25:56.749815 4405 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-single-main-07ce81-openshift-default-5446f4ff6fwvjql"] Nov 04 03:25:56.751326 ip-10-0-134-125 kubenswrapper[4405]: W1104 03:25:56.751301 4405 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5111ad38_b833_4be2_9a7c_e7db5c2bd133.slice/crio-d6f29e4f34a742903616b986a680fa86f972b2c24374c6dc234e5e60aeae202b WatchSource:0}: Error finding container d6f29e4f34a742903616b986a680fa86f972b2c24374c6dc234e5e60aeae202b: Status 404 returned error can't find the container with id d6f29e4f34a742903616b986a680fa86f972b2c24374c6dc234e5e60aeae202b Nov 04 03:25:56.753698 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:25:56.753643 4405 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 04 03:25:56.753794 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:25:56.753730 4405 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 04 03:25:56.753794 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:25:56.753759 4405 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 04 03:25:57.291133 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:25:57.291097 4405 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-07ce81-openshift-default-5446f4ff6fwvjql" event={"ID":"5111ad38-b833-4be2-9a7c-e7db5c2bd133","Type":"ContainerStarted","Data":"38fb836dba0859accccb17373b4887fbef0fcc10896302cc8df2067efa08b6d2"} Nov 04 03:25:57.291529 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:25:57.291159 4405 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-07ce81-openshift-default-5446f4ff6fwvjql" event={"ID":"5111ad38-b833-4be2-9a7c-e7db5c2bd133","Type":"ContainerStarted","Data":"d6f29e4f34a742903616b986a680fa86f972b2c24374c6dc234e5e60aeae202b"} Nov 04 03:25:57.413504 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:25:57.413443 4405 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-single-main-07ce81-openshift-default-5446f4ff6fwvjql" podStartSLOduration=2.413428142 podStartE2EDuration="2.413428142s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 03:25:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 03:25:57.411705007 +0000 UTC m=+3203.842405130" watchObservedRunningTime="2026-11-04 03:25:57.413428142 +0000 UTC m=+3203.844128262" Nov 04 03:25:58.146514 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:25:58.146479 4405 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-single-main-07ce81-openshift-default-5446f4ff6fwvjql" Nov 04 03:25:58.152077 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:25:58.152051 4405 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-single-main-07ce81-openshift-default-5446f4ff6fwvjql" Nov 04 03:25:58.303617 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:25:58.303575 4405 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-single-main-07ce81-openshift-default-5446f4ff6fwvjql" Nov 04 03:25:58.304857 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:25:58.304833 4405 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-single-main-07ce81-openshift-default-5446f4ff6fwvjql" Nov 04 03:26:04.052405 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:26:04.052371 4405 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-07ce81-9ddd44mgvn"] Nov 04 03:26:04.056033 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:26:04.056015 4405 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-07ce81-9ddd44mgvn" Nov 04 03:26:04.113975 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:26:04.113939 4405 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-07ce81-9ddd44mgvn"] Nov 04 03:26:04.160570 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:26:04.160534 4405 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-single-main-07ce81\"" Nov 04 03:26:04.161169 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:26:04.161145 4405 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-single-m-3b68fb47-dockercfg-p8tf9\"" Nov 04 03:26:04.189914 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:26:04.189875 4405 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-07ce81-767654f68x8k9j"] Nov 04 03:26:04.193601 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:26:04.193583 4405 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-07ce81-767654f68x8k9j" Nov 04 03:26:04.196139 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:26:04.196118 4405 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-brv6b\" (UniqueName: \"kubernetes.io/projected/56fc8374-482d-4e8f-8cf9-be24fda3235a-kube-api-access-brv6b\") pod \"payload-pre-processing-e2e-worker-single-main-07ce81-9ddd44mgvn\" (UID: \"56fc8374-482d-4e8f-8cf9-be24fda3235a\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-07ce81-9ddd44mgvn" Nov 04 03:26:04.196210 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:26:04.196156 4405 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/56fc8374-482d-4e8f-8cf9-be24fda3235a-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-single-main-07ce81-9ddd44mgvn\" (UID: \"56fc8374-482d-4e8f-8cf9-be24fda3235a\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-07ce81-9ddd44mgvn" Nov 04 03:26:04.251542 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:26:04.251503 4405 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-07ce81-767654f68x8k9j"] Nov 04 03:26:04.297841 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:26:04.297799 4405 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ln7xk\" (UniqueName: \"kubernetes.io/projected/e398089d-e56a-418b-a820-ca361d65e178-kube-api-access-ln7xk\") pod \"payload-processing-e2e-worker-single-main-07ce81-767654f68x8k9j\" (UID: \"e398089d-e56a-418b-a820-ca361d65e178\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-07ce81-767654f68x8k9j" Nov 04 03:26:04.298025 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:26:04.297888 4405 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-brv6b\" (UniqueName: \"kubernetes.io/projected/56fc8374-482d-4e8f-8cf9-be24fda3235a-kube-api-access-brv6b\") pod \"payload-pre-processing-e2e-worker-single-main-07ce81-9ddd44mgvn\" (UID: \"56fc8374-482d-4e8f-8cf9-be24fda3235a\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-07ce81-9ddd44mgvn" Nov 04 03:26:04.298025 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:26:04.297988 4405 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/56fc8374-482d-4e8f-8cf9-be24fda3235a-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-single-main-07ce81-9ddd44mgvn\" (UID: \"56fc8374-482d-4e8f-8cf9-be24fda3235a\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-07ce81-9ddd44mgvn" Nov 04 03:26:04.298119 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:26:04.298104 4405 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e398089d-e56a-418b-a820-ca361d65e178-plugins-config-volume\") pod \"payload-processing-e2e-worker-single-main-07ce81-767654f68x8k9j\" (UID: \"e398089d-e56a-418b-a820-ca361d65e178\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-07ce81-767654f68x8k9j" Nov 04 03:26:04.298721 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:26:04.298703 4405 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/56fc8374-482d-4e8f-8cf9-be24fda3235a-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-single-main-07ce81-9ddd44mgvn\" (UID: \"56fc8374-482d-4e8f-8cf9-be24fda3235a\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-07ce81-9ddd44mgvn" Nov 04 03:26:04.359614 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:26:04.359575 4405 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-brv6b\" (UniqueName: \"kubernetes.io/projected/56fc8374-482d-4e8f-8cf9-be24fda3235a-kube-api-access-brv6b\") pod \"payload-pre-processing-e2e-worker-single-main-07ce81-9ddd44mgvn\" (UID: \"56fc8374-482d-4e8f-8cf9-be24fda3235a\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-07ce81-9ddd44mgvn" Nov 04 03:26:04.370067 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:26:04.370037 4405 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-07ce81-9ddd44mgvn" Nov 04 03:26:04.399786 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:26:04.399749 4405 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e398089d-e56a-418b-a820-ca361d65e178-plugins-config-volume\") pod \"payload-processing-e2e-worker-single-main-07ce81-767654f68x8k9j\" (UID: \"e398089d-e56a-418b-a820-ca361d65e178\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-07ce81-767654f68x8k9j" Nov 04 03:26:04.399943 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:26:04.399814 4405 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ln7xk\" (UniqueName: \"kubernetes.io/projected/e398089d-e56a-418b-a820-ca361d65e178-kube-api-access-ln7xk\") pod \"payload-processing-e2e-worker-single-main-07ce81-767654f68x8k9j\" (UID: \"e398089d-e56a-418b-a820-ca361d65e178\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-07ce81-767654f68x8k9j" Nov 04 03:26:04.400515 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:26:04.400494 4405 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e398089d-e56a-418b-a820-ca361d65e178-plugins-config-volume\") pod \"payload-processing-e2e-worker-single-main-07ce81-767654f68x8k9j\" (UID: \"e398089d-e56a-418b-a820-ca361d65e178\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-07ce81-767654f68x8k9j" Nov 04 03:26:04.461496 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:26:04.461463 4405 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ln7xk\" (UniqueName: \"kubernetes.io/projected/e398089d-e56a-418b-a820-ca361d65e178-kube-api-access-ln7xk\") pod \"payload-processing-e2e-worker-single-main-07ce81-767654f68x8k9j\" (UID: \"e398089d-e56a-418b-a820-ca361d65e178\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-07ce81-767654f68x8k9j" Nov 04 03:26:04.508108 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:26:04.508081 4405 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-07ce81-767654f68x8k9j" Nov 04 03:26:04.745498 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:26:04.745473 4405 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-07ce81-9ddd44mgvn"] Nov 04 03:26:04.746791 ip-10-0-134-125 kubenswrapper[4405]: W1104 03:26:04.746759 4405 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod56fc8374_482d_4e8f_8cf9_be24fda3235a.slice/crio-84efe3270404402ff9ff22222d8875ea28bee4180afacdb9cff9901f054f12ed WatchSource:0}: Error finding container 84efe3270404402ff9ff22222d8875ea28bee4180afacdb9cff9901f054f12ed: Status 404 returned error can't find the container with id 84efe3270404402ff9ff22222d8875ea28bee4180afacdb9cff9901f054f12ed Nov 04 03:26:04.882537 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:26:04.882514 4405 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-07ce81-767654f68x8k9j"] Nov 04 03:26:04.884399 ip-10-0-134-125 kubenswrapper[4405]: W1104 03:26:04.884368 4405 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode398089d_e56a_418b_a820_ca361d65e178.slice/crio-9859eb6ea91a2e90e42abde64595f6da0a526e48f1d5113b22c04b7d28756e18 WatchSource:0}: Error finding container 9859eb6ea91a2e90e42abde64595f6da0a526e48f1d5113b22c04b7d28756e18: Status 404 returned error can't find the container with id 9859eb6ea91a2e90e42abde64595f6da0a526e48f1d5113b22c04b7d28756e18 Nov 04 03:26:05.343441 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:26:05.343405 4405 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-07ce81-767654f68x8k9j" event={"ID":"e398089d-e56a-418b-a820-ca361d65e178","Type":"ContainerStarted","Data":"076d7b5eb3cecc8cc14986c1ac08ebe2f7f6e7f537d598127f80887d0281a9dc"} Nov 04 03:26:05.344097 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:26:05.343467 4405 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-single-main-07ce81-767654f68x8k9j" Nov 04 03:26:05.344097 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:26:05.343503 4405 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-07ce81-767654f68x8k9j" event={"ID":"e398089d-e56a-418b-a820-ca361d65e178","Type":"ContainerStarted","Data":"9859eb6ea91a2e90e42abde64595f6da0a526e48f1d5113b22c04b7d28756e18"} Nov 04 03:26:05.345036 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:26:05.345015 4405 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-07ce81-9ddd44mgvn" event={"ID":"56fc8374-482d-4e8f-8cf9-be24fda3235a","Type":"ContainerStarted","Data":"6fbbd66c6debcc4ba502a374d108459494b732ee108e7414ea75f8ad1c6fa935"} Nov 04 03:26:05.345124 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:26:05.345052 4405 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-07ce81-9ddd44mgvn" event={"ID":"56fc8374-482d-4e8f-8cf9-be24fda3235a","Type":"ContainerStarted","Data":"84efe3270404402ff9ff22222d8875ea28bee4180afacdb9cff9901f054f12ed"} Nov 04 03:26:05.345170 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:26:05.345133 4405 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-07ce81-9ddd44mgvn" Nov 04 03:26:05.461484 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:26:05.461425 4405 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-single-main-07ce81-767654f68x8k9j" podStartSLOduration=1.461410329 podStartE2EDuration="1.461410329s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 03:26:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 03:26:05.458558571 +0000 UTC m=+3211.889258693" watchObservedRunningTime="2026-11-04 03:26:05.461410329 +0000 UTC m=+3211.892110449" Nov 04 03:26:06.647356 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:26:06.647288 4405 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-07ce81-9ddd44mgvn" podStartSLOduration=2.647272592 podStartE2EDuration="2.647272592s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 03:26:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 03:26:05.574425642 +0000 UTC m=+3212.005125757" watchObservedRunningTime="2026-11-04 03:26:06.647272592 +0000 UTC m=+3213.077972715" Nov 04 03:26:06.648785 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:26:06.648757 4405 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-07ce81-79f4465d54-bnchr"] Nov 04 03:26:06.652322 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:26:06.652301 4405 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-07ce81-79f4465d54-bnchr" Nov 04 03:26:06.704800 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:26:06.704755 4405 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-single-main-07ce81-serving-cert\"" Nov 04 03:26:06.704800 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:26:06.704787 4405 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 04 03:26:06.722397 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:26:06.722363 4405 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d1420326-5df2-4716-989e-7957336b4928-maas-api-tls\") pod \"maas-api-e2e-worker-single-main-07ce81-79f4465d54-bnchr\" (UID: \"d1420326-5df2-4716-989e-7957336b4928\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-07ce81-79f4465d54-bnchr" Nov 04 03:26:06.722595 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:26:06.722429 4405 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d1420326-5df2-4716-989e-7957336b4928-openshift-service-ca\") pod \"maas-api-e2e-worker-single-main-07ce81-79f4465d54-bnchr\" (UID: \"d1420326-5df2-4716-989e-7957336b4928\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-07ce81-79f4465d54-bnchr" Nov 04 03:26:06.722595 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:26:06.722511 4405 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d1420326-5df2-4716-989e-7957336b4928-metrics-certs\") pod \"maas-api-e2e-worker-single-main-07ce81-79f4465d54-bnchr\" (UID: \"d1420326-5df2-4716-989e-7957336b4928\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-07ce81-79f4465d54-bnchr" Nov 04 03:26:06.722595 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:26:06.722569 4405 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9cpcd\" (UniqueName: \"kubernetes.io/projected/d1420326-5df2-4716-989e-7957336b4928-kube-api-access-9cpcd\") pod \"maas-api-e2e-worker-single-main-07ce81-79f4465d54-bnchr\" (UID: \"d1420326-5df2-4716-989e-7957336b4928\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-07ce81-79f4465d54-bnchr" Nov 04 03:26:06.754273 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:26:06.754233 4405 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\"" Nov 04 03:26:06.754456 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:26:06.754442 4405 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-qtjr2\"" Nov 04 03:26:06.754567 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:26:06.754544 4405 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 04 03:26:06.756798 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:26:06.756780 4405 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-07ce81-79f4465d54-bnchr"] Nov 04 03:26:06.824277 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:26:06.824243 4405 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d1420326-5df2-4716-989e-7957336b4928-openshift-service-ca\") pod \"maas-api-e2e-worker-single-main-07ce81-79f4465d54-bnchr\" (UID: \"d1420326-5df2-4716-989e-7957336b4928\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-07ce81-79f4465d54-bnchr" Nov 04 03:26:06.824470 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:26:06.824298 4405 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d1420326-5df2-4716-989e-7957336b4928-metrics-certs\") pod \"maas-api-e2e-worker-single-main-07ce81-79f4465d54-bnchr\" (UID: \"d1420326-5df2-4716-989e-7957336b4928\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-07ce81-79f4465d54-bnchr" Nov 04 03:26:06.824470 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:26:06.824350 4405 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9cpcd\" (UniqueName: \"kubernetes.io/projected/d1420326-5df2-4716-989e-7957336b4928-kube-api-access-9cpcd\") pod \"maas-api-e2e-worker-single-main-07ce81-79f4465d54-bnchr\" (UID: \"d1420326-5df2-4716-989e-7957336b4928\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-07ce81-79f4465d54-bnchr" Nov 04 03:26:06.824470 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:26:06.824455 4405 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d1420326-5df2-4716-989e-7957336b4928-maas-api-tls\") pod \"maas-api-e2e-worker-single-main-07ce81-79f4465d54-bnchr\" (UID: \"d1420326-5df2-4716-989e-7957336b4928\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-07ce81-79f4465d54-bnchr" Nov 04 03:26:06.824648 ip-10-0-134-125 kubenswrapper[4405]: E1104 03:26:06.824579 4405 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-worker-single-main-07ce81-serving-cert: secret "maas-api-e2e-worker-single-main-07ce81-serving-cert" not found Nov 04 03:26:06.824743 ip-10-0-134-125 kubenswrapper[4405]: E1104 03:26:06.824692 4405 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d1420326-5df2-4716-989e-7957336b4928-maas-api-tls podName:d1420326-5df2-4716-989e-7957336b4928 nodeName:}" failed. No retries permitted until 2026-11-04 03:26:07.32465558 +0000 UTC m=+3213.755355684 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/d1420326-5df2-4716-989e-7957336b4928-maas-api-tls") pod "maas-api-e2e-worker-single-main-07ce81-79f4465d54-bnchr" (UID: "d1420326-5df2-4716-989e-7957336b4928") : secret "maas-api-e2e-worker-single-main-07ce81-serving-cert" not found Nov 04 03:26:06.825016 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:26:06.824994 4405 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d1420326-5df2-4716-989e-7957336b4928-openshift-service-ca\") pod \"maas-api-e2e-worker-single-main-07ce81-79f4465d54-bnchr\" (UID: \"d1420326-5df2-4716-989e-7957336b4928\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-07ce81-79f4465d54-bnchr" Nov 04 03:26:06.827068 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:26:06.827048 4405 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d1420326-5df2-4716-989e-7957336b4928-metrics-certs\") pod \"maas-api-e2e-worker-single-main-07ce81-79f4465d54-bnchr\" (UID: \"d1420326-5df2-4716-989e-7957336b4928\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-07ce81-79f4465d54-bnchr" Nov 04 03:26:06.883043 ip-10-0-134-125 kubenswrapper[4405]: E1104 03:26:06.883000 4405 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5f2vb" containerName="extractor" Nov 04 03:26:06.888260 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:26:06.888228 4405 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9cpcd\" (UniqueName: \"kubernetes.io/projected/d1420326-5df2-4716-989e-7957336b4928-kube-api-access-9cpcd\") pod \"maas-api-e2e-worker-single-main-07ce81-79f4465d54-bnchr\" (UID: \"d1420326-5df2-4716-989e-7957336b4928\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-07ce81-79f4465d54-bnchr" Nov 04 03:26:07.329121 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:26:07.329085 4405 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d1420326-5df2-4716-989e-7957336b4928-maas-api-tls\") pod \"maas-api-e2e-worker-single-main-07ce81-79f4465d54-bnchr\" (UID: \"d1420326-5df2-4716-989e-7957336b4928\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-07ce81-79f4465d54-bnchr" Nov 04 03:26:07.331647 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:26:07.331623 4405 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d1420326-5df2-4716-989e-7957336b4928-maas-api-tls\") pod \"maas-api-e2e-worker-single-main-07ce81-79f4465d54-bnchr\" (UID: \"d1420326-5df2-4716-989e-7957336b4928\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-07ce81-79f4465d54-bnchr" Nov 04 03:26:07.566219 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:26:07.566181 4405 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-07ce81-79f4465d54-bnchr" Nov 04 03:26:07.946835 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:26:07.946801 4405 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-07ce81-79f4465d54-bnchr"] Nov 04 03:26:07.948218 ip-10-0-134-125 kubenswrapper[4405]: W1104 03:26:07.948185 4405 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd1420326_5df2_4716_989e_7957336b4928.slice/crio-3214346a68408673886bc98d870be0708423343f50d147ad987ab3789d923322 WatchSource:0}: Error finding container 3214346a68408673886bc98d870be0708423343f50d147ad987ab3789d923322: Status 404 returned error can't find the container with id 3214346a68408673886bc98d870be0708423343f50d147ad987ab3789d923322 Nov 04 03:26:07.949709 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:26:07.949661 4405 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 04 03:26:08.029450 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:26:08.029407 4405 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-bl5mg"] Nov 04 03:26:08.362484 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:26:08.362449 4405 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-07ce81-79f4465d54-bnchr" event={"ID":"d1420326-5df2-4716-989e-7957336b4928","Type":"ContainerStarted","Data":"3214346a68408673886bc98d870be0708423343f50d147ad987ab3789d923322"} Nov 04 03:26:09.369053 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:26:09.369013 4405 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-07ce81-79f4465d54-bnchr" event={"ID":"d1420326-5df2-4716-989e-7957336b4928","Type":"ContainerStarted","Data":"383c07a0377e42b9ebb87f83efac30b4dbcd373a624f246c234b2b9365cc9bd7"} Nov 04 03:26:09.369561 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:26:09.369074 4405 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-07ce81-79f4465d54-bnchr" Nov 04 03:26:09.488274 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:26:09.488210 4405 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-07ce81-79f4465d54-bnchr" podStartSLOduration=2.578131108 podStartE2EDuration="3.488194858s" totalImagesPullingTime="910.06375ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 03:26:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 03:26:09.486894839 +0000 UTC m=+3215.917594957" watchObservedRunningTime="2026-11-04 03:26:09.488194858 +0000 UTC m=+3215.918894981" Nov 04 03:26:15.383469 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:26:15.383441 4405 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-07ce81-79f4465d54-bnchr" Nov 04 03:26:16.354469 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:26:16.354423 4405 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-07ce81-9ddd44mgvn" Nov 04 03:26:16.354656 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:26:16.354608 4405 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-single-main-07ce81-767654f68x8k9j" Nov 04 03:26:17.048935 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:26:17.048897 4405 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-jk2tn"] Nov 04 03:26:17.051133 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:26:17.051111 4405 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-jk2tn"] Nov 04 03:26:17.723038 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:26:17.722993 4405 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ba6e018c-ab91-4734-8b38-572dd172093d" path="/var/lib/kubelet/pods/ba6e018c-ab91-4734-8b38-572dd172093d/volumes" Nov 04 03:26:21.031365 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:26:21.031282 4405 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-bl5mg"] Nov 04 03:26:24.471322 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:26:24.471282 4405 generic.go:356] "Generic (PLEG): container finished" podID="5111ad38-b833-4be2-9a7c-e7db5c2bd133" containerID="38fb836dba0859accccb17373b4887fbef0fcc10896302cc8df2067efa08b6d2" exitCode=0 Nov 04 03:26:24.471884 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:26:24.471354 4405 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-07ce81-openshift-default-5446f4ff6fwvjql" event={"ID":"5111ad38-b833-4be2-9a7c-e7db5c2bd133","Type":"ContainerDied","Data":"38fb836dba0859accccb17373b4887fbef0fcc10896302cc8df2067efa08b6d2"} Nov 04 03:26:24.471884 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:26:24.471796 4405 scope.go:122] "RemoveContainer" containerID="38fb836dba0859accccb17373b4887fbef0fcc10896302cc8df2067efa08b6d2" Nov 04 03:26:24.473067 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:26:24.473037 4405 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 04 03:26:24.473171 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:26:24.473110 4405 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 04 03:26:24.473171 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:26:24.473141 4405 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 04 03:26:25.478262 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:26:25.478222 4405 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-07ce81-openshift-default-5446f4ff6fwvjql" event={"ID":"5111ad38-b833-4be2-9a7c-e7db5c2bd133","Type":"ContainerStarted","Data":"1f34c83c800a216334ce52533b55edf58aeb97d9460da99b5e01caf7b3da6aa5"} Nov 04 03:26:26.146754 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:26:26.146719 4405 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-single-main-07ce81-openshift-default-5446f4ff6fwvjql" Nov 04 03:26:26.147923 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:26:26.147894 4405 patch_prober.go:29] interesting pod/e2e-worker-single-main-07ce81-openshift-default-5446f4ff6fwvjql container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.154:15021/healthz/ready\": dial tcp 10.134.0.154:15021: connect: connection refused" start-of-body= Nov 04 03:26:26.148074 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:26:26.147956 4405 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-07ce81-openshift-default-5446f4ff6fwvjql" podUID="5111ad38-b833-4be2-9a7c-e7db5c2bd133" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.154:15021/healthz/ready\": dial tcp 10.134.0.154:15021: connect: connection refused" Nov 04 03:26:27.146516 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:26:27.146475 4405 patch_prober.go:29] interesting pod/e2e-worker-single-main-07ce81-openshift-default-5446f4ff6fwvjql container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.154:15021/healthz/ready\": dial tcp 10.134.0.154:15021: connect: connection refused" start-of-body= Nov 04 03:26:27.147031 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:26:27.146558 4405 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-07ce81-openshift-default-5446f4ff6fwvjql" podUID="5111ad38-b833-4be2-9a7c-e7db5c2bd133" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.154:15021/healthz/ready\": dial tcp 10.134.0.154:15021: connect: connection refused" Nov 04 03:26:28.146486 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:26:28.146453 4405 patch_prober.go:29] interesting pod/e2e-worker-single-main-07ce81-openshift-default-5446f4ff6fwvjql container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.154:15021/healthz/ready\": dial tcp 10.134.0.154:15021: connect: connection refused" start-of-body= Nov 04 03:26:28.146622 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:26:28.146518 4405 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-07ce81-openshift-default-5446f4ff6fwvjql" podUID="5111ad38-b833-4be2-9a7c-e7db5c2bd133" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.154:15021/healthz/ready\": dial tcp 10.134.0.154:15021: connect: connection refused" Nov 04 03:26:28.305827 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:26:28.305779 4405 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-single-main-07ce81-openshift-default-5446f4ff6fwvjql" Nov 04 03:26:29.059646 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:26:29.059609 4405 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-v4d8q"] Nov 04 03:26:29.117627 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:26:29.117590 4405 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-v4d8q"] Nov 04 03:26:29.147205 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:26:29.147168 4405 patch_prober.go:29] interesting pod/e2e-worker-single-main-07ce81-openshift-default-5446f4ff6fwvjql container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.154:15021/healthz/ready\": dial tcp 10.134.0.154:15021: connect: connection refused" start-of-body= Nov 04 03:26:29.147580 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:26:29.147218 4405 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-07ce81-openshift-default-5446f4ff6fwvjql" podUID="5111ad38-b833-4be2-9a7c-e7db5c2bd133" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.154:15021/healthz/ready\": dial tcp 10.134.0.154:15021: connect: connection refused" Nov 04 03:26:29.720459 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:26:29.720415 4405 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="54df9890-5e4d-44a6-8cf4-329c7544c2e2" path="/var/lib/kubelet/pods/54df9890-5e4d-44a6-8cf4-329c7544c2e2/volumes" Nov 04 03:26:30.147039 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:26:30.147006 4405 patch_prober.go:29] interesting pod/e2e-worker-single-main-07ce81-openshift-default-5446f4ff6fwvjql container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.154:15021/healthz/ready\": dial tcp 10.134.0.154:15021: connect: connection refused" start-of-body= Nov 04 03:26:30.147240 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:26:30.147079 4405 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-07ce81-openshift-default-5446f4ff6fwvjql" podUID="5111ad38-b833-4be2-9a7c-e7db5c2bd133" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.154:15021/healthz/ready\": dial tcp 10.134.0.154:15021: connect: connection refused" Nov 04 03:26:31.151490 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:26:31.151459 4405 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-single-main-07ce81-openshift-default-5446f4ff6fwvjql" Nov 04 03:26:31.152418 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:26:31.152400 4405 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-single-main-07ce81-openshift-default-5446f4ff6fwvjql" Nov 04 03:26:34.224941 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:26:34.224901 4405 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-bl5mg"] Nov 04 03:26:35.285619 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:26:35.285584 4405 scope.go:122] "RemoveContainer" containerID="7ca16528c075b47dc5cabc4e420d8d8867c1312606b728cc04be8a3f819d3eb5" Nov 04 03:26:36.883589 ip-10-0-134-125 kubenswrapper[4405]: E1104 03:26:36.883550 4405 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5f2vb" containerName="extractor" Nov 04 03:26:59.044009 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:26:59.043974 4405 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-bl5mg"] Nov 04 03:27:06.048059 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:27:06.048022 4405 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-g9srh"] Nov 04 03:27:06.069265 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:27:06.069235 4405 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-g9srh"] Nov 04 03:27:06.884737 ip-10-0-134-125 kubenswrapper[4405]: E1104 03:27:06.884700 4405 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5f2vb" containerName="extractor" Nov 04 03:27:07.702257 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:27:07.702217 4405 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="14d1b082-b0f7-4795-ab6e-cfc82d67ae8f" path="/var/lib/kubelet/pods/14d1b082-b0f7-4795-ab6e-cfc82d67ae8f/volumes" Nov 04 03:27:16.048440 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:27:16.048355 4405 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-jdmvl"] Nov 04 03:27:16.098150 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:27:16.098120 4405 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-jdmvl"] Nov 04 03:27:17.700222 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:27:17.700188 4405 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2108b065-456e-48e6-9bc9-78dc33327f20" path="/var/lib/kubelet/pods/2108b065-456e-48e6-9bc9-78dc33327f20/volumes" Nov 04 03:27:28.122154 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:27:28.122121 4405 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-bl5mg"] Nov 04 03:27:31.743272 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:27:31.743238 4405 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-07ce81-767654f68x8k9j"] Nov 04 03:27:31.855411 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:27:31.855372 4405 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-07ce81-9ddd44mgvn"] Nov 04 03:27:34.215557 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:27:34.215516 4405 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-vwcx8_ffbb2d83-5a93-4a5b-bce3-e40afb8f58ea/ovn-acl-logging/0.log" Nov 04 03:27:34.231764 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:27:34.231731 4405 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-vwcx8_ffbb2d83-5a93-4a5b-bce3-e40afb8f58ea/ovn-acl-logging/0.log" Nov 04 03:27:34.845385 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:27:34.845350 4405 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-07ce81-79f4465d54-bnchr"] Nov 04 03:27:34.845576 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:27:34.845550 4405 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-07ce81-79f4465d54-bnchr" podUID="d1420326-5df2-4716-989e-7957336b4928" containerName="maas-api" containerID="cri-o://383c07a0377e42b9ebb87f83efac30b4dbcd373a624f246c234b2b9365cc9bd7" gracePeriod=30 Nov 04 03:27:35.309372 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:27:35.309289 4405 scope.go:122] "RemoveContainer" containerID="7cf5c464c48a4969e75bbe372e5f8e38658be9036ef98cdf56718330104d2c54" Nov 04 03:27:35.318818 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:27:35.318727 4405 scope.go:122] "RemoveContainer" containerID="0ec0750c29bad6a50f632b25f0556a599197fb8ece20f2dfe21f1faf48d96358" Nov 04 03:27:35.328837 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:27:35.328810 4405 scope.go:122] "RemoveContainer" containerID="d85bf0a3fb627a23278d8c0be24df4f7f8d39df6b5e89f6c6b588148a5367b81" Nov 04 03:27:35.378278 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:27:35.378239 4405 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-07ce81-79f4465d54-bnchr" podUID="d1420326-5df2-4716-989e-7957336b4928" containerName="maas-api" probeResult="failure" output="Get \"https://10.134.0.157:8443/health\": dial tcp 10.134.0.157:8443: connect: connection refused" Nov 04 03:27:36.106417 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:27:36.106393 4405 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-07ce81-79f4465d54-bnchr" Nov 04 03:27:36.233202 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:27:36.233112 4405 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d1420326-5df2-4716-989e-7957336b4928-kube-api-access-9cpcd\" (UniqueName: \"kubernetes.io/projected/d1420326-5df2-4716-989e-7957336b4928-kube-api-access-9cpcd\") pod \"d1420326-5df2-4716-989e-7957336b4928\" (UID: \"d1420326-5df2-4716-989e-7957336b4928\") " Nov 04 03:27:36.233394 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:27:36.233220 4405 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d1420326-5df2-4716-989e-7957336b4928-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d1420326-5df2-4716-989e-7957336b4928-maas-api-tls\") pod \"d1420326-5df2-4716-989e-7957336b4928\" (UID: \"d1420326-5df2-4716-989e-7957336b4928\") " Nov 04 03:27:36.233394 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:27:36.233313 4405 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d1420326-5df2-4716-989e-7957336b4928-metrics-certs\" (UniqueName: \"kubernetes.io/secret/d1420326-5df2-4716-989e-7957336b4928-metrics-certs\") pod \"d1420326-5df2-4716-989e-7957336b4928\" (UID: \"d1420326-5df2-4716-989e-7957336b4928\") " Nov 04 03:27:36.233394 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:27:36.233349 4405 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d1420326-5df2-4716-989e-7957336b4928-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d1420326-5df2-4716-989e-7957336b4928-openshift-service-ca\") pod \"d1420326-5df2-4716-989e-7957336b4928\" (UID: \"d1420326-5df2-4716-989e-7957336b4928\") " Nov 04 03:27:36.233810 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:27:36.233782 4405 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d1420326-5df2-4716-989e-7957336b4928-openshift-service-ca" pod "d1420326-5df2-4716-989e-7957336b4928" (UID: "d1420326-5df2-4716-989e-7957336b4928"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 03:27:36.235614 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:27:36.235578 4405 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d1420326-5df2-4716-989e-7957336b4928-metrics-certs" pod "d1420326-5df2-4716-989e-7957336b4928" (UID: "d1420326-5df2-4716-989e-7957336b4928"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 04 03:27:36.235854 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:27:36.235834 4405 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d1420326-5df2-4716-989e-7957336b4928-kube-api-access-9cpcd" pod "d1420326-5df2-4716-989e-7957336b4928" (UID: "d1420326-5df2-4716-989e-7957336b4928"). InnerVolumeSpecName "kube-api-access-9cpcd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 03:27:36.235949 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:27:36.235929 4405 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d1420326-5df2-4716-989e-7957336b4928-maas-api-tls" pod "d1420326-5df2-4716-989e-7957336b4928" (UID: "d1420326-5df2-4716-989e-7957336b4928"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 04 03:27:36.334455 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:27:36.334421 4405 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d1420326-5df2-4716-989e-7957336b4928-metrics-certs\") on node \"ip-10-0-134-125.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:27:36.334455 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:27:36.334450 4405 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d1420326-5df2-4716-989e-7957336b4928-openshift-service-ca\") on node \"ip-10-0-134-125.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:27:36.334455 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:27:36.334461 4405 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9cpcd\" (UniqueName: \"kubernetes.io/projected/d1420326-5df2-4716-989e-7957336b4928-kube-api-access-9cpcd\") on node \"ip-10-0-134-125.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:27:36.335085 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:27:36.334471 4405 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d1420326-5df2-4716-989e-7957336b4928-maas-api-tls\") on node \"ip-10-0-134-125.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:27:36.743587 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:27:36.743527 4405 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-single-main-07ce81-767654f68x8k9j" podUID="e398089d-e56a-418b-a820-ca361d65e178" containerName="payload-processing" containerID="cri-o://076d7b5eb3cecc8cc14986c1ac08ebe2f7f6e7f537d598127f80887d0281a9dc" gracePeriod=25 Nov 04 03:27:36.856769 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:27:36.856722 4405 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-07ce81-9ddd44mgvn" podUID="56fc8374-482d-4e8f-8cf9-be24fda3235a" containerName="payload-pre-processing" containerID="cri-o://6fbbd66c6debcc4ba502a374d108459494b732ee108e7414ea75f8ad1c6fa935" gracePeriod=25 Nov 04 03:27:36.882841 ip-10-0-134-125 kubenswrapper[4405]: E1104 03:27:36.882805 4405 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5f2vb" containerName="extractor" Nov 04 03:27:36.887930 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:27:36.887903 4405 generic.go:356] "Generic (PLEG): container finished" podID="d1420326-5df2-4716-989e-7957336b4928" containerID="383c07a0377e42b9ebb87f83efac30b4dbcd373a624f246c234b2b9365cc9bd7" exitCode=0 Nov 04 03:27:36.888091 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:27:36.887954 4405 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-07ce81-79f4465d54-bnchr" Nov 04 03:27:36.888091 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:27:36.887981 4405 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-07ce81-79f4465d54-bnchr" event={"ID":"d1420326-5df2-4716-989e-7957336b4928","Type":"ContainerDied","Data":"383c07a0377e42b9ebb87f83efac30b4dbcd373a624f246c234b2b9365cc9bd7"} Nov 04 03:27:36.888091 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:27:36.888045 4405 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-07ce81-79f4465d54-bnchr" event={"ID":"d1420326-5df2-4716-989e-7957336b4928","Type":"ContainerDied","Data":"3214346a68408673886bc98d870be0708423343f50d147ad987ab3789d923322"} Nov 04 03:27:36.888091 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:27:36.888076 4405 scope.go:122] "RemoveContainer" containerID="383c07a0377e42b9ebb87f83efac30b4dbcd373a624f246c234b2b9365cc9bd7" Nov 04 03:27:36.898706 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:27:36.898686 4405 scope.go:122] "RemoveContainer" containerID="383c07a0377e42b9ebb87f83efac30b4dbcd373a624f246c234b2b9365cc9bd7" Nov 04 03:27:36.898988 ip-10-0-134-125 kubenswrapper[4405]: E1104 03:27:36.898971 4405 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"383c07a0377e42b9ebb87f83efac30b4dbcd373a624f246c234b2b9365cc9bd7\": container with ID starting with 383c07a0377e42b9ebb87f83efac30b4dbcd373a624f246c234b2b9365cc9bd7 not found: ID does not exist" containerID="383c07a0377e42b9ebb87f83efac30b4dbcd373a624f246c234b2b9365cc9bd7" Nov 04 03:27:36.899034 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:27:36.898998 4405 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"383c07a0377e42b9ebb87f83efac30b4dbcd373a624f246c234b2b9365cc9bd7"} err="failed to get container status \"383c07a0377e42b9ebb87f83efac30b4dbcd373a624f246c234b2b9365cc9bd7\": rpc error: code = NotFound desc = could not find container \"383c07a0377e42b9ebb87f83efac30b4dbcd373a624f246c234b2b9365cc9bd7\": container with ID starting with 383c07a0377e42b9ebb87f83efac30b4dbcd373a624f246c234b2b9365cc9bd7 not found: ID does not exist" Nov 04 03:27:37.052239 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:27:37.052159 4405 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-07ce81-79f4465d54-bnchr"] Nov 04 03:27:37.053874 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:27:37.053852 4405 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-07ce81-79f4465d54-bnchr"] Nov 04 03:27:37.699928 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:27:37.699892 4405 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d1420326-5df2-4716-989e-7957336b4928" path="/var/lib/kubelet/pods/d1420326-5df2-4716-989e-7957336b4928/volumes" Nov 04 03:27:41.351117 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:27:41.351083 4405 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-bl5mg"] Nov 04 03:27:46.352818 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:27:46.352774 4405 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-single-main-07ce81-767654f68x8k9j" podUID="e398089d-e56a-418b-a820-ca361d65e178" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 04 03:27:46.352818 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:27:46.352791 4405 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-07ce81-9ddd44mgvn" podUID="56fc8374-482d-4e8f-8cf9-be24fda3235a" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 04 03:27:51.048282 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:27:51.048249 4405 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-8stcp"] Nov 04 03:27:51.097771 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:27:51.097735 4405 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-8stcp"] Nov 04 03:27:51.700418 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:27:51.700384 4405 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e182532f-6946-4215-a1f6-9015142edcbf" path="/var/lib/kubelet/pods/e182532f-6946-4215-a1f6-9015142edcbf/volumes" Nov 04 03:27:56.352644 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:27:56.352598 4405 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-single-main-07ce81-767654f68x8k9j" podUID="e398089d-e56a-418b-a820-ca361d65e178" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 04 03:27:56.352644 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:27:56.352632 4405 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-07ce81-9ddd44mgvn" podUID="56fc8374-482d-4e8f-8cf9-be24fda3235a" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 04 03:27:56.993047 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:27:56.993009 4405 generic.go:356] "Generic (PLEG): container finished" podID="e398089d-e56a-418b-a820-ca361d65e178" containerID="076d7b5eb3cecc8cc14986c1ac08ebe2f7f6e7f537d598127f80887d0281a9dc" exitCode=0 Nov 04 03:27:56.993215 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:27:56.993048 4405 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-07ce81-767654f68x8k9j" event={"ID":"e398089d-e56a-418b-a820-ca361d65e178","Type":"ContainerDied","Data":"076d7b5eb3cecc8cc14986c1ac08ebe2f7f6e7f537d598127f80887d0281a9dc"} Nov 04 03:27:56.993215 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:27:56.993145 4405 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-07ce81-767654f68x8k9j" event={"ID":"e398089d-e56a-418b-a820-ca361d65e178","Type":"ContainerDied","Data":"9859eb6ea91a2e90e42abde64595f6da0a526e48f1d5113b22c04b7d28756e18"} Nov 04 03:27:56.993215 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:27:56.993175 4405 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9859eb6ea91a2e90e42abde64595f6da0a526e48f1d5113b22c04b7d28756e18" Nov 04 03:27:56.995058 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:27:56.995026 4405 generic.go:356] "Generic (PLEG): container finished" podID="56fc8374-482d-4e8f-8cf9-be24fda3235a" containerID="6fbbd66c6debcc4ba502a374d108459494b732ee108e7414ea75f8ad1c6fa935" exitCode=0 Nov 04 03:27:56.995187 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:27:56.995089 4405 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-07ce81-9ddd44mgvn" event={"ID":"56fc8374-482d-4e8f-8cf9-be24fda3235a","Type":"ContainerDied","Data":"6fbbd66c6debcc4ba502a374d108459494b732ee108e7414ea75f8ad1c6fa935"} Nov 04 03:27:57.001511 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:27:57.001464 4405 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-07ce81-767654f68x8k9j" Nov 04 03:27:57.040520 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:27:57.040491 4405 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e398089d-e56a-418b-a820-ca361d65e178-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e398089d-e56a-418b-a820-ca361d65e178-plugins-config-volume\") pod \"e398089d-e56a-418b-a820-ca361d65e178\" (UID: \"e398089d-e56a-418b-a820-ca361d65e178\") " Nov 04 03:27:57.040747 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:27:57.040643 4405 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e398089d-e56a-418b-a820-ca361d65e178-kube-api-access-ln7xk\" (UniqueName: \"kubernetes.io/projected/e398089d-e56a-418b-a820-ca361d65e178-kube-api-access-ln7xk\") pod \"e398089d-e56a-418b-a820-ca361d65e178\" (UID: \"e398089d-e56a-418b-a820-ca361d65e178\") " Nov 04 03:27:57.040901 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:27:57.040878 4405 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e398089d-e56a-418b-a820-ca361d65e178-plugins-config-volume" pod "e398089d-e56a-418b-a820-ca361d65e178" (UID: "e398089d-e56a-418b-a820-ca361d65e178"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 03:27:57.043327 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:27:57.043290 4405 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e398089d-e56a-418b-a820-ca361d65e178-kube-api-access-ln7xk" pod "e398089d-e56a-418b-a820-ca361d65e178" (UID: "e398089d-e56a-418b-a820-ca361d65e178"). InnerVolumeSpecName "kube-api-access-ln7xk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 03:27:57.108515 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:27:57.108486 4405 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-07ce81-9ddd44mgvn" Nov 04 03:27:57.142094 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:27:57.142062 4405 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e398089d-e56a-418b-a820-ca361d65e178-plugins-config-volume\") on node \"ip-10-0-134-125.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:27:57.142094 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:27:57.142088 4405 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ln7xk\" (UniqueName: \"kubernetes.io/projected/e398089d-e56a-418b-a820-ca361d65e178-kube-api-access-ln7xk\") on node \"ip-10-0-134-125.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:27:57.243222 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:27:57.243188 4405 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/56fc8374-482d-4e8f-8cf9-be24fda3235a-kube-api-access-brv6b\" (UniqueName: \"kubernetes.io/projected/56fc8374-482d-4e8f-8cf9-be24fda3235a-kube-api-access-brv6b\") pod \"56fc8374-482d-4e8f-8cf9-be24fda3235a\" (UID: \"56fc8374-482d-4e8f-8cf9-be24fda3235a\") " Nov 04 03:27:57.243381 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:27:57.243257 4405 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/56fc8374-482d-4e8f-8cf9-be24fda3235a-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/56fc8374-482d-4e8f-8cf9-be24fda3235a-plugins-config-volume\") pod \"56fc8374-482d-4e8f-8cf9-be24fda3235a\" (UID: \"56fc8374-482d-4e8f-8cf9-be24fda3235a\") " Nov 04 03:27:57.243658 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:27:57.243637 4405 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/56fc8374-482d-4e8f-8cf9-be24fda3235a-plugins-config-volume" pod "56fc8374-482d-4e8f-8cf9-be24fda3235a" (UID: "56fc8374-482d-4e8f-8cf9-be24fda3235a"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 03:27:57.245413 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:27:57.245394 4405 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/56fc8374-482d-4e8f-8cf9-be24fda3235a-kube-api-access-brv6b" pod "56fc8374-482d-4e8f-8cf9-be24fda3235a" (UID: "56fc8374-482d-4e8f-8cf9-be24fda3235a"). InnerVolumeSpecName "kube-api-access-brv6b". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 03:27:57.344426 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:27:57.344398 4405 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-brv6b\" (UniqueName: \"kubernetes.io/projected/56fc8374-482d-4e8f-8cf9-be24fda3235a-kube-api-access-brv6b\") on node \"ip-10-0-134-125.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:27:57.344426 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:27:57.344427 4405 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/56fc8374-482d-4e8f-8cf9-be24fda3235a-plugins-config-volume\") on node \"ip-10-0-134-125.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:27:58.002158 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:27:58.002080 4405 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-07ce81-9ddd44mgvn" Nov 04 03:27:58.002158 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:27:58.002092 4405 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-07ce81-9ddd44mgvn" event={"ID":"56fc8374-482d-4e8f-8cf9-be24fda3235a","Type":"ContainerDied","Data":"84efe3270404402ff9ff22222d8875ea28bee4180afacdb9cff9901f054f12ed"} Nov 04 03:27:58.002158 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:27:58.002078 4405 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-07ce81-767654f68x8k9j" Nov 04 03:27:58.002701 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:27:58.002149 4405 scope.go:122] "RemoveContainer" containerID="6fbbd66c6debcc4ba502a374d108459494b732ee108e7414ea75f8ad1c6fa935" Nov 04 03:27:58.165152 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:27:58.165116 4405 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-07ce81-9ddd44mgvn"] Nov 04 03:27:58.214625 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:27:58.214589 4405 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-07ce81-9ddd44mgvn"] Nov 04 03:27:58.277696 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:27:58.277593 4405 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-07ce81-767654f68x8k9j"] Nov 04 03:27:58.281709 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:27:58.281684 4405 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-07ce81-767654f68x8k9j"] Nov 04 03:27:59.700234 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:27:59.700201 4405 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="56fc8374-482d-4e8f-8cf9-be24fda3235a" path="/var/lib/kubelet/pods/56fc8374-482d-4e8f-8cf9-be24fda3235a/volumes" Nov 04 03:27:59.700624 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:27:59.700563 4405 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e398089d-e56a-418b-a820-ca361d65e178" path="/var/lib/kubelet/pods/e398089d-e56a-418b-a820-ca361d65e178/volumes" Nov 04 03:28:06.882893 ip-10-0-134-125 kubenswrapper[4405]: E1104 03:28:06.882847 4405 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5f2vb" containerName="extractor" Nov 04 03:28:09.681505 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:28:09.681471 4405 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-single-main-07ce81-openshift-default-5446f4ff6fwvjql"] Nov 04 03:28:09.681959 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:28:09.681700 4405 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-single-main-07ce81-openshift-default-5446f4ff6fwvjql" podUID="5111ad38-b833-4be2-9a7c-e7db5c2bd133" containerName="istio-proxy" containerID="cri-o://1f34c83c800a216334ce52533b55edf58aeb97d9460da99b5e01caf7b3da6aa5" gracePeriod=30 Nov 04 03:28:11.226336 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:28:11.226295 4405 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-lw772"] Nov 04 03:28:11.276132 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:28:11.276097 4405 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-lw772"] Nov 04 03:28:11.351049 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:28:11.351016 4405 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-9ktbr"] Nov 04 03:28:11.351424 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:28:11.351411 4405 cpu_manager.go:414] "containerMap: removing container" podUID="e398089d-e56a-418b-a820-ca361d65e178" containerName="payload-processing" Nov 04 03:28:11.351475 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:28:11.351426 4405 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e398089d-e56a-418b-a820-ca361d65e178" containerName="payload-processing" Nov 04 03:28:11.351475 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:28:11.351439 4405 cpu_manager.go:414] "containerMap: removing container" podUID="56fc8374-482d-4e8f-8cf9-be24fda3235a" containerName="payload-pre-processing" Nov 04 03:28:11.351475 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:28:11.351444 4405 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="56fc8374-482d-4e8f-8cf9-be24fda3235a" containerName="payload-pre-processing" Nov 04 03:28:11.351475 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:28:11.351458 4405 cpu_manager.go:414] "containerMap: removing container" podUID="d1420326-5df2-4716-989e-7957336b4928" containerName="maas-api" Nov 04 03:28:11.351475 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:28:11.351464 4405 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d1420326-5df2-4716-989e-7957336b4928" containerName="maas-api" Nov 04 03:28:11.351629 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:28:11.351524 4405 memory_manager.go:365] "RemoveStaleState removing state" podUID="e398089d-e56a-418b-a820-ca361d65e178" containerName="payload-processing" Nov 04 03:28:11.351629 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:28:11.351534 4405 memory_manager.go:365] "RemoveStaleState removing state" podUID="56fc8374-482d-4e8f-8cf9-be24fda3235a" containerName="payload-pre-processing" Nov 04 03:28:11.351629 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:28:11.351540 4405 memory_manager.go:365] "RemoveStaleState removing state" podUID="d1420326-5df2-4716-989e-7957336b4928" containerName="maas-api" Nov 04 03:28:11.354607 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:28:11.354588 4405 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-9ktbr" Nov 04 03:28:11.409730 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:28:11.409697 4405 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 04 03:28:11.410011 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:28:11.409985 4405 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-slbq8\"" Nov 04 03:28:11.412815 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:28:11.412788 4405 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-9ktbr"] Nov 04 03:28:11.419482 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:28:11.419457 4405 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 04 03:28:11.456208 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:28:11.456177 4405 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 04 03:28:11.478089 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:28:11.478006 4405 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-khmmr\" (UniqueName: \"kubernetes.io/projected/5f397f15-3135-4cee-9d3a-eb62a71e6d9e-kube-api-access-khmmr\") pod \"maas-api-revoke-sub-3d4156f1-9ktbr\" (UID: \"5f397f15-3135-4cee-9d3a-eb62a71e6d9e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-9ktbr" Nov 04 03:28:11.478244 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:28:11.478123 4405 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5f397f15-3135-4cee-9d3a-eb62a71e6d9e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-9ktbr\" (UID: \"5f397f15-3135-4cee-9d3a-eb62a71e6d9e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-9ktbr" Nov 04 03:28:11.478244 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:28:11.478163 4405 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5f397f15-3135-4cee-9d3a-eb62a71e6d9e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-9ktbr\" (UID: \"5f397f15-3135-4cee-9d3a-eb62a71e6d9e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-9ktbr" Nov 04 03:28:11.579480 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:28:11.579438 4405 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5f397f15-3135-4cee-9d3a-eb62a71e6d9e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-9ktbr\" (UID: \"5f397f15-3135-4cee-9d3a-eb62a71e6d9e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-9ktbr" Nov 04 03:28:11.579480 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:28:11.579485 4405 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5f397f15-3135-4cee-9d3a-eb62a71e6d9e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-9ktbr\" (UID: \"5f397f15-3135-4cee-9d3a-eb62a71e6d9e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-9ktbr" Nov 04 03:28:11.579743 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:28:11.579542 4405 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-khmmr\" (UniqueName: \"kubernetes.io/projected/5f397f15-3135-4cee-9d3a-eb62a71e6d9e-kube-api-access-khmmr\") pod \"maas-api-revoke-sub-3d4156f1-9ktbr\" (UID: \"5f397f15-3135-4cee-9d3a-eb62a71e6d9e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-9ktbr" Nov 04 03:28:11.580198 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:28:11.580173 4405 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5f397f15-3135-4cee-9d3a-eb62a71e6d9e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-9ktbr\" (UID: \"5f397f15-3135-4cee-9d3a-eb62a71e6d9e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-9ktbr" Nov 04 03:28:11.582015 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:28:11.581988 4405 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5f397f15-3135-4cee-9d3a-eb62a71e6d9e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-9ktbr\" (UID: \"5f397f15-3135-4cee-9d3a-eb62a71e6d9e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-9ktbr" Nov 04 03:28:11.640452 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:28:11.640413 4405 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-khmmr\" (UniqueName: \"kubernetes.io/projected/5f397f15-3135-4cee-9d3a-eb62a71e6d9e-kube-api-access-khmmr\") pod \"maas-api-revoke-sub-3d4156f1-9ktbr\" (UID: \"5f397f15-3135-4cee-9d3a-eb62a71e6d9e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-9ktbr" Nov 04 03:28:11.668045 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:28:11.667999 4405 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-9ktbr" Nov 04 03:28:11.700257 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:28:11.700220 4405 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="98ac0ac8-41cc-4be0-80b7-c9b7515aefce" path="/var/lib/kubelet/pods/98ac0ac8-41cc-4be0-80b7-c9b7515aefce/volumes" Nov 04 03:28:12.047443 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:28:12.047337 4405 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-9ktbr"] Nov 04 03:28:12.049937 ip-10-0-134-125 kubenswrapper[4405]: W1104 03:28:12.049905 4405 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5f397f15_3135_4cee_9d3a_eb62a71e6d9e.slice/crio-ee12f32ad320f39559ee26d4847b5f459a15b772cdded9958d9f5a917f757860 WatchSource:0}: Error finding container ee12f32ad320f39559ee26d4847b5f459a15b772cdded9958d9f5a917f757860: Status 404 returned error can't find the container with id ee12f32ad320f39559ee26d4847b5f459a15b772cdded9958d9f5a917f757860 Nov 04 03:28:12.077821 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:28:12.077792 4405 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-9ktbr" event={"ID":"5f397f15-3135-4cee-9d3a-eb62a71e6d9e","Type":"ContainerStarted","Data":"ee12f32ad320f39559ee26d4847b5f459a15b772cdded9958d9f5a917f757860"} Nov 04 03:28:13.046856 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:28:13.046788 4405 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-vnrfp"] Nov 04 03:28:13.084392 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:28:13.084354 4405 generic.go:356] "Generic (PLEG): container finished" podID="5f397f15-3135-4cee-9d3a-eb62a71e6d9e" containerID="772f9d5ae2d1e5692abc588d7d356f8e78c6e802bbe42f5ecf353af71a133c60" exitCode=0 Nov 04 03:28:13.084584 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:28:13.084434 4405 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-9ktbr" event={"ID":"5f397f15-3135-4cee-9d3a-eb62a71e6d9e","Type":"ContainerDied","Data":"772f9d5ae2d1e5692abc588d7d356f8e78c6e802bbe42f5ecf353af71a133c60"} Nov 04 03:28:13.096804 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:28:13.096774 4405 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-vnrfp"] Nov 04 03:28:13.700313 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:28:13.700270 4405 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ca88b883-b203-4513-b004-be51b52827f3" path="/var/lib/kubelet/pods/ca88b883-b203-4513-b004-be51b52827f3/volumes" Nov 04 03:28:14.227062 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:28:14.227037 4405 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-9ktbr" Nov 04 03:28:14.303289 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:28:14.303253 4405 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5f397f15-3135-4cee-9d3a-eb62a71e6d9e-kube-api-access-khmmr\" (UniqueName: \"kubernetes.io/projected/5f397f15-3135-4cee-9d3a-eb62a71e6d9e-kube-api-access-khmmr\") pod \"5f397f15-3135-4cee-9d3a-eb62a71e6d9e\" (UID: \"5f397f15-3135-4cee-9d3a-eb62a71e6d9e\") " Nov 04 03:28:14.303475 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:28:14.303312 4405 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5f397f15-3135-4cee-9d3a-eb62a71e6d9e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5f397f15-3135-4cee-9d3a-eb62a71e6d9e-maas-api-openshift-ca\") pod \"5f397f15-3135-4cee-9d3a-eb62a71e6d9e\" (UID: \"5f397f15-3135-4cee-9d3a-eb62a71e6d9e\") " Nov 04 03:28:14.303475 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:28:14.303351 4405 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5f397f15-3135-4cee-9d3a-eb62a71e6d9e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5f397f15-3135-4cee-9d3a-eb62a71e6d9e-maas-api-xks-ca\") pod \"5f397f15-3135-4cee-9d3a-eb62a71e6d9e\" (UID: \"5f397f15-3135-4cee-9d3a-eb62a71e6d9e\") " Nov 04 03:28:14.303655 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:28:14.303633 4405 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5f397f15-3135-4cee-9d3a-eb62a71e6d9e-maas-api-openshift-ca" pod "5f397f15-3135-4cee-9d3a-eb62a71e6d9e" (UID: "5f397f15-3135-4cee-9d3a-eb62a71e6d9e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 03:28:14.303787 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:28:14.303761 4405 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5f397f15-3135-4cee-9d3a-eb62a71e6d9e-maas-api-openshift-ca\") on node \"ip-10-0-134-125.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:28:14.305693 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:28:14.305640 4405 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5f397f15-3135-4cee-9d3a-eb62a71e6d9e-maas-api-xks-ca" pod "5f397f15-3135-4cee-9d3a-eb62a71e6d9e" (UID: "5f397f15-3135-4cee-9d3a-eb62a71e6d9e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 04 03:28:14.305787 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:28:14.305696 4405 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5f397f15-3135-4cee-9d3a-eb62a71e6d9e-kube-api-access-khmmr" pod "5f397f15-3135-4cee-9d3a-eb62a71e6d9e" (UID: "5f397f15-3135-4cee-9d3a-eb62a71e6d9e"). InnerVolumeSpecName "kube-api-access-khmmr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 03:28:14.404385 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:28:14.404304 4405 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-khmmr\" (UniqueName: \"kubernetes.io/projected/5f397f15-3135-4cee-9d3a-eb62a71e6d9e-kube-api-access-khmmr\") on node \"ip-10-0-134-125.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:28:14.404385 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:28:14.404333 4405 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5f397f15-3135-4cee-9d3a-eb62a71e6d9e-maas-api-xks-ca\") on node \"ip-10-0-134-125.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:28:14.917249 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:28:14.917219 4405 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-single-main-07ce81-openshift-default-5446f4ff6fwvjql" Nov 04 03:28:15.010327 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:28:15.010244 4405 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5111ad38-b833-4be2-9a7c-e7db5c2bd133-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5111ad38-b833-4be2-9a7c-e7db5c2bd133-istio-envoy\") pod \"5111ad38-b833-4be2-9a7c-e7db5c2bd133\" (UID: \"5111ad38-b833-4be2-9a7c-e7db5c2bd133\") " Nov 04 03:28:15.010327 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:28:15.010299 4405 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/5111ad38-b833-4be2-9a7c-e7db5c2bd133-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5111ad38-b833-4be2-9a7c-e7db5c2bd133-istio-podinfo\") pod \"5111ad38-b833-4be2-9a7c-e7db5c2bd133\" (UID: \"5111ad38-b833-4be2-9a7c-e7db5c2bd133\") " Nov 04 03:28:15.010537 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:28:15.010341 4405 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5111ad38-b833-4be2-9a7c-e7db5c2bd133-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5111ad38-b833-4be2-9a7c-e7db5c2bd133-istio-data\") pod \"5111ad38-b833-4be2-9a7c-e7db5c2bd133\" (UID: \"5111ad38-b833-4be2-9a7c-e7db5c2bd133\") " Nov 04 03:28:15.010537 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:28:15.010374 4405 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5111ad38-b833-4be2-9a7c-e7db5c2bd133-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5111ad38-b833-4be2-9a7c-e7db5c2bd133-credential-socket\") pod \"5111ad38-b833-4be2-9a7c-e7db5c2bd133\" (UID: \"5111ad38-b833-4be2-9a7c-e7db5c2bd133\") " Nov 04 03:28:15.010537 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:28:15.010417 4405 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5111ad38-b833-4be2-9a7c-e7db5c2bd133-istio-token\" (UniqueName: \"kubernetes.io/projected/5111ad38-b833-4be2-9a7c-e7db5c2bd133-istio-token\") pod \"5111ad38-b833-4be2-9a7c-e7db5c2bd133\" (UID: \"5111ad38-b833-4be2-9a7c-e7db5c2bd133\") " Nov 04 03:28:15.010537 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:28:15.010465 4405 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5111ad38-b833-4be2-9a7c-e7db5c2bd133-kube-api-access-v85pq\" (UniqueName: \"kubernetes.io/projected/5111ad38-b833-4be2-9a7c-e7db5c2bd133-kube-api-access-v85pq\") pod \"5111ad38-b833-4be2-9a7c-e7db5c2bd133\" (UID: \"5111ad38-b833-4be2-9a7c-e7db5c2bd133\") " Nov 04 03:28:15.010537 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:28:15.010503 4405 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5111ad38-b833-4be2-9a7c-e7db5c2bd133-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5111ad38-b833-4be2-9a7c-e7db5c2bd133-workload-certs\") pod \"5111ad38-b833-4be2-9a7c-e7db5c2bd133\" (UID: \"5111ad38-b833-4be2-9a7c-e7db5c2bd133\") " Nov 04 03:28:15.010914 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:28:15.010539 4405 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5111ad38-b833-4be2-9a7c-e7db5c2bd133-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5111ad38-b833-4be2-9a7c-e7db5c2bd133-workload-socket\") pod \"5111ad38-b833-4be2-9a7c-e7db5c2bd133\" (UID: \"5111ad38-b833-4be2-9a7c-e7db5c2bd133\") " Nov 04 03:28:15.010914 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:28:15.010627 4405 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5111ad38-b833-4be2-9a7c-e7db5c2bd133-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5111ad38-b833-4be2-9a7c-e7db5c2bd133-istiod-ca-cert\") pod \"5111ad38-b833-4be2-9a7c-e7db5c2bd133\" (UID: \"5111ad38-b833-4be2-9a7c-e7db5c2bd133\") " Nov 04 03:28:15.010914 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:28:15.010654 4405 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5111ad38-b833-4be2-9a7c-e7db5c2bd133-istio-data" pod "5111ad38-b833-4be2-9a7c-e7db5c2bd133" (UID: "5111ad38-b833-4be2-9a7c-e7db5c2bd133"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 03:28:15.010914 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:28:15.010724 4405 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5111ad38-b833-4be2-9a7c-e7db5c2bd133-credential-socket" pod "5111ad38-b833-4be2-9a7c-e7db5c2bd133" (UID: "5111ad38-b833-4be2-9a7c-e7db5c2bd133"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 03:28:15.011314 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:28:15.010973 4405 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5111ad38-b833-4be2-9a7c-e7db5c2bd133-workload-certs" pod "5111ad38-b833-4be2-9a7c-e7db5c2bd133" (UID: "5111ad38-b833-4be2-9a7c-e7db5c2bd133"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 03:28:15.011314 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:28:15.011000 4405 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5111ad38-b833-4be2-9a7c-e7db5c2bd133-istio-data\") on node \"ip-10-0-134-125.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:28:15.011314 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:28:15.011020 4405 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5111ad38-b833-4be2-9a7c-e7db5c2bd133-credential-socket\") on node \"ip-10-0-134-125.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:28:15.011314 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:28:15.011237 4405 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5111ad38-b833-4be2-9a7c-e7db5c2bd133-istiod-ca-cert" pod "5111ad38-b833-4be2-9a7c-e7db5c2bd133" (UID: "5111ad38-b833-4be2-9a7c-e7db5c2bd133"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 03:28:15.011574 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:28:15.011545 4405 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5111ad38-b833-4be2-9a7c-e7db5c2bd133-workload-socket" pod "5111ad38-b833-4be2-9a7c-e7db5c2bd133" (UID: "5111ad38-b833-4be2-9a7c-e7db5c2bd133"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 03:28:15.012791 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:28:15.012762 4405 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/5111ad38-b833-4be2-9a7c-e7db5c2bd133-istio-podinfo" pod "5111ad38-b833-4be2-9a7c-e7db5c2bd133" (UID: "5111ad38-b833-4be2-9a7c-e7db5c2bd133"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 04 03:28:15.012909 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:28:15.012803 4405 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5111ad38-b833-4be2-9a7c-e7db5c2bd133-istio-envoy" pod "5111ad38-b833-4be2-9a7c-e7db5c2bd133" (UID: "5111ad38-b833-4be2-9a7c-e7db5c2bd133"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 03:28:15.012909 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:28:15.012836 4405 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5111ad38-b833-4be2-9a7c-e7db5c2bd133-istio-token" pod "5111ad38-b833-4be2-9a7c-e7db5c2bd133" (UID: "5111ad38-b833-4be2-9a7c-e7db5c2bd133"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 03:28:15.012909 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:28:15.012846 4405 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5111ad38-b833-4be2-9a7c-e7db5c2bd133-kube-api-access-v85pq" pod "5111ad38-b833-4be2-9a7c-e7db5c2bd133" (UID: "5111ad38-b833-4be2-9a7c-e7db5c2bd133"). InnerVolumeSpecName "kube-api-access-v85pq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 03:28:15.098632 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:28:15.098585 4405 generic.go:356] "Generic (PLEG): container finished" podID="5111ad38-b833-4be2-9a7c-e7db5c2bd133" containerID="1f34c83c800a216334ce52533b55edf58aeb97d9460da99b5e01caf7b3da6aa5" exitCode=0 Nov 04 03:28:15.098850 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:28:15.098698 4405 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-single-main-07ce81-openshift-default-5446f4ff6fwvjql" Nov 04 03:28:15.098850 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:28:15.098742 4405 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-07ce81-openshift-default-5446f4ff6fwvjql" event={"ID":"5111ad38-b833-4be2-9a7c-e7db5c2bd133","Type":"ContainerDied","Data":"1f34c83c800a216334ce52533b55edf58aeb97d9460da99b5e01caf7b3da6aa5"} Nov 04 03:28:15.098850 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:28:15.098815 4405 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-07ce81-openshift-default-5446f4ff6fwvjql" event={"ID":"5111ad38-b833-4be2-9a7c-e7db5c2bd133","Type":"ContainerDied","Data":"d6f29e4f34a742903616b986a680fa86f972b2c24374c6dc234e5e60aeae202b"} Nov 04 03:28:15.099047 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:28:15.098854 4405 scope.go:122] "RemoveContainer" containerID="1f34c83c800a216334ce52533b55edf58aeb97d9460da99b5e01caf7b3da6aa5" Nov 04 03:28:15.101938 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:28:15.101830 4405 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-9ktbr" event={"ID":"5f397f15-3135-4cee-9d3a-eb62a71e6d9e","Type":"ContainerDied","Data":"ee12f32ad320f39559ee26d4847b5f459a15b772cdded9958d9f5a917f757860"} Nov 04 03:28:15.101938 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:28:15.101885 4405 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-9ktbr" Nov 04 03:28:15.101938 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:28:15.101902 4405 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ee12f32ad320f39559ee26d4847b5f459a15b772cdded9958d9f5a917f757860" Nov 04 03:28:15.111834 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:28:15.111810 4405 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5111ad38-b833-4be2-9a7c-e7db5c2bd133-istiod-ca-cert\") on node \"ip-10-0-134-125.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:28:15.112003 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:28:15.111987 4405 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5111ad38-b833-4be2-9a7c-e7db5c2bd133-istio-envoy\") on node \"ip-10-0-134-125.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:28:15.112133 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:28:15.112085 4405 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5111ad38-b833-4be2-9a7c-e7db5c2bd133-istio-podinfo\") on node \"ip-10-0-134-125.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:28:15.112133 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:28:15.112108 4405 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5111ad38-b833-4be2-9a7c-e7db5c2bd133-istio-token\") on node \"ip-10-0-134-125.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:28:15.112133 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:28:15.112125 4405 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v85pq\" (UniqueName: \"kubernetes.io/projected/5111ad38-b833-4be2-9a7c-e7db5c2bd133-kube-api-access-v85pq\") on node \"ip-10-0-134-125.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:28:15.112348 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:28:15.112141 4405 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5111ad38-b833-4be2-9a7c-e7db5c2bd133-workload-certs\") on node \"ip-10-0-134-125.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:28:15.112348 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:28:15.112157 4405 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5111ad38-b833-4be2-9a7c-e7db5c2bd133-workload-socket\") on node \"ip-10-0-134-125.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:28:15.116439 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:28:15.116417 4405 scope.go:122] "RemoveContainer" containerID="38fb836dba0859accccb17373b4887fbef0fcc10896302cc8df2067efa08b6d2" Nov 04 03:28:15.126889 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:28:15.126863 4405 scope.go:122] "RemoveContainer" containerID="1f34c83c800a216334ce52533b55edf58aeb97d9460da99b5e01caf7b3da6aa5" Nov 04 03:28:15.127232 ip-10-0-134-125 kubenswrapper[4405]: E1104 03:28:15.127211 4405 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1f34c83c800a216334ce52533b55edf58aeb97d9460da99b5e01caf7b3da6aa5\": container with ID starting with 1f34c83c800a216334ce52533b55edf58aeb97d9460da99b5e01caf7b3da6aa5 not found: ID does not exist" containerID="1f34c83c800a216334ce52533b55edf58aeb97d9460da99b5e01caf7b3da6aa5" Nov 04 03:28:15.127286 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:28:15.127246 4405 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1f34c83c800a216334ce52533b55edf58aeb97d9460da99b5e01caf7b3da6aa5"} err="failed to get container status \"1f34c83c800a216334ce52533b55edf58aeb97d9460da99b5e01caf7b3da6aa5\": rpc error: code = NotFound desc = could not find container \"1f34c83c800a216334ce52533b55edf58aeb97d9460da99b5e01caf7b3da6aa5\": container with ID starting with 1f34c83c800a216334ce52533b55edf58aeb97d9460da99b5e01caf7b3da6aa5 not found: ID does not exist" Nov 04 03:28:15.127286 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:28:15.127264 4405 scope.go:122] "RemoveContainer" containerID="38fb836dba0859accccb17373b4887fbef0fcc10896302cc8df2067efa08b6d2" Nov 04 03:28:15.127571 ip-10-0-134-125 kubenswrapper[4405]: E1104 03:28:15.127551 4405 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"38fb836dba0859accccb17373b4887fbef0fcc10896302cc8df2067efa08b6d2\": container with ID starting with 38fb836dba0859accccb17373b4887fbef0fcc10896302cc8df2067efa08b6d2 not found: ID does not exist" containerID="38fb836dba0859accccb17373b4887fbef0fcc10896302cc8df2067efa08b6d2" Nov 04 03:28:15.127639 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:28:15.127577 4405 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"38fb836dba0859accccb17373b4887fbef0fcc10896302cc8df2067efa08b6d2"} err="failed to get container status \"38fb836dba0859accccb17373b4887fbef0fcc10896302cc8df2067efa08b6d2\": rpc error: code = NotFound desc = could not find container \"38fb836dba0859accccb17373b4887fbef0fcc10896302cc8df2067efa08b6d2\": container with ID starting with 38fb836dba0859accccb17373b4887fbef0fcc10896302cc8df2067efa08b6d2 not found: ID does not exist" Nov 04 03:28:15.263615 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:28:15.263529 4405 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-single-main-07ce81-openshift-default-5446f4ff6fwvjql"] Nov 04 03:28:15.270497 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:28:15.270464 4405 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-single-main-07ce81-openshift-default-5446f4ff6fwvjql"] Nov 04 03:28:15.699838 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:28:15.699805 4405 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5111ad38-b833-4be2-9a7c-e7db5c2bd133" path="/var/lib/kubelet/pods/5111ad38-b833-4be2-9a7c-e7db5c2bd133/volumes" Nov 04 03:28:26.054967 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:28:26.054931 4405 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-bl5mg"] Nov 04 03:28:27.975838 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:28:27.975789 4405 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76597d6855-dqn2b container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.139:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 04 03:28:27.976267 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:28:27.975872 4405 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-dqn2b" podUID="415bafea-577e-4fba-925e-eb2c492e3cca" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.139:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 04 03:28:30.194829 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:28:30.194790 4405 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-mw848"] Nov 04 03:28:30.195332 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:28:30.195015 4405 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-mw848" podUID="f540e538-4c5c-41eb-bb64-0d964c4810ea" containerName="manager" containerID="cri-o://36a4c11b22e50789d76014dba1026875b618bab5e94cae29a2aff907507b0031" gracePeriod=10 Nov 04 03:28:31.649193 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:28:31.649164 4405 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-mw848" Nov 04 03:28:31.698496 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:28:31.698462 4405 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f540e538-4c5c-41eb-bb64-0d964c4810ea-kube-api-access-gtf72\" (UniqueName: \"kubernetes.io/projected/f540e538-4c5c-41eb-bb64-0d964c4810ea-kube-api-access-gtf72\") pod \"f540e538-4c5c-41eb-bb64-0d964c4810ea\" (UID: \"f540e538-4c5c-41eb-bb64-0d964c4810ea\") " Nov 04 03:28:31.698640 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:28:31.698583 4405 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f540e538-4c5c-41eb-bb64-0d964c4810ea-extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/f540e538-4c5c-41eb-bb64-0d964c4810ea-extensions-socket-volume\") pod \"f540e538-4c5c-41eb-bb64-0d964c4810ea\" (UID: \"f540e538-4c5c-41eb-bb64-0d964c4810ea\") " Nov 04 03:28:31.698961 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:28:31.698935 4405 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f540e538-4c5c-41eb-bb64-0d964c4810ea-extensions-socket-volume" pod "f540e538-4c5c-41eb-bb64-0d964c4810ea" (UID: "f540e538-4c5c-41eb-bb64-0d964c4810ea"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 03:28:31.701015 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:28:31.700991 4405 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f540e538-4c5c-41eb-bb64-0d964c4810ea-kube-api-access-gtf72" pod "f540e538-4c5c-41eb-bb64-0d964c4810ea" (UID: "f540e538-4c5c-41eb-bb64-0d964c4810ea"). InnerVolumeSpecName "kube-api-access-gtf72". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 03:28:31.799579 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:28:31.799478 4405 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gtf72\" (UniqueName: \"kubernetes.io/projected/f540e538-4c5c-41eb-bb64-0d964c4810ea-kube-api-access-gtf72\") on node \"ip-10-0-134-125.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:28:31.799579 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:28:31.799522 4405 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/f540e538-4c5c-41eb-bb64-0d964c4810ea-extensions-socket-volume\") on node \"ip-10-0-134-125.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:28:32.210128 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:28:32.210088 4405 generic.go:356] "Generic (PLEG): container finished" podID="f540e538-4c5c-41eb-bb64-0d964c4810ea" containerID="36a4c11b22e50789d76014dba1026875b618bab5e94cae29a2aff907507b0031" exitCode=0 Nov 04 03:28:32.210315 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:28:32.210139 4405 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-mw848" Nov 04 03:28:32.210315 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:28:32.210170 4405 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-mw848" event={"ID":"f540e538-4c5c-41eb-bb64-0d964c4810ea","Type":"ContainerDied","Data":"36a4c11b22e50789d76014dba1026875b618bab5e94cae29a2aff907507b0031"} Nov 04 03:28:32.210315 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:28:32.210227 4405 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-mw848" event={"ID":"f540e538-4c5c-41eb-bb64-0d964c4810ea","Type":"ContainerDied","Data":"a8b27ca9db13255821ac7b9ecf2ab016c3eafb11d08f35c81ab8ac7bdbfc2af3"} Nov 04 03:28:32.210315 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:28:32.210249 4405 scope.go:122] "RemoveContainer" containerID="36a4c11b22e50789d76014dba1026875b618bab5e94cae29a2aff907507b0031" Nov 04 03:28:32.231083 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:28:32.231058 4405 scope.go:122] "RemoveContainer" containerID="36a4c11b22e50789d76014dba1026875b618bab5e94cae29a2aff907507b0031" Nov 04 03:28:32.231446 ip-10-0-134-125 kubenswrapper[4405]: E1104 03:28:32.231414 4405 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"36a4c11b22e50789d76014dba1026875b618bab5e94cae29a2aff907507b0031\": container with ID starting with 36a4c11b22e50789d76014dba1026875b618bab5e94cae29a2aff907507b0031 not found: ID does not exist" containerID="36a4c11b22e50789d76014dba1026875b618bab5e94cae29a2aff907507b0031" Nov 04 03:28:32.231495 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:28:32.231457 4405 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"36a4c11b22e50789d76014dba1026875b618bab5e94cae29a2aff907507b0031"} err="failed to get container status \"36a4c11b22e50789d76014dba1026875b618bab5e94cae29a2aff907507b0031\": rpc error: code = NotFound desc = could not find container \"36a4c11b22e50789d76014dba1026875b618bab5e94cae29a2aff907507b0031\": container with ID starting with 36a4c11b22e50789d76014dba1026875b618bab5e94cae29a2aff907507b0031 not found: ID does not exist" Nov 04 03:28:32.374248 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:28:32.374201 4405 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-mw848"] Nov 04 03:28:32.424340 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:28:32.424289 4405 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-mw848"] Nov 04 03:28:33.700494 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:28:33.700452 4405 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f540e538-4c5c-41eb-bb64-0d964c4810ea" path="/var/lib/kubelet/pods/f540e538-4c5c-41eb-bb64-0d964c4810ea/volumes" Nov 04 03:28:34.049761 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:28:34.049683 4405 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-cpcwh"] Nov 04 03:28:34.099693 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:28:34.099641 4405 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-cpcwh"] Nov 04 03:28:35.368825 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:28:35.368790 4405 scope.go:122] "RemoveContainer" containerID="093db2af3c688c900cd8e89831c23df16d07ced00cab05013b3a807cb4c5d141" Nov 04 03:28:35.377636 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:28:35.377617 4405 scope.go:122] "RemoveContainer" containerID="034a5fa4b16710a4bca4ac24b226b29558472d7c002b4059fc4e0c12a44ff46d" Nov 04 03:28:35.699702 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:28:35.699604 4405 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4cfd1631-3c74-44ee-a5b2-5af21c02f74b" path="/var/lib/kubelet/pods/4cfd1631-3c74-44ee-a5b2-5af21c02f74b/volumes" Nov 04 03:28:36.882292 ip-10-0-134-125 kubenswrapper[4405]: E1104 03:28:36.882255 4405 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5f2vb" containerName="extractor" Nov 04 03:28:51.048738 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:28:51.048635 4405 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-7dh59"] Nov 04 03:28:51.098194 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:28:51.098164 4405 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-7dh59"] Nov 04 03:28:51.701789 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:28:51.701759 4405 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="23d11cb3-9270-41ad-999e-54f71f4d310d" path="/var/lib/kubelet/pods/23d11cb3-9270-41ad-999e-54f71f4d310d/volumes" Nov 04 03:29:06.881658 ip-10-0-134-125 kubenswrapper[4405]: E1104 03:29:06.881622 4405 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5f2vb" containerName="extractor" Nov 04 03:29:33.794243 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:29:33.794206 4405 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-jcwkh"] Nov 04 03:29:33.794744 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:29:33.794557 4405 cpu_manager.go:414] "containerMap: removing container" podUID="f540e538-4c5c-41eb-bb64-0d964c4810ea" containerName="manager" Nov 04 03:29:33.794744 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:29:33.794568 4405 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f540e538-4c5c-41eb-bb64-0d964c4810ea" containerName="manager" Nov 04 03:29:33.794744 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:29:33.794583 4405 cpu_manager.go:414] "containerMap: removing container" podUID="5111ad38-b833-4be2-9a7c-e7db5c2bd133" containerName="istio-proxy" Nov 04 03:29:33.794744 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:29:33.794589 4405 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5111ad38-b833-4be2-9a7c-e7db5c2bd133" containerName="istio-proxy" Nov 04 03:29:33.794744 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:29:33.794599 4405 cpu_manager.go:414] "containerMap: removing container" podUID="5f397f15-3135-4cee-9d3a-eb62a71e6d9e" containerName="revoke-keys" Nov 04 03:29:33.794744 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:29:33.794605 4405 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5f397f15-3135-4cee-9d3a-eb62a71e6d9e" containerName="revoke-keys" Nov 04 03:29:33.794744 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:29:33.794710 4405 memory_manager.go:365] "RemoveStaleState removing state" podUID="f540e538-4c5c-41eb-bb64-0d964c4810ea" containerName="manager" Nov 04 03:29:33.794744 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:29:33.794721 4405 memory_manager.go:365] "RemoveStaleState removing state" podUID="5111ad38-b833-4be2-9a7c-e7db5c2bd133" containerName="istio-proxy" Nov 04 03:29:33.794744 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:29:33.794730 4405 memory_manager.go:365] "RemoveStaleState removing state" podUID="5f397f15-3135-4cee-9d3a-eb62a71e6d9e" containerName="revoke-keys" Nov 04 03:29:33.794744 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:29:33.794737 4405 memory_manager.go:365] "RemoveStaleState removing state" podUID="5111ad38-b833-4be2-9a7c-e7db5c2bd133" containerName="istio-proxy" Nov 04 03:29:33.797882 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:29:33.797865 4405 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-jcwkh" Nov 04 03:29:33.846047 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:29:33.846013 4405 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-jcwkh"] Nov 04 03:29:33.876112 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:29:33.876078 4405 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4h9b9\" (UniqueName: \"kubernetes.io/projected/f647d4a1-e49d-4562-8440-6fccab60fc85-kube-api-access-4h9b9\") pod \"kuadrant-operator-controller-manager-678d87766f-jcwkh\" (UID: \"f647d4a1-e49d-4562-8440-6fccab60fc85\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-jcwkh" Nov 04 03:29:33.876323 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:29:33.876250 4405 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/f647d4a1-e49d-4562-8440-6fccab60fc85-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-678d87766f-jcwkh\" (UID: \"f647d4a1-e49d-4562-8440-6fccab60fc85\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-jcwkh" Nov 04 03:29:33.893872 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:29:33.893847 4405 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-pbzpn\"" Nov 04 03:29:33.977285 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:29:33.977247 4405 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4h9b9\" (UniqueName: \"kubernetes.io/projected/f647d4a1-e49d-4562-8440-6fccab60fc85-kube-api-access-4h9b9\") pod \"kuadrant-operator-controller-manager-678d87766f-jcwkh\" (UID: \"f647d4a1-e49d-4562-8440-6fccab60fc85\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-jcwkh" Nov 04 03:29:33.977493 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:29:33.977377 4405 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/f647d4a1-e49d-4562-8440-6fccab60fc85-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-678d87766f-jcwkh\" (UID: \"f647d4a1-e49d-4562-8440-6fccab60fc85\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-jcwkh" Nov 04 03:29:33.977795 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:29:33.977774 4405 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/f647d4a1-e49d-4562-8440-6fccab60fc85-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-678d87766f-jcwkh\" (UID: \"f647d4a1-e49d-4562-8440-6fccab60fc85\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-jcwkh" Nov 04 03:29:34.038457 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:29:34.038427 4405 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4h9b9\" (UniqueName: \"kubernetes.io/projected/f647d4a1-e49d-4562-8440-6fccab60fc85-kube-api-access-4h9b9\") pod \"kuadrant-operator-controller-manager-678d87766f-jcwkh\" (UID: \"f647d4a1-e49d-4562-8440-6fccab60fc85\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-jcwkh" Nov 04 03:29:34.110483 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:29:34.110450 4405 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-jcwkh" Nov 04 03:29:34.585336 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:29:34.585299 4405 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-jcwkh"] Nov 04 03:29:34.624653 ip-10-0-134-125 kubenswrapper[4405]: W1104 03:29:34.624618 4405 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf647d4a1_e49d_4562_8440_6fccab60fc85.slice/crio-5dac41d0e188f7a707410465bd3d25e814619732243202acfc1696f457826aaf WatchSource:0}: Error finding container 5dac41d0e188f7a707410465bd3d25e814619732243202acfc1696f457826aaf: Status 404 returned error can't find the container with id 5dac41d0e188f7a707410465bd3d25e814619732243202acfc1696f457826aaf Nov 04 03:29:35.416932 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:29:35.416897 4405 scope.go:122] "RemoveContainer" containerID="50c71ad3dd2fdaee87a29f92529284365e9b51f6b210c320fb850679e5913258" Nov 04 03:29:35.426349 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:29:35.426329 4405 scope.go:122] "RemoveContainer" containerID="78bc6f432dc5e7c3db66269e82669a8c9c70e34e4db04db0678a85160ef81a50" Nov 04 03:29:35.434750 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:29:35.434732 4405 scope.go:122] "RemoveContainer" containerID="dce3edf67cda61f1c2f37ed58cf318a842444277ae95cc9241f1de480b71cf53" Nov 04 03:29:35.524924 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:29:35.524882 4405 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-jcwkh" event={"ID":"f647d4a1-e49d-4562-8440-6fccab60fc85","Type":"ContainerStarted","Data":"b6552cd47aa0689f9fa1532b4b9400df643607b4cb161742e5970ec1ae6a1e63"} Nov 04 03:29:35.525101 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:29:35.524943 4405 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-jcwkh" event={"ID":"f647d4a1-e49d-4562-8440-6fccab60fc85","Type":"ContainerStarted","Data":"5dac41d0e188f7a707410465bd3d25e814619732243202acfc1696f457826aaf"} Nov 04 03:29:35.525101 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:29:35.524984 4405 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-jcwkh" Nov 04 03:29:35.744287 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:29:35.744191 4405 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-jcwkh" podStartSLOduration=2.74417837 podStartE2EDuration="2.74417837s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 03:29:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 03:29:35.692896595 +0000 UTC m=+3422.123596716" watchObservedRunningTime="2026-11-04 03:29:35.74417837 +0000 UTC m=+3422.174878491" Nov 04 03:29:36.882335 ip-10-0-134-125 kubenswrapper[4405]: E1104 03:29:36.882290 4405 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5f2vb" containerName="extractor" Nov 04 03:29:46.532632 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:29:46.532591 4405 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-jcwkh" Nov 04 03:29:57.975830 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:29:57.975784 4405 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76597d6855-dqn2b container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.139:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 04 03:29:57.976273 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:29:57.975845 4405 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-dqn2b" podUID="415bafea-577e-4fba-925e-eb2c492e3cca" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.139:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 04 03:29:59.622178 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:29:59.622144 4405 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-bl5mg"] Nov 04 03:30:00.169111 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:30:00.169071 4405 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29896050-6vcv8"] Nov 04 03:30:00.169476 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:30:00.169462 4405 cpu_manager.go:414] "containerMap: removing container" podUID="5111ad38-b833-4be2-9a7c-e7db5c2bd133" containerName="istio-proxy" Nov 04 03:30:00.169522 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:30:00.169477 4405 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5111ad38-b833-4be2-9a7c-e7db5c2bd133" containerName="istio-proxy" Nov 04 03:30:00.172710 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:30:00.172693 4405 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29896050-6vcv8" Nov 04 03:30:00.178528 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:30:00.178496 4405 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7jhhq\" (UniqueName: \"kubernetes.io/projected/4b058f6d-1df7-4ceb-a63e-af1bc8a0f420-kube-api-access-7jhhq\") pod \"maas-api-key-cleanup-29896050-6vcv8\" (UID: \"4b058f6d-1df7-4ceb-a63e-af1bc8a0f420\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29896050-6vcv8" Nov 04 03:30:00.178705 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:30:00.178566 4405 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4b058f6d-1df7-4ceb-a63e-af1bc8a0f420-maas-api-service-ca\") pod \"maas-api-key-cleanup-29896050-6vcv8\" (UID: \"4b058f6d-1df7-4ceb-a63e-af1bc8a0f420\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29896050-6vcv8" Nov 04 03:30:00.274480 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:30:00.274442 4405 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 04 03:30:00.274480 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:30:00.274448 4405 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-slbq8\"" Nov 04 03:30:00.274480 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:30:00.274485 4405 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 04 03:30:00.276205 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:30:00.276182 4405 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29896050-6vcv8"] Nov 04 03:30:00.279154 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:30:00.279128 4405 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4b058f6d-1df7-4ceb-a63e-af1bc8a0f420-maas-api-service-ca\") pod \"maas-api-key-cleanup-29896050-6vcv8\" (UID: \"4b058f6d-1df7-4ceb-a63e-af1bc8a0f420\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29896050-6vcv8" Nov 04 03:30:00.279281 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:30:00.279201 4405 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7jhhq\" (UniqueName: \"kubernetes.io/projected/4b058f6d-1df7-4ceb-a63e-af1bc8a0f420-kube-api-access-7jhhq\") pod \"maas-api-key-cleanup-29896050-6vcv8\" (UID: \"4b058f6d-1df7-4ceb-a63e-af1bc8a0f420\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29896050-6vcv8" Nov 04 03:30:00.279751 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:30:00.279735 4405 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4b058f6d-1df7-4ceb-a63e-af1bc8a0f420-maas-api-service-ca\") pod \"maas-api-key-cleanup-29896050-6vcv8\" (UID: \"4b058f6d-1df7-4ceb-a63e-af1bc8a0f420\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29896050-6vcv8" Nov 04 03:30:00.340731 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:30:00.340693 4405 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7jhhq\" (UniqueName: \"kubernetes.io/projected/4b058f6d-1df7-4ceb-a63e-af1bc8a0f420-kube-api-access-7jhhq\") pod \"maas-api-key-cleanup-29896050-6vcv8\" (UID: \"4b058f6d-1df7-4ceb-a63e-af1bc8a0f420\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29896050-6vcv8" Nov 04 03:30:00.486993 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:30:00.486909 4405 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29896050-6vcv8" Nov 04 03:30:00.860760 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:30:00.860728 4405 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29896050-6vcv8"] Nov 04 03:30:00.862903 ip-10-0-134-125 kubenswrapper[4405]: W1104 03:30:00.862864 4405 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4b058f6d_1df7_4ceb_a63e_af1bc8a0f420.slice/crio-2ace9414c386ff80384620c841f37785db9d343eeb0c99cd7f6185286a5cc5cb WatchSource:0}: Error finding container 2ace9414c386ff80384620c841f37785db9d343eeb0c99cd7f6185286a5cc5cb: Status 404 returned error can't find the container with id 2ace9414c386ff80384620c841f37785db9d343eeb0c99cd7f6185286a5cc5cb Nov 04 03:30:01.730763 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:30:01.730726 4405 generic.go:356] "Generic (PLEG): container finished" podID="4b058f6d-1df7-4ceb-a63e-af1bc8a0f420" containerID="5ed89eb4a99bba8df19f3712c7571c58e98aa8a44cfc3e20848575394405a492" exitCode=60 Nov 04 03:30:01.730942 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:30:01.730811 4405 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29896050-6vcv8" event={"ID":"4b058f6d-1df7-4ceb-a63e-af1bc8a0f420","Type":"ContainerDied","Data":"5ed89eb4a99bba8df19f3712c7571c58e98aa8a44cfc3e20848575394405a492"} Nov 04 03:30:01.730942 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:30:01.730869 4405 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29896050-6vcv8" event={"ID":"4b058f6d-1df7-4ceb-a63e-af1bc8a0f420","Type":"ContainerStarted","Data":"2ace9414c386ff80384620c841f37785db9d343eeb0c99cd7f6185286a5cc5cb"} Nov 04 03:30:01.731077 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:30:01.731062 4405 scope.go:122] "RemoveContainer" containerID="5ed89eb4a99bba8df19f3712c7571c58e98aa8a44cfc3e20848575394405a492" Nov 04 03:30:02.737755 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:30:02.737723 4405 generic.go:356] "Generic (PLEG): container finished" podID="4b058f6d-1df7-4ceb-a63e-af1bc8a0f420" containerID="c5e1b3d2a5050e59e07d56f787bc6497fd11ef645c2c3c815db77fbe93c8da46" exitCode=60 Nov 04 03:30:02.738265 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:30:02.737810 4405 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29896050-6vcv8" event={"ID":"4b058f6d-1df7-4ceb-a63e-af1bc8a0f420","Type":"ContainerDied","Data":"c5e1b3d2a5050e59e07d56f787bc6497fd11ef645c2c3c815db77fbe93c8da46"} Nov 04 03:30:02.738265 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:30:02.737878 4405 scope.go:122] "RemoveContainer" containerID="5ed89eb4a99bba8df19f3712c7571c58e98aa8a44cfc3e20848575394405a492" Nov 04 03:30:02.738265 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:30:02.738137 4405 scope.go:122] "RemoveContainer" containerID="c5e1b3d2a5050e59e07d56f787bc6497fd11ef645c2c3c815db77fbe93c8da46" Nov 04 03:30:02.738443 ip-10-0-134-125 kubenswrapper[4405]: E1104 03:30:02.738416 4405 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29896050-6vcv8_odh-ai-gateway-infra(4b058f6d-1df7-4ceb-a63e-af1bc8a0f420)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29896050-6vcv8" podUID="4b058f6d-1df7-4ceb-a63e-af1bc8a0f420" Nov 04 03:30:03.746807 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:30:03.746769 4405 scope.go:122] "RemoveContainer" containerID="c5e1b3d2a5050e59e07d56f787bc6497fd11ef645c2c3c815db77fbe93c8da46" Nov 04 03:30:03.747205 ip-10-0-134-125 kubenswrapper[4405]: E1104 03:30:03.747005 4405 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29896050-6vcv8_odh-ai-gateway-infra(4b058f6d-1df7-4ceb-a63e-af1bc8a0f420)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29896050-6vcv8" podUID="4b058f6d-1df7-4ceb-a63e-af1bc8a0f420" Nov 04 03:30:06.882967 ip-10-0-134-125 kubenswrapper[4405]: E1104 03:30:06.882928 4405 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5f2vb" containerName="extractor" Nov 04 03:30:14.046715 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:30:14.046655 4405 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-cq2lz"] Nov 04 03:30:14.096349 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:30:14.096303 4405 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-cq2lz"] Nov 04 03:30:14.708924 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:30:14.708894 4405 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-bl5mg"] Nov 04 03:30:15.717783 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:30:15.717547 4405 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b583181c-d65d-4b4d-93c8-b9dda58822de" path="/var/lib/kubelet/pods/b583181c-d65d-4b4d-93c8-b9dda58822de/volumes" Nov 04 03:30:16.696707 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:30:16.696334 4405 scope.go:122] "RemoveContainer" containerID="c5e1b3d2a5050e59e07d56f787bc6497fd11ef645c2c3c815db77fbe93c8da46" Nov 04 03:30:17.889094 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:30:17.889006 4405 generic.go:356] "Generic (PLEG): container finished" podID="4b058f6d-1df7-4ceb-a63e-af1bc8a0f420" containerID="c9d437aff711fd472a247291c869e34cf4c60ab9d2a7f8038b367582939f414a" exitCode=60 Nov 04 03:30:17.889576 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:30:17.889081 4405 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29896050-6vcv8" event={"ID":"4b058f6d-1df7-4ceb-a63e-af1bc8a0f420","Type":"ContainerDied","Data":"c9d437aff711fd472a247291c869e34cf4c60ab9d2a7f8038b367582939f414a"} Nov 04 03:30:17.889576 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:30:17.889149 4405 scope.go:122] "RemoveContainer" containerID="c5e1b3d2a5050e59e07d56f787bc6497fd11ef645c2c3c815db77fbe93c8da46" Nov 04 03:30:17.889576 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:30:17.889533 4405 scope.go:122] "RemoveContainer" containerID="c9d437aff711fd472a247291c869e34cf4c60ab9d2a7f8038b367582939f414a" Nov 04 03:30:17.889855 ip-10-0-134-125 kubenswrapper[4405]: E1104 03:30:17.889833 4405 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-29896050-6vcv8_odh-ai-gateway-infra(4b058f6d-1df7-4ceb-a63e-af1bc8a0f420)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29896050-6vcv8" podUID="4b058f6d-1df7-4ceb-a63e-af1bc8a0f420" Nov 04 03:30:19.017262 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:30:19.017226 4405 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29896050-6vcv8"] Nov 04 03:30:19.170893 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:30:19.170868 4405 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29896050-6vcv8" Nov 04 03:30:19.288624 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:30:19.288518 4405 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4b058f6d-1df7-4ceb-a63e-af1bc8a0f420-kube-api-access-7jhhq\" (UniqueName: \"kubernetes.io/projected/4b058f6d-1df7-4ceb-a63e-af1bc8a0f420-kube-api-access-7jhhq\") pod \"4b058f6d-1df7-4ceb-a63e-af1bc8a0f420\" (UID: \"4b058f6d-1df7-4ceb-a63e-af1bc8a0f420\") " Nov 04 03:30:19.288793 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:30:19.288656 4405 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4b058f6d-1df7-4ceb-a63e-af1bc8a0f420-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4b058f6d-1df7-4ceb-a63e-af1bc8a0f420-maas-api-service-ca\") pod \"4b058f6d-1df7-4ceb-a63e-af1bc8a0f420\" (UID: \"4b058f6d-1df7-4ceb-a63e-af1bc8a0f420\") " Nov 04 03:30:19.289278 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:30:19.289248 4405 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4b058f6d-1df7-4ceb-a63e-af1bc8a0f420-maas-api-service-ca" pod "4b058f6d-1df7-4ceb-a63e-af1bc8a0f420" (UID: "4b058f6d-1df7-4ceb-a63e-af1bc8a0f420"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 03:30:19.291829 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:30:19.291801 4405 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4b058f6d-1df7-4ceb-a63e-af1bc8a0f420-kube-api-access-7jhhq" pod "4b058f6d-1df7-4ceb-a63e-af1bc8a0f420" (UID: "4b058f6d-1df7-4ceb-a63e-af1bc8a0f420"). InnerVolumeSpecName "kube-api-access-7jhhq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 03:30:19.390130 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:30:19.390092 4405 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7jhhq\" (UniqueName: \"kubernetes.io/projected/4b058f6d-1df7-4ceb-a63e-af1bc8a0f420-kube-api-access-7jhhq\") on node \"ip-10-0-134-125.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:30:19.390130 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:30:19.390124 4405 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4b058f6d-1df7-4ceb-a63e-af1bc8a0f420-maas-api-service-ca\") on node \"ip-10-0-134-125.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:30:19.901776 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:30:19.901742 4405 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29896050-6vcv8" Nov 04 03:30:19.901964 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:30:19.901741 4405 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29896050-6vcv8" event={"ID":"4b058f6d-1df7-4ceb-a63e-af1bc8a0f420","Type":"ContainerDied","Data":"2ace9414c386ff80384620c841f37785db9d343eeb0c99cd7f6185286a5cc5cb"} Nov 04 03:30:19.901964 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:30:19.901879 4405 scope.go:122] "RemoveContainer" containerID="c9d437aff711fd472a247291c869e34cf4c60ab9d2a7f8038b367582939f414a" Nov 04 03:30:20.065431 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:30:20.065337 4405 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29896050-6vcv8"] Nov 04 03:30:20.114939 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:30:20.114908 4405 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29896050-6vcv8"] Nov 04 03:30:21.710339 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:30:21.710288 4405 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4b058f6d-1df7-4ceb-a63e-af1bc8a0f420" path="/var/lib/kubelet/pods/4b058f6d-1df7-4ceb-a63e-af1bc8a0f420/volumes" Nov 04 03:30:23.877402 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:30:23.877362 4405 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-px88h"] Nov 04 03:30:23.877818 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:30:23.877725 4405 cpu_manager.go:414] "containerMap: removing container" podUID="4b058f6d-1df7-4ceb-a63e-af1bc8a0f420" containerName="cleanup" Nov 04 03:30:23.877818 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:30:23.877737 4405 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4b058f6d-1df7-4ceb-a63e-af1bc8a0f420" containerName="cleanup" Nov 04 03:30:23.877818 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:30:23.877752 4405 cpu_manager.go:414] "containerMap: removing container" podUID="4b058f6d-1df7-4ceb-a63e-af1bc8a0f420" containerName="cleanup" Nov 04 03:30:23.877818 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:30:23.877757 4405 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4b058f6d-1df7-4ceb-a63e-af1bc8a0f420" containerName="cleanup" Nov 04 03:30:23.877818 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:30:23.877767 4405 cpu_manager.go:414] "containerMap: removing container" podUID="4b058f6d-1df7-4ceb-a63e-af1bc8a0f420" containerName="cleanup" Nov 04 03:30:23.877818 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:30:23.877772 4405 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4b058f6d-1df7-4ceb-a63e-af1bc8a0f420" containerName="cleanup" Nov 04 03:30:23.878010 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:30:23.877852 4405 memory_manager.go:365] "RemoveStaleState removing state" podUID="4b058f6d-1df7-4ceb-a63e-af1bc8a0f420" containerName="cleanup" Nov 04 03:30:23.878010 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:30:23.877860 4405 memory_manager.go:365] "RemoveStaleState removing state" podUID="4b058f6d-1df7-4ceb-a63e-af1bc8a0f420" containerName="cleanup" Nov 04 03:30:23.882449 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:30:23.882430 4405 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-px88h" Nov 04 03:30:23.938851 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:30:23.938805 4405 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-px88h"] Nov 04 03:30:23.981864 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:30:23.981824 4405 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 04 03:30:23.981864 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:30:23.981827 4405 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 04 03:30:23.985574 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:30:23.985556 4405 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 04 03:30:23.985744 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:30:23.985725 4405 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-slbq8\"" Nov 04 03:30:24.049818 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:30:24.049779 4405 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-nb52q"] Nov 04 03:30:24.054186 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:30:24.054157 4405 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-nb52q"] Nov 04 03:30:24.068304 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:30:24.068273 4405 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6qj5n\" (UniqueName: \"kubernetes.io/projected/b72bc8d7-5e6d-44c0-81aa-81fa4d6165f7-kube-api-access-6qj5n\") pod \"maas-api-revoke-sub-d2bc93bc-px88h\" (UID: \"b72bc8d7-5e6d-44c0-81aa-81fa4d6165f7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-px88h" Nov 04 03:30:24.068431 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:30:24.068338 4405 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b72bc8d7-5e6d-44c0-81aa-81fa4d6165f7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d2bc93bc-px88h\" (UID: \"b72bc8d7-5e6d-44c0-81aa-81fa4d6165f7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-px88h" Nov 04 03:30:24.068431 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:30:24.068424 4405 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b72bc8d7-5e6d-44c0-81aa-81fa4d6165f7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d2bc93bc-px88h\" (UID: \"b72bc8d7-5e6d-44c0-81aa-81fa4d6165f7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-px88h" Nov 04 03:30:24.169925 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:30:24.169831 4405 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6qj5n\" (UniqueName: \"kubernetes.io/projected/b72bc8d7-5e6d-44c0-81aa-81fa4d6165f7-kube-api-access-6qj5n\") pod \"maas-api-revoke-sub-d2bc93bc-px88h\" (UID: \"b72bc8d7-5e6d-44c0-81aa-81fa4d6165f7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-px88h" Nov 04 03:30:24.169925 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:30:24.169901 4405 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b72bc8d7-5e6d-44c0-81aa-81fa4d6165f7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d2bc93bc-px88h\" (UID: \"b72bc8d7-5e6d-44c0-81aa-81fa4d6165f7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-px88h" Nov 04 03:30:24.170136 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:30:24.169949 4405 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b72bc8d7-5e6d-44c0-81aa-81fa4d6165f7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d2bc93bc-px88h\" (UID: \"b72bc8d7-5e6d-44c0-81aa-81fa4d6165f7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-px88h" Nov 04 03:30:24.170750 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:30:24.170729 4405 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b72bc8d7-5e6d-44c0-81aa-81fa4d6165f7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d2bc93bc-px88h\" (UID: \"b72bc8d7-5e6d-44c0-81aa-81fa4d6165f7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-px88h" Nov 04 03:30:24.172690 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:30:24.172644 4405 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b72bc8d7-5e6d-44c0-81aa-81fa4d6165f7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d2bc93bc-px88h\" (UID: \"b72bc8d7-5e6d-44c0-81aa-81fa4d6165f7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-px88h" Nov 04 03:30:24.231261 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:30:24.231222 4405 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6qj5n\" (UniqueName: \"kubernetes.io/projected/b72bc8d7-5e6d-44c0-81aa-81fa4d6165f7-kube-api-access-6qj5n\") pod \"maas-api-revoke-sub-d2bc93bc-px88h\" (UID: \"b72bc8d7-5e6d-44c0-81aa-81fa4d6165f7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-px88h" Nov 04 03:30:24.494809 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:30:24.494722 4405 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-px88h" Nov 04 03:30:24.710186 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:30:24.710155 4405 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-bl5mg"] Nov 04 03:30:24.865980 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:30:24.865941 4405 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-px88h"] Nov 04 03:30:24.868084 ip-10-0-134-125 kubenswrapper[4405]: W1104 03:30:24.868048 4405 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb72bc8d7_5e6d_44c0_81aa_81fa4d6165f7.slice/crio-fd081ce28d2b179f1dec319f8d31d70d6744c0579b1a9232608204a757aeb794 WatchSource:0}: Error finding container fd081ce28d2b179f1dec319f8d31d70d6744c0579b1a9232608204a757aeb794: Status 404 returned error can't find the container with id fd081ce28d2b179f1dec319f8d31d70d6744c0579b1a9232608204a757aeb794 Nov 04 03:30:24.956389 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:30:24.956351 4405 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-px88h" event={"ID":"b72bc8d7-5e6d-44c0-81aa-81fa4d6165f7","Type":"ContainerStarted","Data":"fd081ce28d2b179f1dec319f8d31d70d6744c0579b1a9232608204a757aeb794"} Nov 04 03:30:25.700273 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:30:25.700231 4405 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f679ff00-5722-4370-afa1-455ae3b28127" path="/var/lib/kubelet/pods/f679ff00-5722-4370-afa1-455ae3b28127/volumes" Nov 04 03:30:25.962723 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:30:25.962605 4405 generic.go:356] "Generic (PLEG): container finished" podID="b72bc8d7-5e6d-44c0-81aa-81fa4d6165f7" containerID="b2b6a6dd444e4e284b324d7726050cb2a15e986dc3800e6e05b24649729a9292" exitCode=0 Nov 04 03:30:25.962723 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:30:25.962696 4405 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-px88h" event={"ID":"b72bc8d7-5e6d-44c0-81aa-81fa4d6165f7","Type":"ContainerDied","Data":"b2b6a6dd444e4e284b324d7726050cb2a15e986dc3800e6e05b24649729a9292"} Nov 04 03:30:27.105978 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:30:27.105952 4405 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-px88h" Nov 04 03:30:27.198616 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:30:27.198576 4405 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b72bc8d7-5e6d-44c0-81aa-81fa4d6165f7-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b72bc8d7-5e6d-44c0-81aa-81fa4d6165f7-maas-api-xks-ca\") pod \"b72bc8d7-5e6d-44c0-81aa-81fa4d6165f7\" (UID: \"b72bc8d7-5e6d-44c0-81aa-81fa4d6165f7\") " Nov 04 03:30:27.198616 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:30:27.198621 4405 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b72bc8d7-5e6d-44c0-81aa-81fa4d6165f7-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b72bc8d7-5e6d-44c0-81aa-81fa4d6165f7-maas-api-openshift-ca\") pod \"b72bc8d7-5e6d-44c0-81aa-81fa4d6165f7\" (UID: \"b72bc8d7-5e6d-44c0-81aa-81fa4d6165f7\") " Nov 04 03:30:27.198876 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:30:27.198690 4405 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b72bc8d7-5e6d-44c0-81aa-81fa4d6165f7-kube-api-access-6qj5n\" (UniqueName: \"kubernetes.io/projected/b72bc8d7-5e6d-44c0-81aa-81fa4d6165f7-kube-api-access-6qj5n\") pod \"b72bc8d7-5e6d-44c0-81aa-81fa4d6165f7\" (UID: \"b72bc8d7-5e6d-44c0-81aa-81fa4d6165f7\") " Nov 04 03:30:27.199053 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:30:27.199031 4405 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b72bc8d7-5e6d-44c0-81aa-81fa4d6165f7-maas-api-openshift-ca" pod "b72bc8d7-5e6d-44c0-81aa-81fa4d6165f7" (UID: "b72bc8d7-5e6d-44c0-81aa-81fa4d6165f7"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 03:30:27.200967 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:30:27.200941 4405 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b72bc8d7-5e6d-44c0-81aa-81fa4d6165f7-kube-api-access-6qj5n" pod "b72bc8d7-5e6d-44c0-81aa-81fa4d6165f7" (UID: "b72bc8d7-5e6d-44c0-81aa-81fa4d6165f7"). InnerVolumeSpecName "kube-api-access-6qj5n". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 03:30:27.201087 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:30:27.200974 4405 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b72bc8d7-5e6d-44c0-81aa-81fa4d6165f7-maas-api-xks-ca" pod "b72bc8d7-5e6d-44c0-81aa-81fa4d6165f7" (UID: "b72bc8d7-5e6d-44c0-81aa-81fa4d6165f7"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 04 03:30:27.299487 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:30:27.299398 4405 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b72bc8d7-5e6d-44c0-81aa-81fa4d6165f7-maas-api-xks-ca\") on node \"ip-10-0-134-125.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:30:27.299487 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:30:27.299428 4405 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b72bc8d7-5e6d-44c0-81aa-81fa4d6165f7-maas-api-openshift-ca\") on node \"ip-10-0-134-125.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:30:27.299487 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:30:27.299441 4405 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6qj5n\" (UniqueName: \"kubernetes.io/projected/b72bc8d7-5e6d-44c0-81aa-81fa4d6165f7-kube-api-access-6qj5n\") on node \"ip-10-0-134-125.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:30:27.973202 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:30:27.973115 4405 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-px88h" event={"ID":"b72bc8d7-5e6d-44c0-81aa-81fa4d6165f7","Type":"ContainerDied","Data":"fd081ce28d2b179f1dec319f8d31d70d6744c0579b1a9232608204a757aeb794"} Nov 04 03:30:27.973202 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:30:27.973133 4405 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-px88h" Nov 04 03:30:27.973202 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:30:27.973164 4405 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="fd081ce28d2b179f1dec319f8d31d70d6744c0579b1a9232608204a757aeb794" Nov 04 03:30:33.131683 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:30:33.131627 4405 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-bl5mg"] Nov 04 03:30:35.466757 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:30:35.466723 4405 scope.go:122] "RemoveContainer" containerID="6652fb30b5bd28b13a8550f2902ffcd9e4171c4f4765d1dfc357248848426a59" Nov 04 03:30:35.475521 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:30:35.475497 4405 scope.go:122] "RemoveContainer" containerID="dc56731eb209dd2bec6c7a1591522e0ba02f09356abde992d7a252e1ed352fe5" Nov 04 03:30:35.484240 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:30:35.484217 4405 scope.go:122] "RemoveContainer" containerID="e6becfeac434fad4041e4c03d37445de2908da100c1a1790fcc6ca3366cb9463" Nov 04 03:30:36.707327 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:30:36.707285 4405 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-single-main-2f48c1-openshift-default-6799d44489xnlnj"] Nov 04 03:30:36.707875 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:30:36.707854 4405 cpu_manager.go:414] "containerMap: removing container" podUID="b72bc8d7-5e6d-44c0-81aa-81fa4d6165f7" containerName="revoke-keys" Nov 04 03:30:36.707947 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:30:36.707874 4405 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b72bc8d7-5e6d-44c0-81aa-81fa4d6165f7" containerName="revoke-keys" Nov 04 03:30:36.708013 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:30:36.707978 4405 memory_manager.go:365] "RemoveStaleState removing state" podUID="b72bc8d7-5e6d-44c0-81aa-81fa4d6165f7" containerName="revoke-keys" Nov 04 03:30:36.708013 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:30:36.708002 4405 memory_manager.go:365] "RemoveStaleState removing state" podUID="4b058f6d-1df7-4ceb-a63e-af1bc8a0f420" containerName="cleanup" Nov 04 03:30:36.712772 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:30:36.712746 4405 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-single-main-2f48c1-openshift-default-6799d44489xnlnj" Nov 04 03:30:36.782766 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:30:36.782735 4405 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-single-main-2f48c1-openshift-default-dockercfg-9z6m6\"" Nov 04 03:30:36.788227 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:30:36.788191 4405 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ea05b8c6-9b14-480b-9544-a17bbd3a88d1-istiod-ca-cert\") pod \"e2e-worker-single-main-2f48c1-openshift-default-6799d44489xnlnj\" (UID: \"ea05b8c6-9b14-480b-9544-a17bbd3a88d1\") " pod="openshift-ingress/e2e-worker-single-main-2f48c1-openshift-default-6799d44489xnlnj" Nov 04 03:30:36.788471 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:30:36.788290 4405 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ea05b8c6-9b14-480b-9544-a17bbd3a88d1-workload-socket\") pod \"e2e-worker-single-main-2f48c1-openshift-default-6799d44489xnlnj\" (UID: \"ea05b8c6-9b14-480b-9544-a17bbd3a88d1\") " pod="openshift-ingress/e2e-worker-single-main-2f48c1-openshift-default-6799d44489xnlnj" Nov 04 03:30:36.788471 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:30:36.788315 4405 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ea05b8c6-9b14-480b-9544-a17bbd3a88d1-istio-token\") pod \"e2e-worker-single-main-2f48c1-openshift-default-6799d44489xnlnj\" (UID: \"ea05b8c6-9b14-480b-9544-a17bbd3a88d1\") " pod="openshift-ingress/e2e-worker-single-main-2f48c1-openshift-default-6799d44489xnlnj" Nov 04 03:30:36.788471 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:30:36.788335 4405 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ea05b8c6-9b14-480b-9544-a17bbd3a88d1-credential-socket\") pod \"e2e-worker-single-main-2f48c1-openshift-default-6799d44489xnlnj\" (UID: \"ea05b8c6-9b14-480b-9544-a17bbd3a88d1\") " pod="openshift-ingress/e2e-worker-single-main-2f48c1-openshift-default-6799d44489xnlnj" Nov 04 03:30:36.788471 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:30:36.788368 4405 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ea05b8c6-9b14-480b-9544-a17bbd3a88d1-istio-envoy\") pod \"e2e-worker-single-main-2f48c1-openshift-default-6799d44489xnlnj\" (UID: \"ea05b8c6-9b14-480b-9544-a17bbd3a88d1\") " pod="openshift-ingress/e2e-worker-single-main-2f48c1-openshift-default-6799d44489xnlnj" Nov 04 03:30:36.788976 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:30:36.788949 4405 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ea05b8c6-9b14-480b-9544-a17bbd3a88d1-workload-certs\") pod \"e2e-worker-single-main-2f48c1-openshift-default-6799d44489xnlnj\" (UID: \"ea05b8c6-9b14-480b-9544-a17bbd3a88d1\") " pod="openshift-ingress/e2e-worker-single-main-2f48c1-openshift-default-6799d44489xnlnj" Nov 04 03:30:36.789068 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:30:36.789013 4405 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wzhxh\" (UniqueName: \"kubernetes.io/projected/ea05b8c6-9b14-480b-9544-a17bbd3a88d1-kube-api-access-wzhxh\") pod \"e2e-worker-single-main-2f48c1-openshift-default-6799d44489xnlnj\" (UID: \"ea05b8c6-9b14-480b-9544-a17bbd3a88d1\") " pod="openshift-ingress/e2e-worker-single-main-2f48c1-openshift-default-6799d44489xnlnj" Nov 04 03:30:36.789128 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:30:36.789109 4405 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ea05b8c6-9b14-480b-9544-a17bbd3a88d1-istio-podinfo\") pod \"e2e-worker-single-main-2f48c1-openshift-default-6799d44489xnlnj\" (UID: \"ea05b8c6-9b14-480b-9544-a17bbd3a88d1\") " pod="openshift-ingress/e2e-worker-single-main-2f48c1-openshift-default-6799d44489xnlnj" Nov 04 03:30:36.789465 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:30:36.789170 4405 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ea05b8c6-9b14-480b-9544-a17bbd3a88d1-istio-data\") pod \"e2e-worker-single-main-2f48c1-openshift-default-6799d44489xnlnj\" (UID: \"ea05b8c6-9b14-480b-9544-a17bbd3a88d1\") " pod="openshift-ingress/e2e-worker-single-main-2f48c1-openshift-default-6799d44489xnlnj" Nov 04 03:30:36.834833 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:30:36.834799 4405 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-single-main-2f48c1-openshift-default-6799d44489xnlnj"] Nov 04 03:30:36.890359 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:30:36.890326 4405 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ea05b8c6-9b14-480b-9544-a17bbd3a88d1-workload-socket\") pod \"e2e-worker-single-main-2f48c1-openshift-default-6799d44489xnlnj\" (UID: \"ea05b8c6-9b14-480b-9544-a17bbd3a88d1\") " pod="openshift-ingress/e2e-worker-single-main-2f48c1-openshift-default-6799d44489xnlnj" Nov 04 03:30:36.890558 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:30:36.890366 4405 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ea05b8c6-9b14-480b-9544-a17bbd3a88d1-istio-token\") pod \"e2e-worker-single-main-2f48c1-openshift-default-6799d44489xnlnj\" (UID: \"ea05b8c6-9b14-480b-9544-a17bbd3a88d1\") " pod="openshift-ingress/e2e-worker-single-main-2f48c1-openshift-default-6799d44489xnlnj" Nov 04 03:30:36.890558 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:30:36.890399 4405 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ea05b8c6-9b14-480b-9544-a17bbd3a88d1-credential-socket\") pod \"e2e-worker-single-main-2f48c1-openshift-default-6799d44489xnlnj\" (UID: \"ea05b8c6-9b14-480b-9544-a17bbd3a88d1\") " pod="openshift-ingress/e2e-worker-single-main-2f48c1-openshift-default-6799d44489xnlnj" Nov 04 03:30:36.890759 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:30:36.890736 4405 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ea05b8c6-9b14-480b-9544-a17bbd3a88d1-istio-envoy\") pod \"e2e-worker-single-main-2f48c1-openshift-default-6799d44489xnlnj\" (UID: \"ea05b8c6-9b14-480b-9544-a17bbd3a88d1\") " pod="openshift-ingress/e2e-worker-single-main-2f48c1-openshift-default-6799d44489xnlnj" Nov 04 03:30:36.890759 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:30:36.890749 4405 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ea05b8c6-9b14-480b-9544-a17bbd3a88d1-workload-socket\") pod \"e2e-worker-single-main-2f48c1-openshift-default-6799d44489xnlnj\" (UID: \"ea05b8c6-9b14-480b-9544-a17bbd3a88d1\") " pod="openshift-ingress/e2e-worker-single-main-2f48c1-openshift-default-6799d44489xnlnj" Nov 04 03:30:36.890901 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:30:36.890775 4405 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ea05b8c6-9b14-480b-9544-a17bbd3a88d1-workload-certs\") pod \"e2e-worker-single-main-2f48c1-openshift-default-6799d44489xnlnj\" (UID: \"ea05b8c6-9b14-480b-9544-a17bbd3a88d1\") " pod="openshift-ingress/e2e-worker-single-main-2f48c1-openshift-default-6799d44489xnlnj" Nov 04 03:30:36.890901 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:30:36.890818 4405 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ea05b8c6-9b14-480b-9544-a17bbd3a88d1-credential-socket\") pod \"e2e-worker-single-main-2f48c1-openshift-default-6799d44489xnlnj\" (UID: \"ea05b8c6-9b14-480b-9544-a17bbd3a88d1\") " pod="openshift-ingress/e2e-worker-single-main-2f48c1-openshift-default-6799d44489xnlnj" Nov 04 03:30:36.890901 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:30:36.890875 4405 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wzhxh\" (UniqueName: \"kubernetes.io/projected/ea05b8c6-9b14-480b-9544-a17bbd3a88d1-kube-api-access-wzhxh\") pod \"e2e-worker-single-main-2f48c1-openshift-default-6799d44489xnlnj\" (UID: \"ea05b8c6-9b14-480b-9544-a17bbd3a88d1\") " pod="openshift-ingress/e2e-worker-single-main-2f48c1-openshift-default-6799d44489xnlnj" Nov 04 03:30:36.891066 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:30:36.890949 4405 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ea05b8c6-9b14-480b-9544-a17bbd3a88d1-istio-podinfo\") pod \"e2e-worker-single-main-2f48c1-openshift-default-6799d44489xnlnj\" (UID: \"ea05b8c6-9b14-480b-9544-a17bbd3a88d1\") " pod="openshift-ingress/e2e-worker-single-main-2f48c1-openshift-default-6799d44489xnlnj" Nov 04 03:30:36.891066 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:30:36.890996 4405 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ea05b8c6-9b14-480b-9544-a17bbd3a88d1-istio-data\") pod \"e2e-worker-single-main-2f48c1-openshift-default-6799d44489xnlnj\" (UID: \"ea05b8c6-9b14-480b-9544-a17bbd3a88d1\") " pod="openshift-ingress/e2e-worker-single-main-2f48c1-openshift-default-6799d44489xnlnj" Nov 04 03:30:36.891066 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:30:36.891052 4405 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ea05b8c6-9b14-480b-9544-a17bbd3a88d1-workload-certs\") pod \"e2e-worker-single-main-2f48c1-openshift-default-6799d44489xnlnj\" (UID: \"ea05b8c6-9b14-480b-9544-a17bbd3a88d1\") " pod="openshift-ingress/e2e-worker-single-main-2f48c1-openshift-default-6799d44489xnlnj" Nov 04 03:30:36.891066 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:30:36.891060 4405 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ea05b8c6-9b14-480b-9544-a17bbd3a88d1-istiod-ca-cert\") pod \"e2e-worker-single-main-2f48c1-openshift-default-6799d44489xnlnj\" (UID: \"ea05b8c6-9b14-480b-9544-a17bbd3a88d1\") " pod="openshift-ingress/e2e-worker-single-main-2f48c1-openshift-default-6799d44489xnlnj" Nov 04 03:30:36.891294 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:30:36.891274 4405 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ea05b8c6-9b14-480b-9544-a17bbd3a88d1-istio-data\") pod \"e2e-worker-single-main-2f48c1-openshift-default-6799d44489xnlnj\" (UID: \"ea05b8c6-9b14-480b-9544-a17bbd3a88d1\") " pod="openshift-ingress/e2e-worker-single-main-2f48c1-openshift-default-6799d44489xnlnj" Nov 04 03:30:36.892000 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:30:36.891972 4405 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ea05b8c6-9b14-480b-9544-a17bbd3a88d1-istiod-ca-cert\") pod \"e2e-worker-single-main-2f48c1-openshift-default-6799d44489xnlnj\" (UID: \"ea05b8c6-9b14-480b-9544-a17bbd3a88d1\") " pod="openshift-ingress/e2e-worker-single-main-2f48c1-openshift-default-6799d44489xnlnj" Nov 04 03:30:36.893039 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:30:36.893018 4405 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ea05b8c6-9b14-480b-9544-a17bbd3a88d1-istio-envoy\") pod \"e2e-worker-single-main-2f48c1-openshift-default-6799d44489xnlnj\" (UID: \"ea05b8c6-9b14-480b-9544-a17bbd3a88d1\") " pod="openshift-ingress/e2e-worker-single-main-2f48c1-openshift-default-6799d44489xnlnj" Nov 04 03:30:36.893992 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:30:36.893962 4405 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ea05b8c6-9b14-480b-9544-a17bbd3a88d1-istio-podinfo\") pod \"e2e-worker-single-main-2f48c1-openshift-default-6799d44489xnlnj\" (UID: \"ea05b8c6-9b14-480b-9544-a17bbd3a88d1\") " pod="openshift-ingress/e2e-worker-single-main-2f48c1-openshift-default-6799d44489xnlnj" Nov 04 03:30:36.894741 ip-10-0-134-125 kubenswrapper[4405]: E1104 03:30:36.894713 4405 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5f2vb" containerName="extractor" Nov 04 03:30:36.951365 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:30:36.951336 4405 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ea05b8c6-9b14-480b-9544-a17bbd3a88d1-istio-token\") pod \"e2e-worker-single-main-2f48c1-openshift-default-6799d44489xnlnj\" (UID: \"ea05b8c6-9b14-480b-9544-a17bbd3a88d1\") " pod="openshift-ingress/e2e-worker-single-main-2f48c1-openshift-default-6799d44489xnlnj" Nov 04 03:30:36.951747 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:30:36.951721 4405 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wzhxh\" (UniqueName: \"kubernetes.io/projected/ea05b8c6-9b14-480b-9544-a17bbd3a88d1-kube-api-access-wzhxh\") pod \"e2e-worker-single-main-2f48c1-openshift-default-6799d44489xnlnj\" (UID: \"ea05b8c6-9b14-480b-9544-a17bbd3a88d1\") " pod="openshift-ingress/e2e-worker-single-main-2f48c1-openshift-default-6799d44489xnlnj" Nov 04 03:30:37.025758 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:30:37.025656 4405 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-single-main-2f48c1-openshift-default-6799d44489xnlnj" Nov 04 03:30:37.449148 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:30:37.449125 4405 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-single-main-2f48c1-openshift-default-6799d44489xnlnj"] Nov 04 03:30:37.451170 ip-10-0-134-125 kubenswrapper[4405]: W1104 03:30:37.451141 4405 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podea05b8c6_9b14_480b_9544_a17bbd3a88d1.slice/crio-08e53ebdd80ad474a77e7d50be23172f146c3e846a7e5c5919ca11f5cf5c79d2 WatchSource:0}: Error finding container 08e53ebdd80ad474a77e7d50be23172f146c3e846a7e5c5919ca11f5cf5c79d2: Status 404 returned error can't find the container with id 08e53ebdd80ad474a77e7d50be23172f146c3e846a7e5c5919ca11f5cf5c79d2 Nov 04 03:30:37.453473 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:30:37.453438 4405 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 04 03:30:37.453566 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:30:37.453511 4405 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 04 03:30:37.453566 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:30:37.453543 4405 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 04 03:30:38.023187 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:30:38.023146 4405 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-2f48c1-openshift-default-6799d44489xnlnj" event={"ID":"ea05b8c6-9b14-480b-9544-a17bbd3a88d1","Type":"ContainerStarted","Data":"78a46d541a5f1fdb22df972dc0b20d13beed5ebbbb23ff9826c52316018c0fd1"} Nov 04 03:30:38.023723 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:30:38.023208 4405 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-2f48c1-openshift-default-6799d44489xnlnj" event={"ID":"ea05b8c6-9b14-480b-9544-a17bbd3a88d1","Type":"ContainerStarted","Data":"08e53ebdd80ad474a77e7d50be23172f146c3e846a7e5c5919ca11f5cf5c79d2"} Nov 04 03:30:38.026620 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:30:38.026597 4405 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-single-main-2f48c1-openshift-default-6799d44489xnlnj" Nov 04 03:30:38.032116 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:30:38.032088 4405 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-single-main-2f48c1-openshift-default-6799d44489xnlnj" Nov 04 03:30:38.145465 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:30:38.145418 4405 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-single-main-2f48c1-openshift-default-6799d44489xnlnj" podStartSLOduration=2.145403862 podStartE2EDuration="2.145403862s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 03:30:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 03:30:38.143023231 +0000 UTC m=+3484.573723353" watchObservedRunningTime="2026-11-04 03:30:38.145403862 +0000 UTC m=+3484.576103982" Nov 04 03:30:39.029589 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:30:39.029551 4405 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-single-main-2f48c1-openshift-default-6799d44489xnlnj" Nov 04 03:30:39.030702 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:30:39.030654 4405 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-single-main-2f48c1-openshift-default-6799d44489xnlnj" Nov 04 03:30:44.731910 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:30:44.731867 4405 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-2f48c1-7847czcwdv"] Nov 04 03:30:44.735616 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:30:44.735600 4405 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-2f48c1-7847czcwdv" Nov 04 03:30:44.767659 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:30:44.767625 4405 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gnglr\" (UniqueName: \"kubernetes.io/projected/e5354816-e8fe-4941-91e0-ad2d7b46634b-kube-api-access-gnglr\") pod \"payload-pre-processing-e2e-worker-single-main-2f48c1-7847czcwdv\" (UID: \"e5354816-e8fe-4941-91e0-ad2d7b46634b\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-2f48c1-7847czcwdv" Nov 04 03:30:44.767869 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:30:44.767770 4405 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e5354816-e8fe-4941-91e0-ad2d7b46634b-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-single-main-2f48c1-7847czcwdv\" (UID: \"e5354816-e8fe-4941-91e0-ad2d7b46634b\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-2f48c1-7847czcwdv" Nov 04 03:30:44.836986 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:30:44.836945 4405 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-single-m-e770d961-dockercfg-j5gls\"" Nov 04 03:30:44.837215 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:30:44.837074 4405 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-single-main-2f48c1\"" Nov 04 03:30:44.839287 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:30:44.839263 4405 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-2f48c1-7847czcwdv"] Nov 04 03:30:44.869579 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:30:44.869543 4405 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gnglr\" (UniqueName: \"kubernetes.io/projected/e5354816-e8fe-4941-91e0-ad2d7b46634b-kube-api-access-gnglr\") pod \"payload-pre-processing-e2e-worker-single-main-2f48c1-7847czcwdv\" (UID: \"e5354816-e8fe-4941-91e0-ad2d7b46634b\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-2f48c1-7847czcwdv" Nov 04 03:30:44.869767 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:30:44.869601 4405 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e5354816-e8fe-4941-91e0-ad2d7b46634b-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-single-main-2f48c1-7847czcwdv\" (UID: \"e5354816-e8fe-4941-91e0-ad2d7b46634b\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-2f48c1-7847czcwdv" Nov 04 03:30:44.870359 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:30:44.870339 4405 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e5354816-e8fe-4941-91e0-ad2d7b46634b-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-single-main-2f48c1-7847czcwdv\" (UID: \"e5354816-e8fe-4941-91e0-ad2d7b46634b\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-2f48c1-7847czcwdv" Nov 04 03:30:44.888548 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:30:44.888520 4405 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-2f48c1-5c55d7989qtlvn"] Nov 04 03:30:44.892096 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:30:44.892081 4405 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-2f48c1-5c55d7989qtlvn" Nov 04 03:30:44.930693 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:30:44.930624 4405 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gnglr\" (UniqueName: \"kubernetes.io/projected/e5354816-e8fe-4941-91e0-ad2d7b46634b-kube-api-access-gnglr\") pod \"payload-pre-processing-e2e-worker-single-main-2f48c1-7847czcwdv\" (UID: \"e5354816-e8fe-4941-91e0-ad2d7b46634b\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-2f48c1-7847czcwdv" Nov 04 03:30:44.933066 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:30:44.933037 4405 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-2f48c1-5c55d7989qtlvn"] Nov 04 03:30:44.970454 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:30:44.970411 4405 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ec1b611f-ce6b-4fb6-aa6f-4566fe349dfb-plugins-config-volume\") pod \"payload-processing-e2e-worker-single-main-2f48c1-5c55d7989qtlvn\" (UID: \"ec1b611f-ce6b-4fb6-aa6f-4566fe349dfb\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-2f48c1-5c55d7989qtlvn" Nov 04 03:30:44.970636 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:30:44.970467 4405 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6hxdp\" (UniqueName: \"kubernetes.io/projected/ec1b611f-ce6b-4fb6-aa6f-4566fe349dfb-kube-api-access-6hxdp\") pod \"payload-processing-e2e-worker-single-main-2f48c1-5c55d7989qtlvn\" (UID: \"ec1b611f-ce6b-4fb6-aa6f-4566fe349dfb\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-2f48c1-5c55d7989qtlvn" Nov 04 03:30:45.049296 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:30:45.049207 4405 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-2f48c1-7847czcwdv" Nov 04 03:30:45.071756 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:30:45.071718 4405 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ec1b611f-ce6b-4fb6-aa6f-4566fe349dfb-plugins-config-volume\") pod \"payload-processing-e2e-worker-single-main-2f48c1-5c55d7989qtlvn\" (UID: \"ec1b611f-ce6b-4fb6-aa6f-4566fe349dfb\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-2f48c1-5c55d7989qtlvn" Nov 04 03:30:45.072709 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:30:45.072657 4405 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6hxdp\" (UniqueName: \"kubernetes.io/projected/ec1b611f-ce6b-4fb6-aa6f-4566fe349dfb-kube-api-access-6hxdp\") pod \"payload-processing-e2e-worker-single-main-2f48c1-5c55d7989qtlvn\" (UID: \"ec1b611f-ce6b-4fb6-aa6f-4566fe349dfb\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-2f48c1-5c55d7989qtlvn" Nov 04 03:30:45.072880 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:30:45.072853 4405 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ec1b611f-ce6b-4fb6-aa6f-4566fe349dfb-plugins-config-volume\") pod \"payload-processing-e2e-worker-single-main-2f48c1-5c55d7989qtlvn\" (UID: \"ec1b611f-ce6b-4fb6-aa6f-4566fe349dfb\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-2f48c1-5c55d7989qtlvn" Nov 04 03:30:45.133992 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:30:45.133957 4405 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6hxdp\" (UniqueName: \"kubernetes.io/projected/ec1b611f-ce6b-4fb6-aa6f-4566fe349dfb-kube-api-access-6hxdp\") pod \"payload-processing-e2e-worker-single-main-2f48c1-5c55d7989qtlvn\" (UID: \"ec1b611f-ce6b-4fb6-aa6f-4566fe349dfb\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-2f48c1-5c55d7989qtlvn" Nov 04 03:30:45.205781 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:30:45.205743 4405 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-2f48c1-5c55d7989qtlvn" Nov 04 03:30:45.425515 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:30:45.425488 4405 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-2f48c1-7847czcwdv"] Nov 04 03:30:45.428139 ip-10-0-134-125 kubenswrapper[4405]: W1104 03:30:45.428113 4405 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode5354816_e8fe_4941_91e0_ad2d7b46634b.slice/crio-4a72f1e5db4ffc0d62492b94b3b1441626b24d256958942a54ad9939a91320ee WatchSource:0}: Error finding container 4a72f1e5db4ffc0d62492b94b3b1441626b24d256958942a54ad9939a91320ee: Status 404 returned error can't find the container with id 4a72f1e5db4ffc0d62492b94b3b1441626b24d256958942a54ad9939a91320ee Nov 04 03:30:45.579682 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:30:45.579641 4405 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-2f48c1-5c55d7989qtlvn"] Nov 04 03:30:45.581832 ip-10-0-134-125 kubenswrapper[4405]: W1104 03:30:45.581801 4405 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podec1b611f_ce6b_4fb6_aa6f_4566fe349dfb.slice/crio-b0b00a925c7ad5548f50b341527e415a897f8d634bd481eaad5778d5535b38e7 WatchSource:0}: Error finding container b0b00a925c7ad5548f50b341527e415a897f8d634bd481eaad5778d5535b38e7: Status 404 returned error can't find the container with id b0b00a925c7ad5548f50b341527e415a897f8d634bd481eaad5778d5535b38e7 Nov 04 03:30:46.071551 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:30:46.071512 4405 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-2f48c1-5c55d7989qtlvn" event={"ID":"ec1b611f-ce6b-4fb6-aa6f-4566fe349dfb","Type":"ContainerStarted","Data":"dae75e4aa0397da60be49b7d73608f8c56a271be3919093265a38ff71c90ed31"} Nov 04 03:30:46.072044 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:30:46.071572 4405 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-2f48c1-5c55d7989qtlvn" event={"ID":"ec1b611f-ce6b-4fb6-aa6f-4566fe349dfb","Type":"ContainerStarted","Data":"b0b00a925c7ad5548f50b341527e415a897f8d634bd481eaad5778d5535b38e7"} Nov 04 03:30:46.072044 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:30:46.071593 4405 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-single-main-2f48c1-5c55d7989qtlvn" Nov 04 03:30:46.073302 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:30:46.073279 4405 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-2f48c1-7847czcwdv" event={"ID":"e5354816-e8fe-4941-91e0-ad2d7b46634b","Type":"ContainerStarted","Data":"33544cd2f6b90a3ed608c3d26f7976a381e80ace69e89f10e77699257f0c53a2"} Nov 04 03:30:46.073382 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:30:46.073321 4405 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-2f48c1-7847czcwdv" event={"ID":"e5354816-e8fe-4941-91e0-ad2d7b46634b","Type":"ContainerStarted","Data":"4a72f1e5db4ffc0d62492b94b3b1441626b24d256958942a54ad9939a91320ee"} Nov 04 03:30:46.073382 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:30:46.073340 4405 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-2f48c1-7847czcwdv" Nov 04 03:30:46.188275 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:30:46.188224 4405 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-single-main-2f48c1-5c55d7989qtlvn" podStartSLOduration=2.188209003 podStartE2EDuration="2.188209003s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 03:30:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 03:30:46.187235831 +0000 UTC m=+3492.617935954" watchObservedRunningTime="2026-11-04 03:30:46.188209003 +0000 UTC m=+3492.618909130" Nov 04 03:30:46.302624 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:30:46.302566 4405 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-2f48c1-7847czcwdv" podStartSLOduration=2.30254867 podStartE2EDuration="2.30254867s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 03:30:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 03:30:46.301821382 +0000 UTC m=+3492.732521501" watchObservedRunningTime="2026-11-04 03:30:46.30254867 +0000 UTC m=+3492.733248791" Nov 04 03:30:47.233319 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:30:47.233285 4405 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-2f48c1-76dc9b7b9b-gtxhc"] Nov 04 03:30:47.236821 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:30:47.236804 4405 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-2f48c1-76dc9b7b9b-gtxhc" Nov 04 03:30:47.295503 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:30:47.295459 4405 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cea03d55-8344-4c8e-995c-b6ec148939cb-openshift-service-ca\") pod \"maas-api-e2e-worker-single-main-2f48c1-76dc9b7b9b-gtxhc\" (UID: \"cea03d55-8344-4c8e-995c-b6ec148939cb\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-2f48c1-76dc9b7b9b-gtxhc" Nov 04 03:30:47.295711 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:30:47.295598 4405 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2w99z\" (UniqueName: \"kubernetes.io/projected/cea03d55-8344-4c8e-995c-b6ec148939cb-kube-api-access-2w99z\") pod \"maas-api-e2e-worker-single-main-2f48c1-76dc9b7b9b-gtxhc\" (UID: \"cea03d55-8344-4c8e-995c-b6ec148939cb\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-2f48c1-76dc9b7b9b-gtxhc" Nov 04 03:30:47.295711 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:30:47.295704 4405 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cea03d55-8344-4c8e-995c-b6ec148939cb-maas-api-tls\") pod \"maas-api-e2e-worker-single-main-2f48c1-76dc9b7b9b-gtxhc\" (UID: \"cea03d55-8344-4c8e-995c-b6ec148939cb\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-2f48c1-76dc9b7b9b-gtxhc" Nov 04 03:30:47.295821 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:30:47.295765 4405 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cea03d55-8344-4c8e-995c-b6ec148939cb-metrics-certs\") pod \"maas-api-e2e-worker-single-main-2f48c1-76dc9b7b9b-gtxhc\" (UID: \"cea03d55-8344-4c8e-995c-b6ec148939cb\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-2f48c1-76dc9b7b9b-gtxhc" Nov 04 03:30:47.338496 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:30:47.338450 4405 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\"" Nov 04 03:30:47.338496 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:30:47.338469 4405 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-single-main-2f48c1-serving-cert\"" Nov 04 03:30:47.338496 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:30:47.338459 4405 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 04 03:30:47.338496 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:30:47.338480 4405 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 04 03:30:47.338848 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:30:47.338601 4405 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-qtjr2\"" Nov 04 03:30:47.340551 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:30:47.340528 4405 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-2f48c1-76dc9b7b9b-gtxhc"] Nov 04 03:30:47.396653 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:30:47.396611 4405 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2w99z\" (UniqueName: \"kubernetes.io/projected/cea03d55-8344-4c8e-995c-b6ec148939cb-kube-api-access-2w99z\") pod \"maas-api-e2e-worker-single-main-2f48c1-76dc9b7b9b-gtxhc\" (UID: \"cea03d55-8344-4c8e-995c-b6ec148939cb\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-2f48c1-76dc9b7b9b-gtxhc" Nov 04 03:30:47.396869 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:30:47.396693 4405 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cea03d55-8344-4c8e-995c-b6ec148939cb-maas-api-tls\") pod \"maas-api-e2e-worker-single-main-2f48c1-76dc9b7b9b-gtxhc\" (UID: \"cea03d55-8344-4c8e-995c-b6ec148939cb\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-2f48c1-76dc9b7b9b-gtxhc" Nov 04 03:30:47.396869 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:30:47.396719 4405 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cea03d55-8344-4c8e-995c-b6ec148939cb-metrics-certs\") pod \"maas-api-e2e-worker-single-main-2f48c1-76dc9b7b9b-gtxhc\" (UID: \"cea03d55-8344-4c8e-995c-b6ec148939cb\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-2f48c1-76dc9b7b9b-gtxhc" Nov 04 03:30:47.396869 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:30:47.396794 4405 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cea03d55-8344-4c8e-995c-b6ec148939cb-openshift-service-ca\") pod \"maas-api-e2e-worker-single-main-2f48c1-76dc9b7b9b-gtxhc\" (UID: \"cea03d55-8344-4c8e-995c-b6ec148939cb\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-2f48c1-76dc9b7b9b-gtxhc" Nov 04 03:30:47.396869 ip-10-0-134-125 kubenswrapper[4405]: E1104 03:30:47.396836 4405 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-worker-single-main-2f48c1-serving-cert: secret "maas-api-e2e-worker-single-main-2f48c1-serving-cert" not found Nov 04 03:30:47.397100 ip-10-0-134-125 kubenswrapper[4405]: E1104 03:30:47.396915 4405 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cea03d55-8344-4c8e-995c-b6ec148939cb-maas-api-tls podName:cea03d55-8344-4c8e-995c-b6ec148939cb nodeName:}" failed. No retries permitted until 2026-11-04 03:30:47.89689537 +0000 UTC m=+3494.327595485 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/cea03d55-8344-4c8e-995c-b6ec148939cb-maas-api-tls") pod "maas-api-e2e-worker-single-main-2f48c1-76dc9b7b9b-gtxhc" (UID: "cea03d55-8344-4c8e-995c-b6ec148939cb") : secret "maas-api-e2e-worker-single-main-2f48c1-serving-cert" not found Nov 04 03:30:47.397556 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:30:47.397532 4405 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cea03d55-8344-4c8e-995c-b6ec148939cb-openshift-service-ca\") pod \"maas-api-e2e-worker-single-main-2f48c1-76dc9b7b9b-gtxhc\" (UID: \"cea03d55-8344-4c8e-995c-b6ec148939cb\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-2f48c1-76dc9b7b9b-gtxhc" Nov 04 03:30:47.399713 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:30:47.399657 4405 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cea03d55-8344-4c8e-995c-b6ec148939cb-metrics-certs\") pod \"maas-api-e2e-worker-single-main-2f48c1-76dc9b7b9b-gtxhc\" (UID: \"cea03d55-8344-4c8e-995c-b6ec148939cb\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-2f48c1-76dc9b7b9b-gtxhc" Nov 04 03:30:47.457812 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:30:47.457771 4405 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2w99z\" (UniqueName: \"kubernetes.io/projected/cea03d55-8344-4c8e-995c-b6ec148939cb-kube-api-access-2w99z\") pod \"maas-api-e2e-worker-single-main-2f48c1-76dc9b7b9b-gtxhc\" (UID: \"cea03d55-8344-4c8e-995c-b6ec148939cb\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-2f48c1-76dc9b7b9b-gtxhc" Nov 04 03:30:47.902322 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:30:47.902290 4405 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cea03d55-8344-4c8e-995c-b6ec148939cb-maas-api-tls\") pod \"maas-api-e2e-worker-single-main-2f48c1-76dc9b7b9b-gtxhc\" (UID: \"cea03d55-8344-4c8e-995c-b6ec148939cb\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-2f48c1-76dc9b7b9b-gtxhc" Nov 04 03:30:47.904953 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:30:47.904927 4405 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cea03d55-8344-4c8e-995c-b6ec148939cb-maas-api-tls\") pod \"maas-api-e2e-worker-single-main-2f48c1-76dc9b7b9b-gtxhc\" (UID: \"cea03d55-8344-4c8e-995c-b6ec148939cb\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-2f48c1-76dc9b7b9b-gtxhc" Nov 04 03:30:48.150248 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:30:48.150218 4405 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-2f48c1-76dc9b7b9b-gtxhc" Nov 04 03:30:48.523633 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:30:48.523607 4405 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-2f48c1-76dc9b7b9b-gtxhc"] Nov 04 03:30:48.526295 ip-10-0-134-125 kubenswrapper[4405]: W1104 03:30:48.526265 4405 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcea03d55_8344_4c8e_995c_b6ec148939cb.slice/crio-f742fbcbf20ca93b9f84f3c7698096651c2815d47e764b9bdbdcdc7e5a082db1 WatchSource:0}: Error finding container f742fbcbf20ca93b9f84f3c7698096651c2815d47e764b9bdbdcdc7e5a082db1: Status 404 returned error can't find the container with id f742fbcbf20ca93b9f84f3c7698096651c2815d47e764b9bdbdcdc7e5a082db1 Nov 04 03:30:49.089266 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:30:49.089223 4405 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-2f48c1-76dc9b7b9b-gtxhc" event={"ID":"cea03d55-8344-4c8e-995c-b6ec148939cb","Type":"ContainerStarted","Data":"f742fbcbf20ca93b9f84f3c7698096651c2815d47e764b9bdbdcdc7e5a082db1"} Nov 04 03:30:50.097905 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:30:50.097859 4405 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-2f48c1-76dc9b7b9b-gtxhc" event={"ID":"cea03d55-8344-4c8e-995c-b6ec148939cb","Type":"ContainerStarted","Data":"b7dec893ba2f9a352a9b986cadd4678366e46f99d6a09779e37239d8fc737144"} Nov 04 03:30:50.098352 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:30:50.097978 4405 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-2f48c1-76dc9b7b9b-gtxhc" Nov 04 03:30:50.215895 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:30:50.215842 4405 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-2f48c1-76dc9b7b9b-gtxhc" podStartSLOduration=2.328408665 podStartE2EDuration="3.215828006s" totalImagesPullingTime="887.419341ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 03:30:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 03:30:50.214618033 +0000 UTC m=+3496.645318143" watchObservedRunningTime="2026-11-04 03:30:50.215828006 +0000 UTC m=+3496.646528125" Nov 04 03:30:56.108863 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:30:56.108832 4405 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-2f48c1-76dc9b7b9b-gtxhc" Nov 04 03:30:56.620426 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:30:56.620393 4405 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-bl5mg"] Nov 04 03:30:57.080475 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:30:57.080393 4405 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-single-main-2f48c1-5c55d7989qtlvn" Nov 04 03:30:57.080475 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:30:57.080458 4405 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-2f48c1-7847czcwdv" Nov 04 03:31:06.182034 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:31:06.182000 4405 generic.go:356] "Generic (PLEG): container finished" podID="ea05b8c6-9b14-480b-9544-a17bbd3a88d1" containerID="78a46d541a5f1fdb22df972dc0b20d13beed5ebbbb23ff9826c52316018c0fd1" exitCode=0 Nov 04 03:31:06.182531 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:31:06.182072 4405 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-2f48c1-openshift-default-6799d44489xnlnj" event={"ID":"ea05b8c6-9b14-480b-9544-a17bbd3a88d1","Type":"ContainerDied","Data":"78a46d541a5f1fdb22df972dc0b20d13beed5ebbbb23ff9826c52316018c0fd1"} Nov 04 03:31:06.182531 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:31:06.182441 4405 scope.go:122] "RemoveContainer" containerID="78a46d541a5f1fdb22df972dc0b20d13beed5ebbbb23ff9826c52316018c0fd1" Nov 04 03:31:06.183697 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:31:06.183646 4405 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 04 03:31:06.183813 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:31:06.183740 4405 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 04 03:31:06.183813 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:31:06.183774 4405 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 04 03:31:06.883891 ip-10-0-134-125 kubenswrapper[4405]: E1104 03:31:06.883854 4405 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5f2vb" containerName="extractor" Nov 04 03:31:07.218588 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:31:07.218374 4405 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-2f48c1-openshift-default-6799d44489xnlnj" event={"ID":"ea05b8c6-9b14-480b-9544-a17bbd3a88d1","Type":"ContainerStarted","Data":"a0f14ab502dad4f93135e547fc0e4ef92b850e6266bc37d0ffab979c07a6f9e9"} Nov 04 03:31:08.025933 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:31:08.025893 4405 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-single-main-2f48c1-openshift-default-6799d44489xnlnj" Nov 04 03:31:08.027203 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:31:08.027176 4405 patch_prober.go:29] interesting pod/e2e-worker-single-main-2f48c1-openshift-default-6799d44489xnlnj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.162:15021/healthz/ready\": dial tcp 10.134.0.162:15021: connect: connection refused" start-of-body= Nov 04 03:31:08.027333 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:31:08.027229 4405 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-2f48c1-openshift-default-6799d44489xnlnj" podUID="ea05b8c6-9b14-480b-9544-a17bbd3a88d1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.162:15021/healthz/ready\": dial tcp 10.134.0.162:15021: connect: connection refused" Nov 04 03:31:09.026482 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:31:09.026448 4405 patch_prober.go:29] interesting pod/e2e-worker-single-main-2f48c1-openshift-default-6799d44489xnlnj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.162:15021/healthz/ready\": dial tcp 10.134.0.162:15021: connect: connection refused" start-of-body= Nov 04 03:31:09.026980 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:31:09.026499 4405 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-2f48c1-openshift-default-6799d44489xnlnj" podUID="ea05b8c6-9b14-480b-9544-a17bbd3a88d1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.162:15021/healthz/ready\": dial tcp 10.134.0.162:15021: connect: connection refused" Nov 04 03:31:09.029587 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:31:09.029557 4405 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-single-main-2f48c1-openshift-default-6799d44489xnlnj" Nov 04 03:31:10.026131 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:31:10.026099 4405 patch_prober.go:29] interesting pod/e2e-worker-single-main-2f48c1-openshift-default-6799d44489xnlnj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.162:15021/healthz/ready\": dial tcp 10.134.0.162:15021: connect: connection refused" start-of-body= Nov 04 03:31:10.026342 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:31:10.026153 4405 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-2f48c1-openshift-default-6799d44489xnlnj" podUID="ea05b8c6-9b14-480b-9544-a17bbd3a88d1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.162:15021/healthz/ready\": dial tcp 10.134.0.162:15021: connect: connection refused" Nov 04 03:31:11.026803 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:31:11.026755 4405 patch_prober.go:29] interesting pod/e2e-worker-single-main-2f48c1-openshift-default-6799d44489xnlnj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.162:15021/healthz/ready\": dial tcp 10.134.0.162:15021: connect: connection refused" start-of-body= Nov 04 03:31:11.027321 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:31:11.026817 4405 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-2f48c1-openshift-default-6799d44489xnlnj" podUID="ea05b8c6-9b14-480b-9544-a17bbd3a88d1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.162:15021/healthz/ready\": dial tcp 10.134.0.162:15021: connect: connection refused" Nov 04 03:31:12.026419 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:31:12.026379 4405 patch_prober.go:29] interesting pod/e2e-worker-single-main-2f48c1-openshift-default-6799d44489xnlnj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.162:15021/healthz/ready\": dial tcp 10.134.0.162:15021: connect: connection refused" start-of-body= Nov 04 03:31:12.026768 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:31:12.026726 4405 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-2f48c1-openshift-default-6799d44489xnlnj" podUID="ea05b8c6-9b14-480b-9544-a17bbd3a88d1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.162:15021/healthz/ready\": dial tcp 10.134.0.162:15021: connect: connection refused" Nov 04 03:31:13.030951 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:31:13.030918 4405 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-single-main-2f48c1-openshift-default-6799d44489xnlnj" Nov 04 03:31:13.032009 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:31:13.031991 4405 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-single-main-2f48c1-openshift-default-6799d44489xnlnj" Nov 04 03:31:14.735318 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:31:14.735281 4405 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-bl5mg"] Nov 04 03:31:35.516142 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:31:35.516105 4405 scope.go:122] "RemoveContainer" containerID="ef907911ce26743bb2e840171c70805ef8023f11671f560a51d71bdb15696216" Nov 04 03:31:35.525333 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:31:35.525315 4405 scope.go:122] "RemoveContainer" containerID="d6daffbc70716eaea0881a6e0e1d4fd308f7688c9fb85ac8941c80de27f45624" Nov 04 03:31:36.886508 ip-10-0-134-125 kubenswrapper[4405]: E1104 03:31:36.886465 4405 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5f2vb" containerName="extractor" Nov 04 03:31:41.945074 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:31:41.945030 4405 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-bl5mg"] Nov 04 03:32:06.884463 ip-10-0-134-125 kubenswrapper[4405]: E1104 03:32:06.884418 4405 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5f2vb" containerName="extractor" Nov 04 03:32:07.717477 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:32:07.717445 4405 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-bl5mg"] Nov 04 03:32:11.954457 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:32:11.954417 4405 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-2f48c1-5c55d7989qtlvn"] Nov 04 03:32:12.066785 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:32:12.066752 4405 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-2f48c1-7847czcwdv"] Nov 04 03:32:15.433650 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:32:15.433607 4405 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-2f48c1-76dc9b7b9b-gtxhc"] Nov 04 03:32:15.434162 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:32:15.433899 4405 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-2f48c1-76dc9b7b9b-gtxhc" podUID="cea03d55-8344-4c8e-995c-b6ec148939cb" containerName="maas-api" containerID="cri-o://b7dec893ba2f9a352a9b986cadd4678366e46f99d6a09779e37239d8fc737144" gracePeriod=30 Nov 04 03:32:16.103429 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:32:16.103378 4405 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-2f48c1-76dc9b7b9b-gtxhc" podUID="cea03d55-8344-4c8e-995c-b6ec148939cb" containerName="maas-api" probeResult="failure" output="Get \"https://10.134.0.165:8443/health\": dial tcp 10.134.0.165:8443: connect: connection refused" Nov 04 03:32:16.564784 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:32:16.564749 4405 generic.go:356] "Generic (PLEG): container finished" podID="cea03d55-8344-4c8e-995c-b6ec148939cb" containerID="b7dec893ba2f9a352a9b986cadd4678366e46f99d6a09779e37239d8fc737144" exitCode=0 Nov 04 03:32:16.565264 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:32:16.564810 4405 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-2f48c1-76dc9b7b9b-gtxhc" event={"ID":"cea03d55-8344-4c8e-995c-b6ec148939cb","Type":"ContainerDied","Data":"b7dec893ba2f9a352a9b986cadd4678366e46f99d6a09779e37239d8fc737144"} Nov 04 03:32:16.695660 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:32:16.695632 4405 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-2f48c1-76dc9b7b9b-gtxhc" Nov 04 03:32:16.810410 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:32:16.810320 4405 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cea03d55-8344-4c8e-995c-b6ec148939cb-kube-api-access-2w99z\" (UniqueName: \"kubernetes.io/projected/cea03d55-8344-4c8e-995c-b6ec148939cb-kube-api-access-2w99z\") pod \"cea03d55-8344-4c8e-995c-b6ec148939cb\" (UID: \"cea03d55-8344-4c8e-995c-b6ec148939cb\") " Nov 04 03:32:16.810410 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:32:16.810378 4405 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/cea03d55-8344-4c8e-995c-b6ec148939cb-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cea03d55-8344-4c8e-995c-b6ec148939cb-maas-api-tls\") pod \"cea03d55-8344-4c8e-995c-b6ec148939cb\" (UID: \"cea03d55-8344-4c8e-995c-b6ec148939cb\") " Nov 04 03:32:16.810410 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:32:16.810410 4405 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/cea03d55-8344-4c8e-995c-b6ec148939cb-metrics-certs\" (UniqueName: \"kubernetes.io/secret/cea03d55-8344-4c8e-995c-b6ec148939cb-metrics-certs\") pod \"cea03d55-8344-4c8e-995c-b6ec148939cb\" (UID: \"cea03d55-8344-4c8e-995c-b6ec148939cb\") " Nov 04 03:32:16.810754 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:32:16.810475 4405 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cea03d55-8344-4c8e-995c-b6ec148939cb-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cea03d55-8344-4c8e-995c-b6ec148939cb-openshift-service-ca\") pod \"cea03d55-8344-4c8e-995c-b6ec148939cb\" (UID: \"cea03d55-8344-4c8e-995c-b6ec148939cb\") " Nov 04 03:32:16.810955 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:32:16.810930 4405 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cea03d55-8344-4c8e-995c-b6ec148939cb-openshift-service-ca" pod "cea03d55-8344-4c8e-995c-b6ec148939cb" (UID: "cea03d55-8344-4c8e-995c-b6ec148939cb"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 03:32:16.813082 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:32:16.813045 4405 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cea03d55-8344-4c8e-995c-b6ec148939cb-kube-api-access-2w99z" pod "cea03d55-8344-4c8e-995c-b6ec148939cb" (UID: "cea03d55-8344-4c8e-995c-b6ec148939cb"). InnerVolumeSpecName "kube-api-access-2w99z". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 03:32:16.813417 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:32:16.813396 4405 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cea03d55-8344-4c8e-995c-b6ec148939cb-metrics-certs" pod "cea03d55-8344-4c8e-995c-b6ec148939cb" (UID: "cea03d55-8344-4c8e-995c-b6ec148939cb"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 04 03:32:16.813417 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:32:16.813394 4405 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cea03d55-8344-4c8e-995c-b6ec148939cb-maas-api-tls" pod "cea03d55-8344-4c8e-995c-b6ec148939cb" (UID: "cea03d55-8344-4c8e-995c-b6ec148939cb"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 04 03:32:16.911766 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:32:16.911724 4405 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2w99z\" (UniqueName: \"kubernetes.io/projected/cea03d55-8344-4c8e-995c-b6ec148939cb-kube-api-access-2w99z\") on node \"ip-10-0-134-125.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:32:16.911766 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:32:16.911759 4405 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cea03d55-8344-4c8e-995c-b6ec148939cb-maas-api-tls\") on node \"ip-10-0-134-125.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:32:16.911766 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:32:16.911771 4405 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cea03d55-8344-4c8e-995c-b6ec148939cb-metrics-certs\") on node \"ip-10-0-134-125.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:32:16.912012 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:32:16.911782 4405 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cea03d55-8344-4c8e-995c-b6ec148939cb-openshift-service-ca\") on node \"ip-10-0-134-125.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:32:16.955544 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:32:16.955480 4405 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-single-main-2f48c1-5c55d7989qtlvn" podUID="ec1b611f-ce6b-4fb6-aa6f-4566fe349dfb" containerName="payload-processing" containerID="cri-o://dae75e4aa0397da60be49b7d73608f8c56a271be3919093265a38ff71c90ed31" gracePeriod=25 Nov 04 03:32:17.068181 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:32:17.068090 4405 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-2f48c1-7847czcwdv" podUID="e5354816-e8fe-4941-91e0-ad2d7b46634b" containerName="payload-pre-processing" containerID="cri-o://33544cd2f6b90a3ed608c3d26f7976a381e80ace69e89f10e77699257f0c53a2" gracePeriod=25 Nov 04 03:32:17.078724 ip-10-0-134-125 kubenswrapper[4405]: W1104 03:32:17.078699 4405 logging.go:55] [core] [Channel #4491 SubChannel #4493]grpc: addrConn.createTransport failed to connect to {Addr: "10.134.0.164:50052", ServerName: "10.134.0.164:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.134.0.164:50052: connect: connection refused" Nov 04 03:32:17.078841 ip-10-0-134-125 kubenswrapper[4405]: W1104 03:32:17.078699 4405 logging.go:55] [core] [Channel #4490 SubChannel #4492]grpc: addrConn.createTransport failed to connect to {Addr: "10.134.0.163:50052", ServerName: "10.134.0.163:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.134.0.163:50052: connect: connection refused" Nov 04 03:32:17.202346 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:32:17.202319 4405 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-2f48c1-5c55d7989qtlvn" Nov 04 03:32:17.314868 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:32:17.314838 4405 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-2f48c1-7847czcwdv" Nov 04 03:32:17.315868 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:32:17.315847 4405 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ec1b611f-ce6b-4fb6-aa6f-4566fe349dfb-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ec1b611f-ce6b-4fb6-aa6f-4566fe349dfb-plugins-config-volume\") pod \"ec1b611f-ce6b-4fb6-aa6f-4566fe349dfb\" (UID: \"ec1b611f-ce6b-4fb6-aa6f-4566fe349dfb\") " Nov 04 03:32:17.315983 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:32:17.315947 4405 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ec1b611f-ce6b-4fb6-aa6f-4566fe349dfb-kube-api-access-6hxdp\" (UniqueName: \"kubernetes.io/projected/ec1b611f-ce6b-4fb6-aa6f-4566fe349dfb-kube-api-access-6hxdp\") pod \"ec1b611f-ce6b-4fb6-aa6f-4566fe349dfb\" (UID: \"ec1b611f-ce6b-4fb6-aa6f-4566fe349dfb\") " Nov 04 03:32:17.316194 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:32:17.316174 4405 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ec1b611f-ce6b-4fb6-aa6f-4566fe349dfb-plugins-config-volume" pod "ec1b611f-ce6b-4fb6-aa6f-4566fe349dfb" (UID: "ec1b611f-ce6b-4fb6-aa6f-4566fe349dfb"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 03:32:17.316272 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:32:17.316260 4405 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ec1b611f-ce6b-4fb6-aa6f-4566fe349dfb-plugins-config-volume\") on node \"ip-10-0-134-125.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:32:17.318172 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:32:17.318152 4405 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ec1b611f-ce6b-4fb6-aa6f-4566fe349dfb-kube-api-access-6hxdp" pod "ec1b611f-ce6b-4fb6-aa6f-4566fe349dfb" (UID: "ec1b611f-ce6b-4fb6-aa6f-4566fe349dfb"). InnerVolumeSpecName "kube-api-access-6hxdp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 03:32:17.416913 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:32:17.416877 4405 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e5354816-e8fe-4941-91e0-ad2d7b46634b-kube-api-access-gnglr\" (UniqueName: \"kubernetes.io/projected/e5354816-e8fe-4941-91e0-ad2d7b46634b-kube-api-access-gnglr\") pod \"e5354816-e8fe-4941-91e0-ad2d7b46634b\" (UID: \"e5354816-e8fe-4941-91e0-ad2d7b46634b\") " Nov 04 03:32:17.417074 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:32:17.416939 4405 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e5354816-e8fe-4941-91e0-ad2d7b46634b-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e5354816-e8fe-4941-91e0-ad2d7b46634b-plugins-config-volume\") pod \"e5354816-e8fe-4941-91e0-ad2d7b46634b\" (UID: \"e5354816-e8fe-4941-91e0-ad2d7b46634b\") " Nov 04 03:32:17.417356 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:32:17.417333 4405 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6hxdp\" (UniqueName: \"kubernetes.io/projected/ec1b611f-ce6b-4fb6-aa6f-4566fe349dfb-kube-api-access-6hxdp\") on node \"ip-10-0-134-125.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:32:17.417395 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:32:17.417364 4405 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e5354816-e8fe-4941-91e0-ad2d7b46634b-plugins-config-volume" pod "e5354816-e8fe-4941-91e0-ad2d7b46634b" (UID: "e5354816-e8fe-4941-91e0-ad2d7b46634b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 03:32:17.419327 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:32:17.419301 4405 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e5354816-e8fe-4941-91e0-ad2d7b46634b-kube-api-access-gnglr" pod "e5354816-e8fe-4941-91e0-ad2d7b46634b" (UID: "e5354816-e8fe-4941-91e0-ad2d7b46634b"). InnerVolumeSpecName "kube-api-access-gnglr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 03:32:17.518596 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:32:17.518560 4405 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gnglr\" (UniqueName: \"kubernetes.io/projected/e5354816-e8fe-4941-91e0-ad2d7b46634b-kube-api-access-gnglr\") on node \"ip-10-0-134-125.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:32:17.518596 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:32:17.518593 4405 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e5354816-e8fe-4941-91e0-ad2d7b46634b-plugins-config-volume\") on node \"ip-10-0-134-125.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:32:17.571251 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:32:17.571213 4405 generic.go:356] "Generic (PLEG): container finished" podID="ec1b611f-ce6b-4fb6-aa6f-4566fe349dfb" containerID="dae75e4aa0397da60be49b7d73608f8c56a271be3919093265a38ff71c90ed31" exitCode=0 Nov 04 03:32:17.571730 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:32:17.571265 4405 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-2f48c1-5c55d7989qtlvn" Nov 04 03:32:17.571730 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:32:17.571296 4405 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-2f48c1-5c55d7989qtlvn" event={"ID":"ec1b611f-ce6b-4fb6-aa6f-4566fe349dfb","Type":"ContainerDied","Data":"dae75e4aa0397da60be49b7d73608f8c56a271be3919093265a38ff71c90ed31"} Nov 04 03:32:17.571730 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:32:17.571346 4405 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-2f48c1-5c55d7989qtlvn" event={"ID":"ec1b611f-ce6b-4fb6-aa6f-4566fe349dfb","Type":"ContainerDied","Data":"b0b00a925c7ad5548f50b341527e415a897f8d634bd481eaad5778d5535b38e7"} Nov 04 03:32:17.571730 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:32:17.571367 4405 scope.go:122] "RemoveContainer" containerID="dae75e4aa0397da60be49b7d73608f8c56a271be3919093265a38ff71c90ed31" Nov 04 03:32:17.572983 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:32:17.572962 4405 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-2f48c1-76dc9b7b9b-gtxhc" Nov 04 03:32:17.573082 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:32:17.572980 4405 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-2f48c1-76dc9b7b9b-gtxhc" event={"ID":"cea03d55-8344-4c8e-995c-b6ec148939cb","Type":"ContainerDied","Data":"f742fbcbf20ca93b9f84f3c7698096651c2815d47e764b9bdbdcdc7e5a082db1"} Nov 04 03:32:17.575797 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:32:17.575767 4405 generic.go:356] "Generic (PLEG): container finished" podID="e5354816-e8fe-4941-91e0-ad2d7b46634b" containerID="33544cd2f6b90a3ed608c3d26f7976a381e80ace69e89f10e77699257f0c53a2" exitCode=0 Nov 04 03:32:17.575934 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:32:17.575820 4405 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-2f48c1-7847czcwdv" event={"ID":"e5354816-e8fe-4941-91e0-ad2d7b46634b","Type":"ContainerDied","Data":"33544cd2f6b90a3ed608c3d26f7976a381e80ace69e89f10e77699257f0c53a2"} Nov 04 03:32:17.575934 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:32:17.575822 4405 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-2f48c1-7847czcwdv" Nov 04 03:32:17.575934 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:32:17.575859 4405 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-2f48c1-7847czcwdv" event={"ID":"e5354816-e8fe-4941-91e0-ad2d7b46634b","Type":"ContainerDied","Data":"4a72f1e5db4ffc0d62492b94b3b1441626b24d256958942a54ad9939a91320ee"} Nov 04 03:32:17.588523 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:32:17.588311 4405 scope.go:122] "RemoveContainer" containerID="dae75e4aa0397da60be49b7d73608f8c56a271be3919093265a38ff71c90ed31" Nov 04 03:32:17.588873 ip-10-0-134-125 kubenswrapper[4405]: E1104 03:32:17.588841 4405 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dae75e4aa0397da60be49b7d73608f8c56a271be3919093265a38ff71c90ed31\": container with ID starting with dae75e4aa0397da60be49b7d73608f8c56a271be3919093265a38ff71c90ed31 not found: ID does not exist" containerID="dae75e4aa0397da60be49b7d73608f8c56a271be3919093265a38ff71c90ed31" Nov 04 03:32:17.588961 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:32:17.588886 4405 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dae75e4aa0397da60be49b7d73608f8c56a271be3919093265a38ff71c90ed31"} err="failed to get container status \"dae75e4aa0397da60be49b7d73608f8c56a271be3919093265a38ff71c90ed31\": rpc error: code = NotFound desc = could not find container \"dae75e4aa0397da60be49b7d73608f8c56a271be3919093265a38ff71c90ed31\": container with ID starting with dae75e4aa0397da60be49b7d73608f8c56a271be3919093265a38ff71c90ed31 not found: ID does not exist" Nov 04 03:32:17.588961 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:32:17.588911 4405 scope.go:122] "RemoveContainer" containerID="b7dec893ba2f9a352a9b986cadd4678366e46f99d6a09779e37239d8fc737144" Nov 04 03:32:17.599100 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:32:17.599078 4405 scope.go:122] "RemoveContainer" containerID="33544cd2f6b90a3ed608c3d26f7976a381e80ace69e89f10e77699257f0c53a2" Nov 04 03:32:17.607989 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:32:17.607960 4405 scope.go:122] "RemoveContainer" containerID="33544cd2f6b90a3ed608c3d26f7976a381e80ace69e89f10e77699257f0c53a2" Nov 04 03:32:17.608348 ip-10-0-134-125 kubenswrapper[4405]: E1104 03:32:17.608326 4405 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"33544cd2f6b90a3ed608c3d26f7976a381e80ace69e89f10e77699257f0c53a2\": container with ID starting with 33544cd2f6b90a3ed608c3d26f7976a381e80ace69e89f10e77699257f0c53a2 not found: ID does not exist" containerID="33544cd2f6b90a3ed608c3d26f7976a381e80ace69e89f10e77699257f0c53a2" Nov 04 03:32:17.608433 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:32:17.608360 4405 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"33544cd2f6b90a3ed608c3d26f7976a381e80ace69e89f10e77699257f0c53a2"} err="failed to get container status \"33544cd2f6b90a3ed608c3d26f7976a381e80ace69e89f10e77699257f0c53a2\": rpc error: code = NotFound desc = could not find container \"33544cd2f6b90a3ed608c3d26f7976a381e80ace69e89f10e77699257f0c53a2\": container with ID starting with 33544cd2f6b90a3ed608c3d26f7976a381e80ace69e89f10e77699257f0c53a2 not found: ID does not exist" Nov 04 03:32:17.735853 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:32:17.735751 4405 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-2f48c1-76dc9b7b9b-gtxhc"] Nov 04 03:32:17.739056 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:32:17.739027 4405 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-2f48c1-76dc9b7b9b-gtxhc"] Nov 04 03:32:17.846968 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:32:17.846932 4405 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-2f48c1-7847czcwdv"] Nov 04 03:32:17.851038 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:32:17.851001 4405 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-2f48c1-7847czcwdv"] Nov 04 03:32:17.958918 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:32:17.958884 4405 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-2f48c1-5c55d7989qtlvn"] Nov 04 03:32:17.964252 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:32:17.964222 4405 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-2f48c1-5c55d7989qtlvn"] Nov 04 03:32:18.078924 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:32:18.078827 4405 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-single-main-2f48c1-5c55d7989qtlvn" podUID="ec1b611f-ce6b-4fb6-aa6f-4566fe349dfb" containerName="payload-processing" probeResult="failure" output="timeout: failed to connect service \"10.134.0.164:50052\" within 1s: context deadline exceeded" Nov 04 03:32:18.079111 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:32:18.078827 4405 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-2f48c1-7847czcwdv" podUID="e5354816-e8fe-4941-91e0-ad2d7b46634b" containerName="payload-pre-processing" probeResult="failure" output="timeout: failed to connect service \"10.134.0.163:50052\" within 1s: context deadline exceeded" Nov 04 03:32:19.700727 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:32:19.700690 4405 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cea03d55-8344-4c8e-995c-b6ec148939cb" path="/var/lib/kubelet/pods/cea03d55-8344-4c8e-995c-b6ec148939cb/volumes" Nov 04 03:32:19.701159 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:32:19.701071 4405 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e5354816-e8fe-4941-91e0-ad2d7b46634b" path="/var/lib/kubelet/pods/e5354816-e8fe-4941-91e0-ad2d7b46634b/volumes" Nov 04 03:32:19.701390 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:32:19.701376 4405 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ec1b611f-ce6b-4fb6-aa6f-4566fe349dfb" path="/var/lib/kubelet/pods/ec1b611f-ce6b-4fb6-aa6f-4566fe349dfb/volumes" Nov 04 03:32:20.938827 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:32:20.938790 4405 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-bl5mg"] Nov 04 03:32:34.243601 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:32:34.243569 4405 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-vwcx8_ffbb2d83-5a93-4a5b-bce3-e40afb8f58ea/ovn-acl-logging/0.log" Nov 04 03:32:34.261020 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:32:34.260993 4405 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-vwcx8_ffbb2d83-5a93-4a5b-bce3-e40afb8f58ea/ovn-acl-logging/0.log" Nov 04 03:32:35.550818 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:32:35.550783 4405 scope.go:122] "RemoveContainer" containerID="b4ddd9014b68b14b9f39769deb693e041106d61935a030006da2aa71f0314210" Nov 04 03:32:35.559403 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:32:35.559379 4405 scope.go:122] "RemoveContainer" containerID="076d7b5eb3cecc8cc14986c1ac08ebe2f7f6e7f537d598127f80887d0281a9dc" Nov 04 03:32:36.882954 ip-10-0-134-125 kubenswrapper[4405]: E1104 03:32:36.882918 4405 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5f2vb" containerName="extractor" Nov 04 03:32:49.472949 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:32:49.472913 4405 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-2f48c1-76dc9b7b9b-s9qdj"] Nov 04 03:32:49.473429 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:32:49.473320 4405 cpu_manager.go:414] "containerMap: removing container" podUID="ec1b611f-ce6b-4fb6-aa6f-4566fe349dfb" containerName="payload-processing" Nov 04 03:32:49.473429 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:32:49.473333 4405 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ec1b611f-ce6b-4fb6-aa6f-4566fe349dfb" containerName="payload-processing" Nov 04 03:32:49.473429 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:32:49.473344 4405 cpu_manager.go:414] "containerMap: removing container" podUID="e5354816-e8fe-4941-91e0-ad2d7b46634b" containerName="payload-pre-processing" Nov 04 03:32:49.473429 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:32:49.473350 4405 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e5354816-e8fe-4941-91e0-ad2d7b46634b" containerName="payload-pre-processing" Nov 04 03:32:49.473429 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:32:49.473367 4405 cpu_manager.go:414] "containerMap: removing container" podUID="cea03d55-8344-4c8e-995c-b6ec148939cb" containerName="maas-api" Nov 04 03:32:49.473429 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:32:49.473372 4405 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cea03d55-8344-4c8e-995c-b6ec148939cb" containerName="maas-api" Nov 04 03:32:49.473650 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:32:49.473440 4405 memory_manager.go:365] "RemoveStaleState removing state" podUID="cea03d55-8344-4c8e-995c-b6ec148939cb" containerName="maas-api" Nov 04 03:32:49.473650 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:32:49.473454 4405 memory_manager.go:365] "RemoveStaleState removing state" podUID="ec1b611f-ce6b-4fb6-aa6f-4566fe349dfb" containerName="payload-processing" Nov 04 03:32:49.473650 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:32:49.473460 4405 memory_manager.go:365] "RemoveStaleState removing state" podUID="e5354816-e8fe-4941-91e0-ad2d7b46634b" containerName="payload-pre-processing" Nov 04 03:32:49.476560 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:32:49.476534 4405 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-2f48c1-76dc9b7b9b-s9qdj" Nov 04 03:32:49.528465 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:32:49.528432 4405 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fbpbd\" (UniqueName: \"kubernetes.io/projected/ccd515f0-2917-4ba8-af70-485ef4415914-kube-api-access-fbpbd\") pod \"maas-api-e2e-worker-single-main-2f48c1-76dc9b7b9b-s9qdj\" (UID: \"ccd515f0-2917-4ba8-af70-485ef4415914\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-2f48c1-76dc9b7b9b-s9qdj" Nov 04 03:32:49.528645 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:32:49.528563 4405 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ccd515f0-2917-4ba8-af70-485ef4415914-metrics-certs\") pod \"maas-api-e2e-worker-single-main-2f48c1-76dc9b7b9b-s9qdj\" (UID: \"ccd515f0-2917-4ba8-af70-485ef4415914\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-2f48c1-76dc9b7b9b-s9qdj" Nov 04 03:32:49.528645 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:32:49.528591 4405 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ccd515f0-2917-4ba8-af70-485ef4415914-openshift-service-ca\") pod \"maas-api-e2e-worker-single-main-2f48c1-76dc9b7b9b-s9qdj\" (UID: \"ccd515f0-2917-4ba8-af70-485ef4415914\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-2f48c1-76dc9b7b9b-s9qdj" Nov 04 03:32:49.528755 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:32:49.528711 4405 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ccd515f0-2917-4ba8-af70-485ef4415914-maas-api-tls\") pod \"maas-api-e2e-worker-single-main-2f48c1-76dc9b7b9b-s9qdj\" (UID: \"ccd515f0-2917-4ba8-af70-485ef4415914\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-2f48c1-76dc9b7b9b-s9qdj" Nov 04 03:32:49.528952 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:32:49.528932 4405 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-single-main-2f48c1-serving-cert\"" Nov 04 03:32:49.529040 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:32:49.528944 4405 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 04 03:32:49.578736 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:32:49.578698 4405 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 04 03:32:49.578924 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:32:49.578741 4405 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-qtjr2\"" Nov 04 03:32:49.578924 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:32:49.578760 4405 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\"" Nov 04 03:32:49.580746 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:32:49.580724 4405 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-2f48c1-76dc9b7b9b-s9qdj"] Nov 04 03:32:49.629885 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:32:49.629855 4405 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fbpbd\" (UniqueName: \"kubernetes.io/projected/ccd515f0-2917-4ba8-af70-485ef4415914-kube-api-access-fbpbd\") pod \"maas-api-e2e-worker-single-main-2f48c1-76dc9b7b9b-s9qdj\" (UID: \"ccd515f0-2917-4ba8-af70-485ef4415914\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-2f48c1-76dc9b7b9b-s9qdj" Nov 04 03:32:49.630058 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:32:49.629958 4405 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ccd515f0-2917-4ba8-af70-485ef4415914-metrics-certs\") pod \"maas-api-e2e-worker-single-main-2f48c1-76dc9b7b9b-s9qdj\" (UID: \"ccd515f0-2917-4ba8-af70-485ef4415914\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-2f48c1-76dc9b7b9b-s9qdj" Nov 04 03:32:49.630058 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:32:49.629987 4405 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ccd515f0-2917-4ba8-af70-485ef4415914-openshift-service-ca\") pod \"maas-api-e2e-worker-single-main-2f48c1-76dc9b7b9b-s9qdj\" (UID: \"ccd515f0-2917-4ba8-af70-485ef4415914\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-2f48c1-76dc9b7b9b-s9qdj" Nov 04 03:32:49.630058 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:32:49.630016 4405 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ccd515f0-2917-4ba8-af70-485ef4415914-maas-api-tls\") pod \"maas-api-e2e-worker-single-main-2f48c1-76dc9b7b9b-s9qdj\" (UID: \"ccd515f0-2917-4ba8-af70-485ef4415914\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-2f48c1-76dc9b7b9b-s9qdj" Nov 04 03:32:49.630176 ip-10-0-134-125 kubenswrapper[4405]: E1104 03:32:49.630122 4405 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-worker-single-main-2f48c1-serving-cert: secret "maas-api-e2e-worker-single-main-2f48c1-serving-cert" not found Nov 04 03:32:49.630215 ip-10-0-134-125 kubenswrapper[4405]: E1104 03:32:49.630197 4405 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ccd515f0-2917-4ba8-af70-485ef4415914-maas-api-tls podName:ccd515f0-2917-4ba8-af70-485ef4415914 nodeName:}" failed. No retries permitted until 2026-11-04 03:32:50.130176796 +0000 UTC m=+3616.560876894 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/ccd515f0-2917-4ba8-af70-485ef4415914-maas-api-tls") pod "maas-api-e2e-worker-single-main-2f48c1-76dc9b7b9b-s9qdj" (UID: "ccd515f0-2917-4ba8-af70-485ef4415914") : secret "maas-api-e2e-worker-single-main-2f48c1-serving-cert" not found Nov 04 03:32:49.630580 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:32:49.630559 4405 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ccd515f0-2917-4ba8-af70-485ef4415914-openshift-service-ca\") pod \"maas-api-e2e-worker-single-main-2f48c1-76dc9b7b9b-s9qdj\" (UID: \"ccd515f0-2917-4ba8-af70-485ef4415914\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-2f48c1-76dc9b7b9b-s9qdj" Nov 04 03:32:49.632506 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:32:49.632483 4405 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ccd515f0-2917-4ba8-af70-485ef4415914-metrics-certs\") pod \"maas-api-e2e-worker-single-main-2f48c1-76dc9b7b9b-s9qdj\" (UID: \"ccd515f0-2917-4ba8-af70-485ef4415914\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-2f48c1-76dc9b7b9b-s9qdj" Nov 04 03:32:49.691239 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:32:49.691208 4405 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fbpbd\" (UniqueName: \"kubernetes.io/projected/ccd515f0-2917-4ba8-af70-485ef4415914-kube-api-access-fbpbd\") pod \"maas-api-e2e-worker-single-main-2f48c1-76dc9b7b9b-s9qdj\" (UID: \"ccd515f0-2917-4ba8-af70-485ef4415914\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-2f48c1-76dc9b7b9b-s9qdj" Nov 04 03:32:50.135066 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:32:50.135015 4405 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ccd515f0-2917-4ba8-af70-485ef4415914-maas-api-tls\") pod \"maas-api-e2e-worker-single-main-2f48c1-76dc9b7b9b-s9qdj\" (UID: \"ccd515f0-2917-4ba8-af70-485ef4415914\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-2f48c1-76dc9b7b9b-s9qdj" Nov 04 03:32:50.137565 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:32:50.137544 4405 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ccd515f0-2917-4ba8-af70-485ef4415914-maas-api-tls\") pod \"maas-api-e2e-worker-single-main-2f48c1-76dc9b7b9b-s9qdj\" (UID: \"ccd515f0-2917-4ba8-af70-485ef4415914\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-2f48c1-76dc9b7b9b-s9qdj" Nov 04 03:32:50.390211 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:32:50.390105 4405 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-2f48c1-76dc9b7b9b-s9qdj" Nov 04 03:32:50.772853 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:32:50.772825 4405 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-2f48c1-76dc9b7b9b-s9qdj"] Nov 04 03:32:50.775317 ip-10-0-134-125 kubenswrapper[4405]: W1104 03:32:50.775286 4405 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podccd515f0_2917_4ba8_af70_485ef4415914.slice/crio-34810bb623b0b9a3699f1455ec2ff78c5bc6ce98feb65fce023329bd3a431843 WatchSource:0}: Error finding container 34810bb623b0b9a3699f1455ec2ff78c5bc6ce98feb65fce023329bd3a431843: Status 404 returned error can't find the container with id 34810bb623b0b9a3699f1455ec2ff78c5bc6ce98feb65fce023329bd3a431843 Nov 04 03:32:50.776930 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:32:50.776909 4405 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 04 03:32:51.736967 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:32:51.736932 4405 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-2f48c1-76dc9b7b9b-s9qdj" event={"ID":"ccd515f0-2917-4ba8-af70-485ef4415914","Type":"ContainerStarted","Data":"34810bb623b0b9a3699f1455ec2ff78c5bc6ce98feb65fce023329bd3a431843"} Nov 04 03:32:52.743102 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:32:52.743067 4405 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-2f48c1-76dc9b7b9b-s9qdj" event={"ID":"ccd515f0-2917-4ba8-af70-485ef4415914","Type":"ContainerStarted","Data":"9ee16b7fde546df2a6a585bd6b491b77e17799d071d5b0bbab0d579d7480f5e1"} Nov 04 03:32:52.743533 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:32:52.743141 4405 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-2f48c1-76dc9b7b9b-s9qdj" Nov 04 03:32:52.860205 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:32:52.860142 4405 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-2f48c1-76dc9b7b9b-s9qdj" podStartSLOduration=2.873508215 podStartE2EDuration="3.860124589s" totalImagesPullingTime="986.616374ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 03:32:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 03:32:52.860063251 +0000 UTC m=+3619.290763372" watchObservedRunningTime="2026-11-04 03:32:52.860124589 +0000 UTC m=+3619.290824710" Nov 04 03:32:58.519742 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:32:58.519698 4405 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-bl5mg"] Nov 04 03:32:58.753993 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:32:58.753959 4405 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-2f48c1-76dc9b7b9b-s9qdj" Nov 04 03:33:06.883078 ip-10-0-134-125 kubenswrapper[4405]: E1104 03:33:06.883031 4405 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5f2vb" containerName="extractor" Nov 04 03:33:15.047767 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:33:15.047727 4405 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-9ktbr"] Nov 04 03:33:15.069903 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:33:15.069874 4405 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-9ktbr"] Nov 04 03:33:15.699857 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:33:15.699826 4405 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5f397f15-3135-4cee-9d3a-eb62a71e6d9e" path="/var/lib/kubelet/pods/5f397f15-3135-4cee-9d3a-eb62a71e6d9e/volumes" Nov 04 03:33:18.358962 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:33:18.358923 4405 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-2f48c1-76dc9b7b9b-s9qdj"] Nov 04 03:33:18.359449 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:33:18.359157 4405 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-2f48c1-76dc9b7b9b-s9qdj" podUID="ccd515f0-2917-4ba8-af70-485ef4415914" containerName="maas-api" containerID="cri-o://9ee16b7fde546df2a6a585bd6b491b77e17799d071d5b0bbab0d579d7480f5e1" gracePeriod=30 Nov 04 03:33:18.619330 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:33:18.619267 4405 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-2f48c1-76dc9b7b9b-s9qdj" Nov 04 03:33:18.704824 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:33:18.704788 4405 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ccd515f0-2917-4ba8-af70-485ef4415914-kube-api-access-fbpbd\" (UniqueName: \"kubernetes.io/projected/ccd515f0-2917-4ba8-af70-485ef4415914-kube-api-access-fbpbd\") pod \"ccd515f0-2917-4ba8-af70-485ef4415914\" (UID: \"ccd515f0-2917-4ba8-af70-485ef4415914\") " Nov 04 03:33:18.705025 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:33:18.704857 4405 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ccd515f0-2917-4ba8-af70-485ef4415914-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ccd515f0-2917-4ba8-af70-485ef4415914-maas-api-tls\") pod \"ccd515f0-2917-4ba8-af70-485ef4415914\" (UID: \"ccd515f0-2917-4ba8-af70-485ef4415914\") " Nov 04 03:33:18.705025 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:33:18.704889 4405 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ccd515f0-2917-4ba8-af70-485ef4415914-metrics-certs\" (UniqueName: \"kubernetes.io/secret/ccd515f0-2917-4ba8-af70-485ef4415914-metrics-certs\") pod \"ccd515f0-2917-4ba8-af70-485ef4415914\" (UID: \"ccd515f0-2917-4ba8-af70-485ef4415914\") " Nov 04 03:33:18.705025 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:33:18.704933 4405 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ccd515f0-2917-4ba8-af70-485ef4415914-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ccd515f0-2917-4ba8-af70-485ef4415914-openshift-service-ca\") pod \"ccd515f0-2917-4ba8-af70-485ef4415914\" (UID: \"ccd515f0-2917-4ba8-af70-485ef4415914\") " Nov 04 03:33:18.705394 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:33:18.705363 4405 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ccd515f0-2917-4ba8-af70-485ef4415914-openshift-service-ca" pod "ccd515f0-2917-4ba8-af70-485ef4415914" (UID: "ccd515f0-2917-4ba8-af70-485ef4415914"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 03:33:18.707157 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:33:18.707126 4405 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ccd515f0-2917-4ba8-af70-485ef4415914-metrics-certs" pod "ccd515f0-2917-4ba8-af70-485ef4415914" (UID: "ccd515f0-2917-4ba8-af70-485ef4415914"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 04 03:33:18.707252 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:33:18.707168 4405 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ccd515f0-2917-4ba8-af70-485ef4415914-maas-api-tls" pod "ccd515f0-2917-4ba8-af70-485ef4415914" (UID: "ccd515f0-2917-4ba8-af70-485ef4415914"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 04 03:33:18.707252 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:33:18.707225 4405 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ccd515f0-2917-4ba8-af70-485ef4415914-kube-api-access-fbpbd" pod "ccd515f0-2917-4ba8-af70-485ef4415914" (UID: "ccd515f0-2917-4ba8-af70-485ef4415914"). InnerVolumeSpecName "kube-api-access-fbpbd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 03:33:18.812718 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:33:18.806846 4405 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ccd515f0-2917-4ba8-af70-485ef4415914-maas-api-tls\") on node \"ip-10-0-134-125.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:33:18.812718 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:33:18.806895 4405 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ccd515f0-2917-4ba8-af70-485ef4415914-metrics-certs\") on node \"ip-10-0-134-125.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:33:18.812718 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:33:18.806911 4405 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ccd515f0-2917-4ba8-af70-485ef4415914-openshift-service-ca\") on node \"ip-10-0-134-125.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:33:18.812718 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:33:18.806925 4405 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fbpbd\" (UniqueName: \"kubernetes.io/projected/ccd515f0-2917-4ba8-af70-485ef4415914-kube-api-access-fbpbd\") on node \"ip-10-0-134-125.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:33:18.864885 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:33:18.864814 4405 generic.go:356] "Generic (PLEG): container finished" podID="ccd515f0-2917-4ba8-af70-485ef4415914" containerID="9ee16b7fde546df2a6a585bd6b491b77e17799d071d5b0bbab0d579d7480f5e1" exitCode=0 Nov 04 03:33:18.865169 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:33:18.865004 4405 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-2f48c1-76dc9b7b9b-s9qdj" event={"ID":"ccd515f0-2917-4ba8-af70-485ef4415914","Type":"ContainerDied","Data":"9ee16b7fde546df2a6a585bd6b491b77e17799d071d5b0bbab0d579d7480f5e1"} Nov 04 03:33:18.865169 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:33:18.865095 4405 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-2f48c1-76dc9b7b9b-s9qdj" event={"ID":"ccd515f0-2917-4ba8-af70-485ef4415914","Type":"ContainerDied","Data":"34810bb623b0b9a3699f1455ec2ff78c5bc6ce98feb65fce023329bd3a431843"} Nov 04 03:33:18.865169 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:33:18.865132 4405 scope.go:122] "RemoveContainer" containerID="9ee16b7fde546df2a6a585bd6b491b77e17799d071d5b0bbab0d579d7480f5e1" Nov 04 03:33:18.865503 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:33:18.865482 4405 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-2f48c1-76dc9b7b9b-s9qdj" Nov 04 03:33:18.882452 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:33:18.882428 4405 scope.go:122] "RemoveContainer" containerID="9ee16b7fde546df2a6a585bd6b491b77e17799d071d5b0bbab0d579d7480f5e1" Nov 04 03:33:18.882782 ip-10-0-134-125 kubenswrapper[4405]: E1104 03:33:18.882757 4405 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9ee16b7fde546df2a6a585bd6b491b77e17799d071d5b0bbab0d579d7480f5e1\": container with ID starting with 9ee16b7fde546df2a6a585bd6b491b77e17799d071d5b0bbab0d579d7480f5e1 not found: ID does not exist" containerID="9ee16b7fde546df2a6a585bd6b491b77e17799d071d5b0bbab0d579d7480f5e1" Nov 04 03:33:18.882842 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:33:18.882794 4405 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9ee16b7fde546df2a6a585bd6b491b77e17799d071d5b0bbab0d579d7480f5e1"} err="failed to get container status \"9ee16b7fde546df2a6a585bd6b491b77e17799d071d5b0bbab0d579d7480f5e1\": rpc error: code = NotFound desc = could not find container \"9ee16b7fde546df2a6a585bd6b491b77e17799d071d5b0bbab0d579d7480f5e1\": container with ID starting with 9ee16b7fde546df2a6a585bd6b491b77e17799d071d5b0bbab0d579d7480f5e1 not found: ID does not exist" Nov 04 03:33:19.034345 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:33:19.034308 4405 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-2f48c1-76dc9b7b9b-s9qdj"] Nov 04 03:33:19.037819 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:33:19.037793 4405 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-2f48c1-76dc9b7b9b-s9qdj"] Nov 04 03:33:19.701546 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:33:19.701511 4405 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ccd515f0-2917-4ba8-af70-485ef4415914" path="/var/lib/kubelet/pods/ccd515f0-2917-4ba8-af70-485ef4415914/volumes" Nov 04 03:33:20.921120 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:33:20.921084 4405 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-single-main-2f48c1-openshift-default-6799d44489xnlnj"] Nov 04 03:33:20.921566 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:33:20.921327 4405 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-single-main-2f48c1-openshift-default-6799d44489xnlnj" podUID="ea05b8c6-9b14-480b-9544-a17bbd3a88d1" containerName="istio-proxy" containerID="cri-o://a0f14ab502dad4f93135e547fc0e4ef92b850e6266bc37d0ffab979c07a6f9e9" gracePeriod=30 Nov 04 03:33:24.740834 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:33:24.740792 4405 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-bl5mg"] Nov 04 03:33:26.171549 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:33:26.171527 4405 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-single-main-2f48c1-openshift-default-6799d44489xnlnj" Nov 04 03:33:26.276596 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:33:26.276504 4405 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/ea05b8c6-9b14-480b-9544-a17bbd3a88d1-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ea05b8c6-9b14-480b-9544-a17bbd3a88d1-istio-podinfo\") pod \"ea05b8c6-9b14-480b-9544-a17bbd3a88d1\" (UID: \"ea05b8c6-9b14-480b-9544-a17bbd3a88d1\") " Nov 04 03:33:26.276795 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:33:26.276596 4405 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ea05b8c6-9b14-480b-9544-a17bbd3a88d1-istio-token\" (UniqueName: \"kubernetes.io/projected/ea05b8c6-9b14-480b-9544-a17bbd3a88d1-istio-token\") pod \"ea05b8c6-9b14-480b-9544-a17bbd3a88d1\" (UID: \"ea05b8c6-9b14-480b-9544-a17bbd3a88d1\") " Nov 04 03:33:26.276795 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:33:26.276651 4405 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ea05b8c6-9b14-480b-9544-a17bbd3a88d1-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ea05b8c6-9b14-480b-9544-a17bbd3a88d1-istio-envoy\") pod \"ea05b8c6-9b14-480b-9544-a17bbd3a88d1\" (UID: \"ea05b8c6-9b14-480b-9544-a17bbd3a88d1\") " Nov 04 03:33:26.276795 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:33:26.276697 4405 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ea05b8c6-9b14-480b-9544-a17bbd3a88d1-kube-api-access-wzhxh\" (UniqueName: \"kubernetes.io/projected/ea05b8c6-9b14-480b-9544-a17bbd3a88d1-kube-api-access-wzhxh\") pod \"ea05b8c6-9b14-480b-9544-a17bbd3a88d1\" (UID: \"ea05b8c6-9b14-480b-9544-a17bbd3a88d1\") " Nov 04 03:33:26.276795 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:33:26.276729 4405 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ea05b8c6-9b14-480b-9544-a17bbd3a88d1-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ea05b8c6-9b14-480b-9544-a17bbd3a88d1-workload-socket\") pod \"ea05b8c6-9b14-480b-9544-a17bbd3a88d1\" (UID: \"ea05b8c6-9b14-480b-9544-a17bbd3a88d1\") " Nov 04 03:33:26.276795 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:33:26.276765 4405 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ea05b8c6-9b14-480b-9544-a17bbd3a88d1-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ea05b8c6-9b14-480b-9544-a17bbd3a88d1-credential-socket\") pod \"ea05b8c6-9b14-480b-9544-a17bbd3a88d1\" (UID: \"ea05b8c6-9b14-480b-9544-a17bbd3a88d1\") " Nov 04 03:33:26.277062 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:33:26.276796 4405 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ea05b8c6-9b14-480b-9544-a17bbd3a88d1-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ea05b8c6-9b14-480b-9544-a17bbd3a88d1-istiod-ca-cert\") pod \"ea05b8c6-9b14-480b-9544-a17bbd3a88d1\" (UID: \"ea05b8c6-9b14-480b-9544-a17bbd3a88d1\") " Nov 04 03:33:26.277062 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:33:26.276826 4405 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ea05b8c6-9b14-480b-9544-a17bbd3a88d1-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ea05b8c6-9b14-480b-9544-a17bbd3a88d1-istio-data\") pod \"ea05b8c6-9b14-480b-9544-a17bbd3a88d1\" (UID: \"ea05b8c6-9b14-480b-9544-a17bbd3a88d1\") " Nov 04 03:33:26.277062 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:33:26.276923 4405 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ea05b8c6-9b14-480b-9544-a17bbd3a88d1-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ea05b8c6-9b14-480b-9544-a17bbd3a88d1-workload-certs\") pod \"ea05b8c6-9b14-480b-9544-a17bbd3a88d1\" (UID: \"ea05b8c6-9b14-480b-9544-a17bbd3a88d1\") " Nov 04 03:33:26.277209 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:33:26.277086 4405 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ea05b8c6-9b14-480b-9544-a17bbd3a88d1-workload-socket" pod "ea05b8c6-9b14-480b-9544-a17bbd3a88d1" (UID: "ea05b8c6-9b14-480b-9544-a17bbd3a88d1"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 03:33:26.277264 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:33:26.277246 4405 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ea05b8c6-9b14-480b-9544-a17bbd3a88d1-workload-socket\") on node \"ip-10-0-134-125.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:33:26.277366 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:33:26.277337 4405 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ea05b8c6-9b14-480b-9544-a17bbd3a88d1-istiod-ca-cert" pod "ea05b8c6-9b14-480b-9544-a17bbd3a88d1" (UID: "ea05b8c6-9b14-480b-9544-a17bbd3a88d1"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 03:33:26.277441 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:33:26.277359 4405 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ea05b8c6-9b14-480b-9544-a17bbd3a88d1-credential-socket" pod "ea05b8c6-9b14-480b-9544-a17bbd3a88d1" (UID: "ea05b8c6-9b14-480b-9544-a17bbd3a88d1"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 03:33:26.277693 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:33:26.277594 4405 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ea05b8c6-9b14-480b-9544-a17bbd3a88d1-workload-certs" pod "ea05b8c6-9b14-480b-9544-a17bbd3a88d1" (UID: "ea05b8c6-9b14-480b-9544-a17bbd3a88d1"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 03:33:26.277693 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:33:26.277631 4405 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ea05b8c6-9b14-480b-9544-a17bbd3a88d1-istio-data" pod "ea05b8c6-9b14-480b-9544-a17bbd3a88d1" (UID: "ea05b8c6-9b14-480b-9544-a17bbd3a88d1"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 03:33:26.279140 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:33:26.279111 4405 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ea05b8c6-9b14-480b-9544-a17bbd3a88d1-istio-podinfo" pod "ea05b8c6-9b14-480b-9544-a17bbd3a88d1" (UID: "ea05b8c6-9b14-480b-9544-a17bbd3a88d1"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 04 03:33:26.279267 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:33:26.279192 4405 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ea05b8c6-9b14-480b-9544-a17bbd3a88d1-kube-api-access-wzhxh" pod "ea05b8c6-9b14-480b-9544-a17bbd3a88d1" (UID: "ea05b8c6-9b14-480b-9544-a17bbd3a88d1"). InnerVolumeSpecName "kube-api-access-wzhxh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 03:33:26.279408 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:33:26.279389 4405 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ea05b8c6-9b14-480b-9544-a17bbd3a88d1-istio-envoy" pod "ea05b8c6-9b14-480b-9544-a17bbd3a88d1" (UID: "ea05b8c6-9b14-480b-9544-a17bbd3a88d1"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 03:33:26.279519 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:33:26.279427 4405 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ea05b8c6-9b14-480b-9544-a17bbd3a88d1-istio-token" pod "ea05b8c6-9b14-480b-9544-a17bbd3a88d1" (UID: "ea05b8c6-9b14-480b-9544-a17bbd3a88d1"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 03:33:26.378185 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:33:26.378150 4405 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ea05b8c6-9b14-480b-9544-a17bbd3a88d1-istio-token\") on node \"ip-10-0-134-125.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:33:26.378185 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:33:26.378178 4405 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ea05b8c6-9b14-480b-9544-a17bbd3a88d1-istio-envoy\") on node \"ip-10-0-134-125.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:33:26.378185 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:33:26.378188 4405 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wzhxh\" (UniqueName: \"kubernetes.io/projected/ea05b8c6-9b14-480b-9544-a17bbd3a88d1-kube-api-access-wzhxh\") on node \"ip-10-0-134-125.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:33:26.378430 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:33:26.378197 4405 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ea05b8c6-9b14-480b-9544-a17bbd3a88d1-credential-socket\") on node \"ip-10-0-134-125.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:33:26.378430 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:33:26.378207 4405 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ea05b8c6-9b14-480b-9544-a17bbd3a88d1-istiod-ca-cert\") on node \"ip-10-0-134-125.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:33:26.378430 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:33:26.378215 4405 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ea05b8c6-9b14-480b-9544-a17bbd3a88d1-istio-data\") on node \"ip-10-0-134-125.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:33:26.378430 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:33:26.378225 4405 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ea05b8c6-9b14-480b-9544-a17bbd3a88d1-workload-certs\") on node \"ip-10-0-134-125.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:33:26.378430 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:33:26.378235 4405 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ea05b8c6-9b14-480b-9544-a17bbd3a88d1-istio-podinfo\") on node \"ip-10-0-134-125.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:33:26.907933 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:33:26.907897 4405 generic.go:356] "Generic (PLEG): container finished" podID="ea05b8c6-9b14-480b-9544-a17bbd3a88d1" containerID="a0f14ab502dad4f93135e547fc0e4ef92b850e6266bc37d0ffab979c07a6f9e9" exitCode=0 Nov 04 03:33:26.908096 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:33:26.907955 4405 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-single-main-2f48c1-openshift-default-6799d44489xnlnj" Nov 04 03:33:26.908096 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:33:26.907985 4405 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-2f48c1-openshift-default-6799d44489xnlnj" event={"ID":"ea05b8c6-9b14-480b-9544-a17bbd3a88d1","Type":"ContainerDied","Data":"a0f14ab502dad4f93135e547fc0e4ef92b850e6266bc37d0ffab979c07a6f9e9"} Nov 04 03:33:26.908096 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:33:26.908044 4405 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-2f48c1-openshift-default-6799d44489xnlnj" event={"ID":"ea05b8c6-9b14-480b-9544-a17bbd3a88d1","Type":"ContainerDied","Data":"08e53ebdd80ad474a77e7d50be23172f146c3e846a7e5c5919ca11f5cf5c79d2"} Nov 04 03:33:26.908096 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:33:26.908074 4405 scope.go:122] "RemoveContainer" containerID="a0f14ab502dad4f93135e547fc0e4ef92b850e6266bc37d0ffab979c07a6f9e9" Nov 04 03:33:26.917229 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:33:26.917212 4405 scope.go:122] "RemoveContainer" containerID="78a46d541a5f1fdb22df972dc0b20d13beed5ebbbb23ff9826c52316018c0fd1" Nov 04 03:33:26.925720 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:33:26.925700 4405 scope.go:122] "RemoveContainer" containerID="a0f14ab502dad4f93135e547fc0e4ef92b850e6266bc37d0ffab979c07a6f9e9" Nov 04 03:33:26.925999 ip-10-0-134-125 kubenswrapper[4405]: E1104 03:33:26.925967 4405 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a0f14ab502dad4f93135e547fc0e4ef92b850e6266bc37d0ffab979c07a6f9e9\": container with ID starting with a0f14ab502dad4f93135e547fc0e4ef92b850e6266bc37d0ffab979c07a6f9e9 not found: ID does not exist" containerID="a0f14ab502dad4f93135e547fc0e4ef92b850e6266bc37d0ffab979c07a6f9e9" Nov 04 03:33:26.926095 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:33:26.926011 4405 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a0f14ab502dad4f93135e547fc0e4ef92b850e6266bc37d0ffab979c07a6f9e9"} err="failed to get container status \"a0f14ab502dad4f93135e547fc0e4ef92b850e6266bc37d0ffab979c07a6f9e9\": rpc error: code = NotFound desc = could not find container \"a0f14ab502dad4f93135e547fc0e4ef92b850e6266bc37d0ffab979c07a6f9e9\": container with ID starting with a0f14ab502dad4f93135e547fc0e4ef92b850e6266bc37d0ffab979c07a6f9e9 not found: ID does not exist" Nov 04 03:33:26.926095 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:33:26.926035 4405 scope.go:122] "RemoveContainer" containerID="78a46d541a5f1fdb22df972dc0b20d13beed5ebbbb23ff9826c52316018c0fd1" Nov 04 03:33:26.926363 ip-10-0-134-125 kubenswrapper[4405]: E1104 03:33:26.926343 4405 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"78a46d541a5f1fdb22df972dc0b20d13beed5ebbbb23ff9826c52316018c0fd1\": container with ID starting with 78a46d541a5f1fdb22df972dc0b20d13beed5ebbbb23ff9826c52316018c0fd1 not found: ID does not exist" containerID="78a46d541a5f1fdb22df972dc0b20d13beed5ebbbb23ff9826c52316018c0fd1" Nov 04 03:33:26.926438 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:33:26.926366 4405 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"78a46d541a5f1fdb22df972dc0b20d13beed5ebbbb23ff9826c52316018c0fd1"} err="failed to get container status \"78a46d541a5f1fdb22df972dc0b20d13beed5ebbbb23ff9826c52316018c0fd1\": rpc error: code = NotFound desc = could not find container \"78a46d541a5f1fdb22df972dc0b20d13beed5ebbbb23ff9826c52316018c0fd1\": container with ID starting with 78a46d541a5f1fdb22df972dc0b20d13beed5ebbbb23ff9826c52316018c0fd1 not found: ID does not exist" Nov 04 03:33:27.122350 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:33:27.122318 4405 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-single-main-2f48c1-openshift-default-6799d44489xnlnj"] Nov 04 03:33:27.125942 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:33:27.125920 4405 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-single-main-2f48c1-openshift-default-6799d44489xnlnj"] Nov 04 03:33:27.699761 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:33:27.699726 4405 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ea05b8c6-9b14-480b-9544-a17bbd3a88d1" path="/var/lib/kubelet/pods/ea05b8c6-9b14-480b-9544-a17bbd3a88d1/volumes" Nov 04 03:33:28.072290 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:33:28.072214 4405 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-same-id-a-b05dd916-kserve-5b4cdf54d9-ls2cr"] Nov 04 03:33:28.072587 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:33:28.072575 4405 cpu_manager.go:414] "containerMap: removing container" podUID="ea05b8c6-9b14-480b-9544-a17bbd3a88d1" containerName="istio-proxy" Nov 04 03:33:28.072644 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:33:28.072589 4405 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ea05b8c6-9b14-480b-9544-a17bbd3a88d1" containerName="istio-proxy" Nov 04 03:33:28.072644 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:33:28.072611 4405 cpu_manager.go:414] "containerMap: removing container" podUID="ea05b8c6-9b14-480b-9544-a17bbd3a88d1" containerName="istio-proxy" Nov 04 03:33:28.072644 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:33:28.072617 4405 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ea05b8c6-9b14-480b-9544-a17bbd3a88d1" containerName="istio-proxy" Nov 04 03:33:28.072644 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:33:28.072635 4405 cpu_manager.go:414] "containerMap: removing container" podUID="ccd515f0-2917-4ba8-af70-485ef4415914" containerName="maas-api" Nov 04 03:33:28.072644 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:33:28.072640 4405 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ccd515f0-2917-4ba8-af70-485ef4415914" containerName="maas-api" Nov 04 03:33:28.072850 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:33:28.072715 4405 memory_manager.go:365] "RemoveStaleState removing state" podUID="ccd515f0-2917-4ba8-af70-485ef4415914" containerName="maas-api" Nov 04 03:33:28.072850 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:33:28.072730 4405 memory_manager.go:365] "RemoveStaleState removing state" podUID="ea05b8c6-9b14-480b-9544-a17bbd3a88d1" containerName="istio-proxy" Nov 04 03:33:28.072850 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:33:28.072736 4405 memory_manager.go:365] "RemoveStaleState removing state" podUID="ea05b8c6-9b14-480b-9544-a17bbd3a88d1" containerName="istio-proxy" Nov 04 03:33:28.077353 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:33:28.077328 4405 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-b05dd916-kserve-5b4cdf54d9-ls2cr" Nov 04 03:33:28.157624 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:33:28.157593 4405 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-a-b05dd916-kserve-5b4cdf54d9-ls2cr"] Nov 04 03:33:28.181141 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:33:28.181113 4405 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-same-id-a-b05dd916-kserve-self-signed-certs\"" Nov 04 03:33:28.194140 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:33:28.194112 4405 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-47blz\" (UniqueName: \"kubernetes.io/projected/a0ae7796-3bbd-4eff-89c0-4e188ddd23a4-kube-api-access-47blz\") pod \"e2e-same-id-a-b05dd916-kserve-5b4cdf54d9-ls2cr\" (UID: \"a0ae7796-3bbd-4eff-89c0-4e188ddd23a4\") " pod="llm/e2e-same-id-a-b05dd916-kserve-5b4cdf54d9-ls2cr" Nov 04 03:33:28.194304 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:33:28.194148 4405 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a0ae7796-3bbd-4eff-89c0-4e188ddd23a4-dshm\") pod \"e2e-same-id-a-b05dd916-kserve-5b4cdf54d9-ls2cr\" (UID: \"a0ae7796-3bbd-4eff-89c0-4e188ddd23a4\") " pod="llm/e2e-same-id-a-b05dd916-kserve-5b4cdf54d9-ls2cr" Nov 04 03:33:28.194304 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:33:28.194187 4405 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a0ae7796-3bbd-4eff-89c0-4e188ddd23a4-model-cache\") pod \"e2e-same-id-a-b05dd916-kserve-5b4cdf54d9-ls2cr\" (UID: \"a0ae7796-3bbd-4eff-89c0-4e188ddd23a4\") " pod="llm/e2e-same-id-a-b05dd916-kserve-5b4cdf54d9-ls2cr" Nov 04 03:33:28.194304 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:33:28.194251 4405 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a0ae7796-3bbd-4eff-89c0-4e188ddd23a4-home\") pod \"e2e-same-id-a-b05dd916-kserve-5b4cdf54d9-ls2cr\" (UID: \"a0ae7796-3bbd-4eff-89c0-4e188ddd23a4\") " pod="llm/e2e-same-id-a-b05dd916-kserve-5b4cdf54d9-ls2cr" Nov 04 03:33:28.194487 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:33:28.194412 4405 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a0ae7796-3bbd-4eff-89c0-4e188ddd23a4-tls-certs\") pod \"e2e-same-id-a-b05dd916-kserve-5b4cdf54d9-ls2cr\" (UID: \"a0ae7796-3bbd-4eff-89c0-4e188ddd23a4\") " pod="llm/e2e-same-id-a-b05dd916-kserve-5b4cdf54d9-ls2cr" Nov 04 03:33:28.194487 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:33:28.194468 4405 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a0ae7796-3bbd-4eff-89c0-4e188ddd23a4-tmp-dir\") pod \"e2e-same-id-a-b05dd916-kserve-5b4cdf54d9-ls2cr\" (UID: \"a0ae7796-3bbd-4eff-89c0-4e188ddd23a4\") " pod="llm/e2e-same-id-a-b05dd916-kserve-5b4cdf54d9-ls2cr" Nov 04 03:33:28.296178 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:33:28.296144 4405 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-47blz\" (UniqueName: \"kubernetes.io/projected/a0ae7796-3bbd-4eff-89c0-4e188ddd23a4-kube-api-access-47blz\") pod \"e2e-same-id-a-b05dd916-kserve-5b4cdf54d9-ls2cr\" (UID: \"a0ae7796-3bbd-4eff-89c0-4e188ddd23a4\") " pod="llm/e2e-same-id-a-b05dd916-kserve-5b4cdf54d9-ls2cr" Nov 04 03:33:28.296178 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:33:28.296184 4405 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a0ae7796-3bbd-4eff-89c0-4e188ddd23a4-dshm\") pod \"e2e-same-id-a-b05dd916-kserve-5b4cdf54d9-ls2cr\" (UID: \"a0ae7796-3bbd-4eff-89c0-4e188ddd23a4\") " pod="llm/e2e-same-id-a-b05dd916-kserve-5b4cdf54d9-ls2cr" Nov 04 03:33:28.296453 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:33:28.296212 4405 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a0ae7796-3bbd-4eff-89c0-4e188ddd23a4-model-cache\") pod \"e2e-same-id-a-b05dd916-kserve-5b4cdf54d9-ls2cr\" (UID: \"a0ae7796-3bbd-4eff-89c0-4e188ddd23a4\") " pod="llm/e2e-same-id-a-b05dd916-kserve-5b4cdf54d9-ls2cr" Nov 04 03:33:28.296453 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:33:28.296241 4405 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a0ae7796-3bbd-4eff-89c0-4e188ddd23a4-home\") pod \"e2e-same-id-a-b05dd916-kserve-5b4cdf54d9-ls2cr\" (UID: \"a0ae7796-3bbd-4eff-89c0-4e188ddd23a4\") " pod="llm/e2e-same-id-a-b05dd916-kserve-5b4cdf54d9-ls2cr" Nov 04 03:33:28.296453 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:33:28.296288 4405 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a0ae7796-3bbd-4eff-89c0-4e188ddd23a4-tls-certs\") pod \"e2e-same-id-a-b05dd916-kserve-5b4cdf54d9-ls2cr\" (UID: \"a0ae7796-3bbd-4eff-89c0-4e188ddd23a4\") " pod="llm/e2e-same-id-a-b05dd916-kserve-5b4cdf54d9-ls2cr" Nov 04 03:33:28.296453 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:33:28.296319 4405 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a0ae7796-3bbd-4eff-89c0-4e188ddd23a4-tmp-dir\") pod \"e2e-same-id-a-b05dd916-kserve-5b4cdf54d9-ls2cr\" (UID: \"a0ae7796-3bbd-4eff-89c0-4e188ddd23a4\") " pod="llm/e2e-same-id-a-b05dd916-kserve-5b4cdf54d9-ls2cr" Nov 04 03:33:28.296721 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:33:28.296698 4405 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a0ae7796-3bbd-4eff-89c0-4e188ddd23a4-model-cache\") pod \"e2e-same-id-a-b05dd916-kserve-5b4cdf54d9-ls2cr\" (UID: \"a0ae7796-3bbd-4eff-89c0-4e188ddd23a4\") " pod="llm/e2e-same-id-a-b05dd916-kserve-5b4cdf54d9-ls2cr" Nov 04 03:33:28.296790 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:33:28.296733 4405 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a0ae7796-3bbd-4eff-89c0-4e188ddd23a4-tmp-dir\") pod \"e2e-same-id-a-b05dd916-kserve-5b4cdf54d9-ls2cr\" (UID: \"a0ae7796-3bbd-4eff-89c0-4e188ddd23a4\") " pod="llm/e2e-same-id-a-b05dd916-kserve-5b4cdf54d9-ls2cr" Nov 04 03:33:28.296790 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:33:28.296770 4405 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a0ae7796-3bbd-4eff-89c0-4e188ddd23a4-home\") pod \"e2e-same-id-a-b05dd916-kserve-5b4cdf54d9-ls2cr\" (UID: \"a0ae7796-3bbd-4eff-89c0-4e188ddd23a4\") " pod="llm/e2e-same-id-a-b05dd916-kserve-5b4cdf54d9-ls2cr" Nov 04 03:33:28.298627 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:33:28.298605 4405 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a0ae7796-3bbd-4eff-89c0-4e188ddd23a4-dshm\") pod \"e2e-same-id-a-b05dd916-kserve-5b4cdf54d9-ls2cr\" (UID: \"a0ae7796-3bbd-4eff-89c0-4e188ddd23a4\") " pod="llm/e2e-same-id-a-b05dd916-kserve-5b4cdf54d9-ls2cr" Nov 04 03:33:28.298927 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:33:28.298912 4405 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a0ae7796-3bbd-4eff-89c0-4e188ddd23a4-tls-certs\") pod \"e2e-same-id-a-b05dd916-kserve-5b4cdf54d9-ls2cr\" (UID: \"a0ae7796-3bbd-4eff-89c0-4e188ddd23a4\") " pod="llm/e2e-same-id-a-b05dd916-kserve-5b4cdf54d9-ls2cr" Nov 04 03:33:28.357180 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:33:28.357147 4405 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-47blz\" (UniqueName: \"kubernetes.io/projected/a0ae7796-3bbd-4eff-89c0-4e188ddd23a4-kube-api-access-47blz\") pod \"e2e-same-id-a-b05dd916-kserve-5b4cdf54d9-ls2cr\" (UID: \"a0ae7796-3bbd-4eff-89c0-4e188ddd23a4\") " pod="llm/e2e-same-id-a-b05dd916-kserve-5b4cdf54d9-ls2cr" Nov 04 03:33:28.389313 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:33:28.389281 4405 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-b05dd916-kserve-5b4cdf54d9-ls2cr" Nov 04 03:33:28.761477 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:33:28.761452 4405 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-a-b05dd916-kserve-5b4cdf54d9-ls2cr"] Nov 04 03:33:28.763690 ip-10-0-134-125 kubenswrapper[4405]: W1104 03:33:28.763642 4405 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poda0ae7796_3bbd_4eff_89c0_4e188ddd23a4.slice/crio-d79b9c7de2b9698e6fd702006c932f8339ed5c68c6cdebebed4f6076fe93932e WatchSource:0}: Error finding container d79b9c7de2b9698e6fd702006c932f8339ed5c68c6cdebebed4f6076fe93932e: Status 404 returned error can't find the container with id d79b9c7de2b9698e6fd702006c932f8339ed5c68c6cdebebed4f6076fe93932e Nov 04 03:33:28.920238 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:33:28.920203 4405 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-b05dd916-kserve-5b4cdf54d9-ls2cr" event={"ID":"a0ae7796-3bbd-4eff-89c0-4e188ddd23a4","Type":"ContainerStarted","Data":"47dccc0cd3d05530d926fbbe28284227972baca0dd8af9729ed573a4bad943c8"} Nov 04 03:33:28.920401 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:33:28.920254 4405 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-b05dd916-kserve-5b4cdf54d9-ls2cr" event={"ID":"a0ae7796-3bbd-4eff-89c0-4e188ddd23a4","Type":"ContainerStarted","Data":"d79b9c7de2b9698e6fd702006c932f8339ed5c68c6cdebebed4f6076fe93932e"} Nov 04 03:33:29.036629 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:33:29.036528 4405 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-same-id-a-b05dd916-kserve-5b4cdf54d9-ls2cr" podStartSLOduration=1.036510857 podStartE2EDuration="1.036510857s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 03:33:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 03:33:29.036150166 +0000 UTC m=+3655.466850289" watchObservedRunningTime="2026-11-04 03:33:29.036510857 +0000 UTC m=+3655.467210977" Nov 04 03:33:29.389849 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:33:29.389808 4405 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-same-id-a-b05dd916-kserve-5b4cdf54d9-ls2cr" Nov 04 03:33:36.882900 ip-10-0-134-125 kubenswrapper[4405]: E1104 03:33:36.882862 4405 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5f2vb" containerName="extractor" Nov 04 03:33:38.390408 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:33:38.390367 4405 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-same-id-a-b05dd916-kserve-5b4cdf54d9-ls2cr" Nov 04 03:33:38.403255 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:33:38.403224 4405 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-same-id-a-b05dd916-kserve-5b4cdf54d9-ls2cr" Nov 04 03:33:38.996191 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:33:38.996162 4405 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-same-id-a-b05dd916-kserve-5b4cdf54d9-ls2cr" Nov 04 03:33:39.232521 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:33:39.232473 4405 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-bl5mg"] Nov 04 03:33:59.850254 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:33:59.850213 4405 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-bl5mg"] Nov 04 03:34:02.995868 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:34:02.995823 4405 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76597d6855-7c95j container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.81:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 04 03:34:02.996519 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:34:02.996480 4405 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-7c95j" podUID="76df1601-21fb-40ab-a5ee-433f367453c7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.81:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 04 03:34:06.882991 ip-10-0-134-125 kubenswrapper[4405]: E1104 03:34:06.882946 4405 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5f2vb" containerName="extractor" Nov 04 03:34:29.447602 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:34:29.447562 4405 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-rpzrs"] Nov 04 03:34:29.450986 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:34:29.450963 4405 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-rpzrs" Nov 04 03:34:29.508526 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:34:29.508490 4405 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-rpzrs"] Nov 04 03:34:29.528770 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:34:29.528733 4405 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ca8a537a-6cd6-4543-b2fa-b8ad5f9fc659-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-452b6d7b-rpzrs\" (UID: \"ca8a537a-6cd6-4543-b2fa-b8ad5f9fc659\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-rpzrs" Nov 04 03:34:29.528970 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:34:29.528782 4405 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ca8a537a-6cd6-4543-b2fa-b8ad5f9fc659-maas-api-xks-ca\") pod \"maas-api-revoke-sub-452b6d7b-rpzrs\" (UID: \"ca8a537a-6cd6-4543-b2fa-b8ad5f9fc659\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-rpzrs" Nov 04 03:34:29.529062 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:34:29.529036 4405 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ms46q\" (UniqueName: \"kubernetes.io/projected/ca8a537a-6cd6-4543-b2fa-b8ad5f9fc659-kube-api-access-ms46q\") pod \"maas-api-revoke-sub-452b6d7b-rpzrs\" (UID: \"ca8a537a-6cd6-4543-b2fa-b8ad5f9fc659\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-rpzrs" Nov 04 03:34:29.552645 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:34:29.552614 4405 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 04 03:34:29.552849 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:34:29.552755 4405 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-slbq8\"" Nov 04 03:34:29.552849 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:34:29.552785 4405 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 04 03:34:29.552849 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:34:29.552810 4405 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 04 03:34:29.629857 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:34:29.629823 4405 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ca8a537a-6cd6-4543-b2fa-b8ad5f9fc659-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-452b6d7b-rpzrs\" (UID: \"ca8a537a-6cd6-4543-b2fa-b8ad5f9fc659\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-rpzrs" Nov 04 03:34:29.630054 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:34:29.629865 4405 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ca8a537a-6cd6-4543-b2fa-b8ad5f9fc659-maas-api-xks-ca\") pod \"maas-api-revoke-sub-452b6d7b-rpzrs\" (UID: \"ca8a537a-6cd6-4543-b2fa-b8ad5f9fc659\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-rpzrs" Nov 04 03:34:29.630054 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:34:29.629934 4405 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ms46q\" (UniqueName: \"kubernetes.io/projected/ca8a537a-6cd6-4543-b2fa-b8ad5f9fc659-kube-api-access-ms46q\") pod \"maas-api-revoke-sub-452b6d7b-rpzrs\" (UID: \"ca8a537a-6cd6-4543-b2fa-b8ad5f9fc659\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-rpzrs" Nov 04 03:34:29.630685 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:34:29.630641 4405 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ca8a537a-6cd6-4543-b2fa-b8ad5f9fc659-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-452b6d7b-rpzrs\" (UID: \"ca8a537a-6cd6-4543-b2fa-b8ad5f9fc659\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-rpzrs" Nov 04 03:34:29.632505 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:34:29.632486 4405 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ca8a537a-6cd6-4543-b2fa-b8ad5f9fc659-maas-api-xks-ca\") pod \"maas-api-revoke-sub-452b6d7b-rpzrs\" (UID: \"ca8a537a-6cd6-4543-b2fa-b8ad5f9fc659\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-rpzrs" Nov 04 03:34:29.690956 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:34:29.690918 4405 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ms46q\" (UniqueName: \"kubernetes.io/projected/ca8a537a-6cd6-4543-b2fa-b8ad5f9fc659-kube-api-access-ms46q\") pod \"maas-api-revoke-sub-452b6d7b-rpzrs\" (UID: \"ca8a537a-6cd6-4543-b2fa-b8ad5f9fc659\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-rpzrs" Nov 04 03:34:29.764295 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:34:29.764203 4405 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-rpzrs" Nov 04 03:34:30.145211 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:34:30.145181 4405 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-rpzrs"] Nov 04 03:34:30.148047 ip-10-0-134-125 kubenswrapper[4405]: W1104 03:34:30.148005 4405 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podca8a537a_6cd6_4543_b2fa_b8ad5f9fc659.slice/crio-14d3aeb8bb3c2724a0ab8542ac21734e4ac444300048d99d265ef0f005a68219 WatchSource:0}: Error finding container 14d3aeb8bb3c2724a0ab8542ac21734e4ac444300048d99d265ef0f005a68219: Status 404 returned error can't find the container with id 14d3aeb8bb3c2724a0ab8542ac21734e4ac444300048d99d265ef0f005a68219 Nov 04 03:34:30.300536 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:34:30.300503 4405 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-rpzrs" event={"ID":"ca8a537a-6cd6-4543-b2fa-b8ad5f9fc659","Type":"ContainerStarted","Data":"14d3aeb8bb3c2724a0ab8542ac21734e4ac444300048d99d265ef0f005a68219"} Nov 04 03:34:30.838345 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:34:30.838309 4405 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-bl5mg"] Nov 04 03:34:31.308535 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:34:31.308429 4405 generic.go:356] "Generic (PLEG): container finished" podID="ca8a537a-6cd6-4543-b2fa-b8ad5f9fc659" containerID="fc144a1b5bd37b8ca1ea5af55f088d3eae73e2a891e364574efb134ad5eda4a0" exitCode=0 Nov 04 03:34:31.308535 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:34:31.308495 4405 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-rpzrs" event={"ID":"ca8a537a-6cd6-4543-b2fa-b8ad5f9fc659","Type":"ContainerDied","Data":"fc144a1b5bd37b8ca1ea5af55f088d3eae73e2a891e364574efb134ad5eda4a0"} Nov 04 03:34:32.476229 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:34:32.476193 4405 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-rpzrs" Nov 04 03:34:32.562725 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:34:32.561902 4405 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ca8a537a-6cd6-4543-b2fa-b8ad5f9fc659-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ca8a537a-6cd6-4543-b2fa-b8ad5f9fc659-maas-api-xks-ca\") pod \"ca8a537a-6cd6-4543-b2fa-b8ad5f9fc659\" (UID: \"ca8a537a-6cd6-4543-b2fa-b8ad5f9fc659\") " Nov 04 03:34:32.562725 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:34:32.561984 4405 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ca8a537a-6cd6-4543-b2fa-b8ad5f9fc659-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ca8a537a-6cd6-4543-b2fa-b8ad5f9fc659-maas-api-openshift-ca\") pod \"ca8a537a-6cd6-4543-b2fa-b8ad5f9fc659\" (UID: \"ca8a537a-6cd6-4543-b2fa-b8ad5f9fc659\") " Nov 04 03:34:32.562725 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:34:32.562147 4405 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ca8a537a-6cd6-4543-b2fa-b8ad5f9fc659-kube-api-access-ms46q\" (UniqueName: \"kubernetes.io/projected/ca8a537a-6cd6-4543-b2fa-b8ad5f9fc659-kube-api-access-ms46q\") pod \"ca8a537a-6cd6-4543-b2fa-b8ad5f9fc659\" (UID: \"ca8a537a-6cd6-4543-b2fa-b8ad5f9fc659\") " Nov 04 03:34:32.563150 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:34:32.563084 4405 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ca8a537a-6cd6-4543-b2fa-b8ad5f9fc659-maas-api-openshift-ca" pod "ca8a537a-6cd6-4543-b2fa-b8ad5f9fc659" (UID: "ca8a537a-6cd6-4543-b2fa-b8ad5f9fc659"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 03:34:32.565766 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:34:32.565653 4405 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ca8a537a-6cd6-4543-b2fa-b8ad5f9fc659-maas-api-xks-ca" pod "ca8a537a-6cd6-4543-b2fa-b8ad5f9fc659" (UID: "ca8a537a-6cd6-4543-b2fa-b8ad5f9fc659"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 04 03:34:32.567708 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:34:32.567651 4405 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ca8a537a-6cd6-4543-b2fa-b8ad5f9fc659-kube-api-access-ms46q" pod "ca8a537a-6cd6-4543-b2fa-b8ad5f9fc659" (UID: "ca8a537a-6cd6-4543-b2fa-b8ad5f9fc659"). InnerVolumeSpecName "kube-api-access-ms46q". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 03:34:32.663461 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:34:32.663390 4405 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ms46q\" (UniqueName: \"kubernetes.io/projected/ca8a537a-6cd6-4543-b2fa-b8ad5f9fc659-kube-api-access-ms46q\") on node \"ip-10-0-134-125.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:34:32.663461 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:34:32.663430 4405 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ca8a537a-6cd6-4543-b2fa-b8ad5f9fc659-maas-api-xks-ca\") on node \"ip-10-0-134-125.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:34:32.663461 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:34:32.663440 4405 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ca8a537a-6cd6-4543-b2fa-b8ad5f9fc659-maas-api-openshift-ca\") on node \"ip-10-0-134-125.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:34:33.332503 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:34:33.332464 4405 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-rpzrs" event={"ID":"ca8a537a-6cd6-4543-b2fa-b8ad5f9fc659","Type":"ContainerDied","Data":"14d3aeb8bb3c2724a0ab8542ac21734e4ac444300048d99d265ef0f005a68219"} Nov 04 03:34:33.336689 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:34:33.332781 4405 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-rpzrs" Nov 04 03:34:33.336689 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:34:33.332836 4405 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="14d3aeb8bb3c2724a0ab8542ac21734e4ac444300048d99d265ef0f005a68219" Nov 04 03:34:35.669051 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:34:35.669018 4405 scope.go:122] "RemoveContainer" containerID="772f9d5ae2d1e5692abc588d7d356f8e78c6e802bbe42f5ecf353af71a133c60" Nov 04 03:34:36.884321 ip-10-0-134-125 kubenswrapper[4405]: E1104 03:34:36.884279 4405 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5f2vb" containerName="extractor" Nov 04 03:34:42.369484 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:34:42.369452 4405 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-a-b05dd916-kserve-5b4cdf54d9-ls2cr"] Nov 04 03:34:46.342382 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:34:46.342294 4405 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-bl5mg"] Nov 04 03:34:53.242917 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:34:53.242881 4405 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-jsfp9"] Nov 04 03:34:53.243342 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:34:53.243246 4405 cpu_manager.go:414] "containerMap: removing container" podUID="ca8a537a-6cd6-4543-b2fa-b8ad5f9fc659" containerName="revoke-keys" Nov 04 03:34:53.243342 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:34:53.243258 4405 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ca8a537a-6cd6-4543-b2fa-b8ad5f9fc659" containerName="revoke-keys" Nov 04 03:34:53.243342 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:34:53.243328 4405 memory_manager.go:365] "RemoveStaleState removing state" podUID="ca8a537a-6cd6-4543-b2fa-b8ad5f9fc659" containerName="revoke-keys" Nov 04 03:34:53.246450 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:34:53.246433 4405 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-jsfp9" Nov 04 03:34:53.301893 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:34:53.301853 4405 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 04 03:34:53.302110 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:34:53.301942 4405 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 04 03:34:53.302110 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:34:53.301989 4405 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 04 03:34:53.304626 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:34:53.304577 4405 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/577a3035-6086-41dd-aa8e-1b0713d4a664-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3e31a506-jsfp9\" (UID: \"577a3035-6086-41dd-aa8e-1b0713d4a664\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-jsfp9" Nov 04 03:34:53.304789 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:34:53.304765 4405 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-468sd\" (UniqueName: \"kubernetes.io/projected/577a3035-6086-41dd-aa8e-1b0713d4a664-kube-api-access-468sd\") pod \"maas-api-revoke-sub-3e31a506-jsfp9\" (UID: \"577a3035-6086-41dd-aa8e-1b0713d4a664\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-jsfp9" Nov 04 03:34:53.304864 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:34:53.304822 4405 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/577a3035-6086-41dd-aa8e-1b0713d4a664-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3e31a506-jsfp9\" (UID: \"577a3035-6086-41dd-aa8e-1b0713d4a664\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-jsfp9" Nov 04 03:34:53.351911 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:34:53.351875 4405 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-slbq8\"" Nov 04 03:34:53.353578 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:34:53.353555 4405 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-jsfp9"] Nov 04 03:34:53.405564 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:34:53.405522 4405 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-468sd\" (UniqueName: \"kubernetes.io/projected/577a3035-6086-41dd-aa8e-1b0713d4a664-kube-api-access-468sd\") pod \"maas-api-revoke-sub-3e31a506-jsfp9\" (UID: \"577a3035-6086-41dd-aa8e-1b0713d4a664\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-jsfp9" Nov 04 03:34:53.405803 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:34:53.405581 4405 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/577a3035-6086-41dd-aa8e-1b0713d4a664-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3e31a506-jsfp9\" (UID: \"577a3035-6086-41dd-aa8e-1b0713d4a664\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-jsfp9" Nov 04 03:34:53.405803 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:34:53.405611 4405 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/577a3035-6086-41dd-aa8e-1b0713d4a664-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3e31a506-jsfp9\" (UID: \"577a3035-6086-41dd-aa8e-1b0713d4a664\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-jsfp9" Nov 04 03:34:53.406221 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:34:53.406201 4405 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/577a3035-6086-41dd-aa8e-1b0713d4a664-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3e31a506-jsfp9\" (UID: \"577a3035-6086-41dd-aa8e-1b0713d4a664\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-jsfp9" Nov 04 03:34:53.408116 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:34:53.408094 4405 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/577a3035-6086-41dd-aa8e-1b0713d4a664-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3e31a506-jsfp9\" (UID: \"577a3035-6086-41dd-aa8e-1b0713d4a664\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-jsfp9" Nov 04 03:34:53.466923 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:34:53.466887 4405 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-468sd\" (UniqueName: \"kubernetes.io/projected/577a3035-6086-41dd-aa8e-1b0713d4a664-kube-api-access-468sd\") pod \"maas-api-revoke-sub-3e31a506-jsfp9\" (UID: \"577a3035-6086-41dd-aa8e-1b0713d4a664\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-jsfp9" Nov 04 03:34:53.559772 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:34:53.559689 4405 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-jsfp9" Nov 04 03:34:53.931572 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:34:53.931544 4405 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-jsfp9"] Nov 04 03:34:53.934538 ip-10-0-134-125 kubenswrapper[4405]: W1104 03:34:53.934500 4405 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod577a3035_6086_41dd_aa8e_1b0713d4a664.slice/crio-15780d396e46dc1a2c4d7ee0d508aa14d3a320f0bd93392a69e0b0bae844cf6a WatchSource:0}: Error finding container 15780d396e46dc1a2c4d7ee0d508aa14d3a320f0bd93392a69e0b0bae844cf6a: Status 404 returned error can't find the container with id 15780d396e46dc1a2c4d7ee0d508aa14d3a320f0bd93392a69e0b0bae844cf6a Nov 04 03:34:54.486595 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:34:54.486513 4405 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-jsfp9" event={"ID":"577a3035-6086-41dd-aa8e-1b0713d4a664","Type":"ContainerStarted","Data":"0cc3862ede90a79322ba01c8524ffaa2825bd808738bbcdd13506c30028cf40b"} Nov 04 03:34:54.486595 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:34:54.486570 4405 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-jsfp9" event={"ID":"577a3035-6086-41dd-aa8e-1b0713d4a664","Type":"ContainerStarted","Data":"15780d396e46dc1a2c4d7ee0d508aa14d3a320f0bd93392a69e0b0bae844cf6a"} Nov 04 03:34:54.603418 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:34:54.603343 4405 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-jsfp9" podStartSLOduration=1.603321818 podStartE2EDuration="1.603321818s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 03:34:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 03:34:54.601747277 +0000 UTC m=+3741.032447399" watchObservedRunningTime="2026-11-04 03:34:54.603321818 +0000 UTC m=+3741.034021938" Nov 04 03:34:55.495051 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:34:55.495011 4405 generic.go:356] "Generic (PLEG): container finished" podID="577a3035-6086-41dd-aa8e-1b0713d4a664" containerID="0cc3862ede90a79322ba01c8524ffaa2825bd808738bbcdd13506c30028cf40b" exitCode=0 Nov 04 03:34:55.495479 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:34:55.495108 4405 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-jsfp9" event={"ID":"577a3035-6086-41dd-aa8e-1b0713d4a664","Type":"ContainerDied","Data":"0cc3862ede90a79322ba01c8524ffaa2825bd808738bbcdd13506c30028cf40b"} Nov 04 03:34:56.990286 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:34:56.990260 4405 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-jsfp9" Nov 04 03:34:57.082258 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:34:57.082219 4405 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/577a3035-6086-41dd-aa8e-1b0713d4a664-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/577a3035-6086-41dd-aa8e-1b0713d4a664-maas-api-openshift-ca\") pod \"577a3035-6086-41dd-aa8e-1b0713d4a664\" (UID: \"577a3035-6086-41dd-aa8e-1b0713d4a664\") " Nov 04 03:34:57.082444 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:34:57.082272 4405 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/577a3035-6086-41dd-aa8e-1b0713d4a664-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/577a3035-6086-41dd-aa8e-1b0713d4a664-maas-api-xks-ca\") pod \"577a3035-6086-41dd-aa8e-1b0713d4a664\" (UID: \"577a3035-6086-41dd-aa8e-1b0713d4a664\") " Nov 04 03:34:57.082444 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:34:57.082299 4405 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/577a3035-6086-41dd-aa8e-1b0713d4a664-kube-api-access-468sd\" (UniqueName: \"kubernetes.io/projected/577a3035-6086-41dd-aa8e-1b0713d4a664-kube-api-access-468sd\") pod \"577a3035-6086-41dd-aa8e-1b0713d4a664\" (UID: \"577a3035-6086-41dd-aa8e-1b0713d4a664\") " Nov 04 03:34:57.082725 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:34:57.082642 4405 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/577a3035-6086-41dd-aa8e-1b0713d4a664-maas-api-openshift-ca" pod "577a3035-6086-41dd-aa8e-1b0713d4a664" (UID: "577a3035-6086-41dd-aa8e-1b0713d4a664"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 03:34:57.084542 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:34:57.084520 4405 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/577a3035-6086-41dd-aa8e-1b0713d4a664-maas-api-xks-ca" pod "577a3035-6086-41dd-aa8e-1b0713d4a664" (UID: "577a3035-6086-41dd-aa8e-1b0713d4a664"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 04 03:34:57.084542 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:34:57.084529 4405 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/577a3035-6086-41dd-aa8e-1b0713d4a664-kube-api-access-468sd" pod "577a3035-6086-41dd-aa8e-1b0713d4a664" (UID: "577a3035-6086-41dd-aa8e-1b0713d4a664"). InnerVolumeSpecName "kube-api-access-468sd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 03:34:57.184796 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:34:57.184686 4405 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/577a3035-6086-41dd-aa8e-1b0713d4a664-maas-api-openshift-ca\") on node \"ip-10-0-134-125.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:34:57.184796 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:34:57.184733 4405 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/577a3035-6086-41dd-aa8e-1b0713d4a664-maas-api-xks-ca\") on node \"ip-10-0-134-125.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:34:57.184796 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:34:57.184750 4405 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-468sd\" (UniqueName: \"kubernetes.io/projected/577a3035-6086-41dd-aa8e-1b0713d4a664-kube-api-access-468sd\") on node \"ip-10-0-134-125.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:34:57.389830 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:34:57.389791 4405 kuberuntime_container.go:898] "Killing container with a grace period" pod="llm/e2e-same-id-a-b05dd916-kserve-5b4cdf54d9-ls2cr" podUID="a0ae7796-3bbd-4eff-89c0-4e188ddd23a4" containerName="main" containerID="cri-o://47dccc0cd3d05530d926fbbe28284227972baca0dd8af9729ed573a4bad943c8" gracePeriod=45 Nov 04 03:34:57.578848 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:34:57.578796 4405 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-jsfp9" event={"ID":"577a3035-6086-41dd-aa8e-1b0713d4a664","Type":"ContainerDied","Data":"15780d396e46dc1a2c4d7ee0d508aa14d3a320f0bd93392a69e0b0bae844cf6a"} Nov 04 03:34:57.579069 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:34:57.578861 4405 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="15780d396e46dc1a2c4d7ee0d508aa14d3a320f0bd93392a69e0b0bae844cf6a" Nov 04 03:34:57.579069 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:34:57.578909 4405 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-jsfp9" Nov 04 03:34:57.581200 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:34:57.581170 4405 generic.go:356] "Generic (PLEG): container finished" podID="a0ae7796-3bbd-4eff-89c0-4e188ddd23a4" containerID="47dccc0cd3d05530d926fbbe28284227972baca0dd8af9729ed573a4bad943c8" exitCode=0 Nov 04 03:34:57.581372 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:34:57.581214 4405 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-b05dd916-kserve-5b4cdf54d9-ls2cr" event={"ID":"a0ae7796-3bbd-4eff-89c0-4e188ddd23a4","Type":"ContainerDied","Data":"47dccc0cd3d05530d926fbbe28284227972baca0dd8af9729ed573a4bad943c8"} Nov 04 03:34:57.975904 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:34:57.975863 4405 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76597d6855-dqn2b container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.139:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 04 03:34:57.976113 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:34:57.975933 4405 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-dqn2b" podUID="415bafea-577e-4fba-925e-eb2c492e3cca" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.139:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 04 03:34:57.989768 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:34:57.989741 4405 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-b05dd916-kserve-5b4cdf54d9-ls2cr" Nov 04 03:34:57.990205 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:34:57.990183 4405 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a0ae7796-3bbd-4eff-89c0-4e188ddd23a4-home\" (UniqueName: \"kubernetes.io/empty-dir/a0ae7796-3bbd-4eff-89c0-4e188ddd23a4-home\") pod \"a0ae7796-3bbd-4eff-89c0-4e188ddd23a4\" (UID: \"a0ae7796-3bbd-4eff-89c0-4e188ddd23a4\") " Nov 04 03:34:57.990323 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:34:57.990217 4405 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a0ae7796-3bbd-4eff-89c0-4e188ddd23a4-tls-certs\" (UniqueName: \"kubernetes.io/secret/a0ae7796-3bbd-4eff-89c0-4e188ddd23a4-tls-certs\") pod \"a0ae7796-3bbd-4eff-89c0-4e188ddd23a4\" (UID: \"a0ae7796-3bbd-4eff-89c0-4e188ddd23a4\") " Nov 04 03:34:57.990323 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:34:57.990237 4405 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a0ae7796-3bbd-4eff-89c0-4e188ddd23a4-dshm\" (UniqueName: \"kubernetes.io/empty-dir/a0ae7796-3bbd-4eff-89c0-4e188ddd23a4-dshm\") pod \"a0ae7796-3bbd-4eff-89c0-4e188ddd23a4\" (UID: \"a0ae7796-3bbd-4eff-89c0-4e188ddd23a4\") " Nov 04 03:34:57.990323 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:34:57.990253 4405 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a0ae7796-3bbd-4eff-89c0-4e188ddd23a4-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a0ae7796-3bbd-4eff-89c0-4e188ddd23a4-tmp-dir\") pod \"a0ae7796-3bbd-4eff-89c0-4e188ddd23a4\" (UID: \"a0ae7796-3bbd-4eff-89c0-4e188ddd23a4\") " Nov 04 03:34:57.990781 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:34:57.990629 4405 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a0ae7796-3bbd-4eff-89c0-4e188ddd23a4-tmp-dir" pod "a0ae7796-3bbd-4eff-89c0-4e188ddd23a4" (UID: "a0ae7796-3bbd-4eff-89c0-4e188ddd23a4"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 03:34:57.990863 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:34:57.990843 4405 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a0ae7796-3bbd-4eff-89c0-4e188ddd23a4-home" pod "a0ae7796-3bbd-4eff-89c0-4e188ddd23a4" (UID: "a0ae7796-3bbd-4eff-89c0-4e188ddd23a4"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 03:34:57.999695 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:34:57.995843 4405 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a0ae7796-3bbd-4eff-89c0-4e188ddd23a4-dshm" pod "a0ae7796-3bbd-4eff-89c0-4e188ddd23a4" (UID: "a0ae7796-3bbd-4eff-89c0-4e188ddd23a4"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 03:34:58.008289 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:34:58.008230 4405 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a0ae7796-3bbd-4eff-89c0-4e188ddd23a4-tls-certs" pod "a0ae7796-3bbd-4eff-89c0-4e188ddd23a4" (UID: "a0ae7796-3bbd-4eff-89c0-4e188ddd23a4"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 04 03:34:58.091032 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:34:58.090992 4405 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a0ae7796-3bbd-4eff-89c0-4e188ddd23a4-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a0ae7796-3bbd-4eff-89c0-4e188ddd23a4-model-cache\") pod \"a0ae7796-3bbd-4eff-89c0-4e188ddd23a4\" (UID: \"a0ae7796-3bbd-4eff-89c0-4e188ddd23a4\") " Nov 04 03:34:58.091219 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:34:58.091082 4405 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a0ae7796-3bbd-4eff-89c0-4e188ddd23a4-kube-api-access-47blz\" (UniqueName: \"kubernetes.io/projected/a0ae7796-3bbd-4eff-89c0-4e188ddd23a4-kube-api-access-47blz\") pod \"a0ae7796-3bbd-4eff-89c0-4e188ddd23a4\" (UID: \"a0ae7796-3bbd-4eff-89c0-4e188ddd23a4\") " Nov 04 03:34:58.091334 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:34:58.091305 4405 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a0ae7796-3bbd-4eff-89c0-4e188ddd23a4-model-cache" pod "a0ae7796-3bbd-4eff-89c0-4e188ddd23a4" (UID: "a0ae7796-3bbd-4eff-89c0-4e188ddd23a4"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 03:34:58.091380 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:34:58.091331 4405 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a0ae7796-3bbd-4eff-89c0-4e188ddd23a4-home\") on node \"ip-10-0-134-125.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:34:58.091380 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:34:58.091348 4405 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a0ae7796-3bbd-4eff-89c0-4e188ddd23a4-tls-certs\") on node \"ip-10-0-134-125.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:34:58.091380 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:34:58.091363 4405 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a0ae7796-3bbd-4eff-89c0-4e188ddd23a4-dshm\") on node \"ip-10-0-134-125.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:34:58.091380 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:34:58.091376 4405 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a0ae7796-3bbd-4eff-89c0-4e188ddd23a4-tmp-dir\") on node \"ip-10-0-134-125.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:34:58.093526 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:34:58.093477 4405 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a0ae7796-3bbd-4eff-89c0-4e188ddd23a4-kube-api-access-47blz" pod "a0ae7796-3bbd-4eff-89c0-4e188ddd23a4" (UID: "a0ae7796-3bbd-4eff-89c0-4e188ddd23a4"). InnerVolumeSpecName "kube-api-access-47blz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 03:34:58.192354 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:34:58.192313 4405 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a0ae7796-3bbd-4eff-89c0-4e188ddd23a4-model-cache\") on node \"ip-10-0-134-125.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:34:58.192354 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:34:58.192350 4405 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-47blz\" (UniqueName: \"kubernetes.io/projected/a0ae7796-3bbd-4eff-89c0-4e188ddd23a4-kube-api-access-47blz\") on node \"ip-10-0-134-125.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:34:58.587076 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:34:58.587043 4405 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-b05dd916-kserve-5b4cdf54d9-ls2cr" event={"ID":"a0ae7796-3bbd-4eff-89c0-4e188ddd23a4","Type":"ContainerDied","Data":"d79b9c7de2b9698e6fd702006c932f8339ed5c68c6cdebebed4f6076fe93932e"} Nov 04 03:34:58.587260 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:34:58.587086 4405 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-b05dd916-kserve-5b4cdf54d9-ls2cr" Nov 04 03:34:58.587260 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:34:58.587093 4405 scope.go:122] "RemoveContainer" containerID="47dccc0cd3d05530d926fbbe28284227972baca0dd8af9729ed573a4bad943c8" Nov 04 03:34:58.752786 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:34:58.752744 4405 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-a-b05dd916-kserve-5b4cdf54d9-ls2cr"] Nov 04 03:34:58.802474 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:34:58.802442 4405 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["llm/e2e-same-id-a-b05dd916-kserve-5b4cdf54d9-ls2cr"] Nov 04 03:34:58.985243 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:34:58.985205 4405 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/e2e-same-id-a-b05dd916-kserve-5b4cdf54d9-ls2cr" podUID="a0ae7796-3bbd-4eff-89c0-4e188ddd23a4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.167:8000/ready\": context deadline exceeded" Nov 04 03:34:59.700120 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:34:59.700087 4405 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a0ae7796-3bbd-4eff-89c0-4e188ddd23a4" path="/var/lib/kubelet/pods/a0ae7796-3bbd-4eff-89c0-4e188ddd23a4/volumes" Nov 04 03:34:59.916102 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:34:59.916068 4405 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-bl5mg"] Nov 04 03:35:06.883411 ip-10-0-134-125 kubenswrapper[4405]: E1104 03:35:06.883371 4405 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5f2vb" containerName="extractor" Nov 04 03:35:11.923599 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:35:11.923562 4405 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-bl5mg"] Nov 04 03:35:15.005908 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:35:15.005822 4405 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76597d6855-xjqqw container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.138:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 04 03:35:15.006631 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:35:15.006588 4405 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-xjqqw" podUID="c97cb26c-6fc7-45db-9984-aa459ddb54df" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.138:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 04 03:35:17.537774 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:35:17.537734 4405 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c01e047c-sgxrj"] Nov 04 03:35:17.538200 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:35:17.538107 4405 cpu_manager.go:414] "containerMap: removing container" podUID="a0ae7796-3bbd-4eff-89c0-4e188ddd23a4" containerName="main" Nov 04 03:35:17.538200 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:35:17.538118 4405 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a0ae7796-3bbd-4eff-89c0-4e188ddd23a4" containerName="main" Nov 04 03:35:17.538200 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:35:17.538134 4405 cpu_manager.go:414] "containerMap: removing container" podUID="577a3035-6086-41dd-aa8e-1b0713d4a664" containerName="revoke-keys" Nov 04 03:35:17.538200 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:35:17.538140 4405 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="577a3035-6086-41dd-aa8e-1b0713d4a664" containerName="revoke-keys" Nov 04 03:35:17.538341 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:35:17.538208 4405 memory_manager.go:365] "RemoveStaleState removing state" podUID="577a3035-6086-41dd-aa8e-1b0713d4a664" containerName="revoke-keys" Nov 04 03:35:17.538341 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:35:17.538224 4405 memory_manager.go:365] "RemoveStaleState removing state" podUID="a0ae7796-3bbd-4eff-89c0-4e188ddd23a4" containerName="main" Nov 04 03:35:17.541398 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:35:17.541375 4405 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c01e047c-sgxrj" Nov 04 03:35:17.613513 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:35:17.613469 4405 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0577ac72-e357-4c18-ad55-6f1566b4b6ca-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c01e047c-sgxrj\" (UID: \"0577ac72-e357-4c18-ad55-6f1566b4b6ca\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c01e047c-sgxrj" Nov 04 03:35:17.613718 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:35:17.613594 4405 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0577ac72-e357-4c18-ad55-6f1566b4b6ca-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c01e047c-sgxrj\" (UID: \"0577ac72-e357-4c18-ad55-6f1566b4b6ca\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c01e047c-sgxrj" Nov 04 03:35:17.613718 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:35:17.613661 4405 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fj4s8\" (UniqueName: \"kubernetes.io/projected/0577ac72-e357-4c18-ad55-6f1566b4b6ca-kube-api-access-fj4s8\") pod \"maas-api-revoke-sub-c01e047c-sgxrj\" (UID: \"0577ac72-e357-4c18-ad55-6f1566b4b6ca\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c01e047c-sgxrj" Nov 04 03:35:17.643006 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:35:17.642971 4405 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 04 03:35:17.643006 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:35:17.642969 4405 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 04 03:35:17.643006 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:35:17.643005 4405 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-slbq8\"" Nov 04 03:35:17.643346 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:35:17.642985 4405 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 04 03:35:17.645132 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:35:17.645107 4405 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c01e047c-sgxrj"] Nov 04 03:35:17.715270 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:35:17.715225 4405 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0577ac72-e357-4c18-ad55-6f1566b4b6ca-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c01e047c-sgxrj\" (UID: \"0577ac72-e357-4c18-ad55-6f1566b4b6ca\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c01e047c-sgxrj" Nov 04 03:35:17.715482 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:35:17.715287 4405 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fj4s8\" (UniqueName: \"kubernetes.io/projected/0577ac72-e357-4c18-ad55-6f1566b4b6ca-kube-api-access-fj4s8\") pod \"maas-api-revoke-sub-c01e047c-sgxrj\" (UID: \"0577ac72-e357-4c18-ad55-6f1566b4b6ca\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c01e047c-sgxrj" Nov 04 03:35:17.715482 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:35:17.715351 4405 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0577ac72-e357-4c18-ad55-6f1566b4b6ca-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c01e047c-sgxrj\" (UID: \"0577ac72-e357-4c18-ad55-6f1566b4b6ca\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c01e047c-sgxrj" Nov 04 03:35:17.715908 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:35:17.715887 4405 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0577ac72-e357-4c18-ad55-6f1566b4b6ca-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c01e047c-sgxrj\" (UID: \"0577ac72-e357-4c18-ad55-6f1566b4b6ca\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c01e047c-sgxrj" Nov 04 03:35:17.717880 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:35:17.717858 4405 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0577ac72-e357-4c18-ad55-6f1566b4b6ca-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c01e047c-sgxrj\" (UID: \"0577ac72-e357-4c18-ad55-6f1566b4b6ca\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c01e047c-sgxrj" Nov 04 03:35:17.776418 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:35:17.776382 4405 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fj4s8\" (UniqueName: \"kubernetes.io/projected/0577ac72-e357-4c18-ad55-6f1566b4b6ca-kube-api-access-fj4s8\") pod \"maas-api-revoke-sub-c01e047c-sgxrj\" (UID: \"0577ac72-e357-4c18-ad55-6f1566b4b6ca\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c01e047c-sgxrj" Nov 04 03:35:17.856160 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:35:17.856131 4405 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c01e047c-sgxrj" Nov 04 03:35:18.226936 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:35:18.226854 4405 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c01e047c-sgxrj"] Nov 04 03:35:18.229424 ip-10-0-134-125 kubenswrapper[4405]: W1104 03:35:18.229390 4405 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0577ac72_e357_4c18_ad55_6f1566b4b6ca.slice/crio-ee39542f5ff50407b2094b30fb8cad764345ba1e671eb603e9c04060571b98e8 WatchSource:0}: Error finding container ee39542f5ff50407b2094b30fb8cad764345ba1e671eb603e9c04060571b98e8: Status 404 returned error can't find the container with id ee39542f5ff50407b2094b30fb8cad764345ba1e671eb603e9c04060571b98e8 Nov 04 03:35:18.705995 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:35:18.705942 4405 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c01e047c-sgxrj" event={"ID":"0577ac72-e357-4c18-ad55-6f1566b4b6ca","Type":"ContainerStarted","Data":"922e2385b3d60b6317c286c6b9ad327747441dab8a6386f2319bc402757318c7"} Nov 04 03:35:18.705995 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:35:18.705999 4405 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c01e047c-sgxrj" event={"ID":"0577ac72-e357-4c18-ad55-6f1566b4b6ca","Type":"ContainerStarted","Data":"ee39542f5ff50407b2094b30fb8cad764345ba1e671eb603e9c04060571b98e8"} Nov 04 03:35:18.822282 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:35:18.822175 4405 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c01e047c-sgxrj" podStartSLOduration=1.822161626 podStartE2EDuration="1.822161626s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 03:35:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 03:35:18.821291008 +0000 UTC m=+3765.251991128" watchObservedRunningTime="2026-11-04 03:35:18.822161626 +0000 UTC m=+3765.252861808" Nov 04 03:35:19.712611 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:35:19.712562 4405 generic.go:356] "Generic (PLEG): container finished" podID="0577ac72-e357-4c18-ad55-6f1566b4b6ca" containerID="922e2385b3d60b6317c286c6b9ad327747441dab8a6386f2319bc402757318c7" exitCode=0 Nov 04 03:35:19.713015 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:35:19.712635 4405 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c01e047c-sgxrj" event={"ID":"0577ac72-e357-4c18-ad55-6f1566b4b6ca","Type":"ContainerDied","Data":"922e2385b3d60b6317c286c6b9ad327747441dab8a6386f2319bc402757318c7"} Nov 04 03:35:20.852077 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:35:20.852049 4405 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c01e047c-sgxrj" Nov 04 03:35:20.949217 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:35:20.949170 4405 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0577ac72-e357-4c18-ad55-6f1566b4b6ca-kube-api-access-fj4s8\" (UniqueName: \"kubernetes.io/projected/0577ac72-e357-4c18-ad55-6f1566b4b6ca-kube-api-access-fj4s8\") pod \"0577ac72-e357-4c18-ad55-6f1566b4b6ca\" (UID: \"0577ac72-e357-4c18-ad55-6f1566b4b6ca\") " Nov 04 03:35:20.949412 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:35:20.949278 4405 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0577ac72-e357-4c18-ad55-6f1566b4b6ca-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0577ac72-e357-4c18-ad55-6f1566b4b6ca-maas-api-openshift-ca\") pod \"0577ac72-e357-4c18-ad55-6f1566b4b6ca\" (UID: \"0577ac72-e357-4c18-ad55-6f1566b4b6ca\") " Nov 04 03:35:20.949412 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:35:20.949326 4405 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0577ac72-e357-4c18-ad55-6f1566b4b6ca-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0577ac72-e357-4c18-ad55-6f1566b4b6ca-maas-api-xks-ca\") pod \"0577ac72-e357-4c18-ad55-6f1566b4b6ca\" (UID: \"0577ac72-e357-4c18-ad55-6f1566b4b6ca\") " Nov 04 03:35:20.949750 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:35:20.949720 4405 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0577ac72-e357-4c18-ad55-6f1566b4b6ca-maas-api-openshift-ca" pod "0577ac72-e357-4c18-ad55-6f1566b4b6ca" (UID: "0577ac72-e357-4c18-ad55-6f1566b4b6ca"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 03:35:20.951557 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:35:20.951532 4405 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0577ac72-e357-4c18-ad55-6f1566b4b6ca-maas-api-xks-ca" pod "0577ac72-e357-4c18-ad55-6f1566b4b6ca" (UID: "0577ac72-e357-4c18-ad55-6f1566b4b6ca"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 04 03:35:20.951557 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:35:20.951537 4405 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0577ac72-e357-4c18-ad55-6f1566b4b6ca-kube-api-access-fj4s8" pod "0577ac72-e357-4c18-ad55-6f1566b4b6ca" (UID: "0577ac72-e357-4c18-ad55-6f1566b4b6ca"). InnerVolumeSpecName "kube-api-access-fj4s8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 03:35:21.050584 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:35:21.050488 4405 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fj4s8\" (UniqueName: \"kubernetes.io/projected/0577ac72-e357-4c18-ad55-6f1566b4b6ca-kube-api-access-fj4s8\") on node \"ip-10-0-134-125.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:35:21.050584 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:35:21.050523 4405 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0577ac72-e357-4c18-ad55-6f1566b4b6ca-maas-api-openshift-ca\") on node \"ip-10-0-134-125.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:35:21.050584 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:35:21.050535 4405 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0577ac72-e357-4c18-ad55-6f1566b4b6ca-maas-api-xks-ca\") on node \"ip-10-0-134-125.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:35:21.722274 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:35:21.722247 4405 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c01e047c-sgxrj" Nov 04 03:35:21.722474 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:35:21.722241 4405 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c01e047c-sgxrj" event={"ID":"0577ac72-e357-4c18-ad55-6f1566b4b6ca","Type":"ContainerDied","Data":"ee39542f5ff50407b2094b30fb8cad764345ba1e671eb603e9c04060571b98e8"} Nov 04 03:35:21.722474 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:35:21.722375 4405 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ee39542f5ff50407b2094b30fb8cad764345ba1e671eb603e9c04060571b98e8" Nov 04 03:35:28.047230 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:35:28.047183 4405 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-px88h"] Nov 04 03:35:28.096912 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:35:28.096878 4405 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-px88h"] Nov 04 03:35:29.699949 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:35:29.699905 4405 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b72bc8d7-5e6d-44c0-81aa-81fa4d6165f7" path="/var/lib/kubelet/pods/b72bc8d7-5e6d-44c0-81aa-81fa4d6165f7/volumes" Nov 04 03:35:30.347136 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:35:30.347098 4405 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-bl5mg"] Nov 04 03:35:30.730340 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:35:30.730255 4405 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-79b4f86cc7-dwtpl"] Nov 04 03:35:30.730863 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:35:30.730643 4405 cpu_manager.go:414] "containerMap: removing container" podUID="0577ac72-e357-4c18-ad55-6f1566b4b6ca" containerName="revoke-keys" Nov 04 03:35:30.730863 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:35:30.730654 4405 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0577ac72-e357-4c18-ad55-6f1566b4b6ca" containerName="revoke-keys" Nov 04 03:35:30.730863 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:35:30.730739 4405 memory_manager.go:365] "RemoveStaleState removing state" podUID="0577ac72-e357-4c18-ad55-6f1566b4b6ca" containerName="revoke-keys" Nov 04 03:35:30.734639 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:35:30.734621 4405 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-79b4f86cc7-dwtpl" Nov 04 03:35:30.751443 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:35:30.751416 4405 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-78d5c4d8b5-4q4db"] Nov 04 03:35:30.754731 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:35:30.754714 4405 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-78d5c4d8b5-4q4db" Nov 04 03:35:30.790764 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:35:30.790638 4405 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 04 03:35:30.799442 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:35:30.799411 4405 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 04 03:35:30.799726 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:35:30.799701 4405 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\"" Nov 04 03:35:30.836501 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:35:30.836413 4405 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-qtjr2\"" Nov 04 03:35:30.836501 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:35:30.836416 4405 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-serving-cert\"" Nov 04 03:35:30.839443 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:35:30.839414 4405 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-79b4f86cc7-dwtpl"] Nov 04 03:35:30.845899 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:35:30.845864 4405 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p87cv\" (UniqueName: \"kubernetes.io/projected/1a63263a-ab89-45ed-82da-2a2c1dbe274a-kube-api-access-p87cv\") pod \"maas-api-79b4f86cc7-dwtpl\" (UID: \"1a63263a-ab89-45ed-82da-2a2c1dbe274a\") " pod="odh-ai-gateway-infra/maas-api-79b4f86cc7-dwtpl" Nov 04 03:35:30.846047 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:35:30.845909 4405 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1a63263a-ab89-45ed-82da-2a2c1dbe274a-openshift-service-ca\") pod \"maas-api-79b4f86cc7-dwtpl\" (UID: \"1a63263a-ab89-45ed-82da-2a2c1dbe274a\") " pod="odh-ai-gateway-infra/maas-api-79b4f86cc7-dwtpl" Nov 04 03:35:30.846047 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:35:30.845964 4405 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1a63263a-ab89-45ed-82da-2a2c1dbe274a-maas-api-tls\") pod \"maas-api-79b4f86cc7-dwtpl\" (UID: \"1a63263a-ab89-45ed-82da-2a2c1dbe274a\") " pod="odh-ai-gateway-infra/maas-api-79b4f86cc7-dwtpl" Nov 04 03:35:30.846165 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:35:30.846147 4405 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/58b86d8b-324e-40b7-8862-203885a84f7a-maas-api-tls\") pod \"maas-api-78d5c4d8b5-4q4db\" (UID: \"58b86d8b-324e-40b7-8862-203885a84f7a\") " pod="odh-ai-gateway-infra/maas-api-78d5c4d8b5-4q4db" Nov 04 03:35:30.846225 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:35:30.846194 4405 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/58b86d8b-324e-40b7-8862-203885a84f7a-openshift-service-ca\") pod \"maas-api-78d5c4d8b5-4q4db\" (UID: \"58b86d8b-324e-40b7-8862-203885a84f7a\") " pod="odh-ai-gateway-infra/maas-api-78d5c4d8b5-4q4db" Nov 04 03:35:30.846274 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:35:30.846256 4405 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1a63263a-ab89-45ed-82da-2a2c1dbe274a-metrics-certs\") pod \"maas-api-79b4f86cc7-dwtpl\" (UID: \"1a63263a-ab89-45ed-82da-2a2c1dbe274a\") " pod="odh-ai-gateway-infra/maas-api-79b4f86cc7-dwtpl" Nov 04 03:35:30.846345 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:35:30.846326 4405 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5mm69\" (UniqueName: \"kubernetes.io/projected/58b86d8b-324e-40b7-8862-203885a84f7a-kube-api-access-5mm69\") pod \"maas-api-78d5c4d8b5-4q4db\" (UID: \"58b86d8b-324e-40b7-8862-203885a84f7a\") " pod="odh-ai-gateway-infra/maas-api-78d5c4d8b5-4q4db" Nov 04 03:35:30.846567 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:35:30.846528 4405 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/58b86d8b-324e-40b7-8862-203885a84f7a-metrics-certs\") pod \"maas-api-78d5c4d8b5-4q4db\" (UID: \"58b86d8b-324e-40b7-8862-203885a84f7a\") " pod="odh-ai-gateway-infra/maas-api-78d5c4d8b5-4q4db" Nov 04 03:35:30.859751 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:35:30.859718 4405 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-78d5c4d8b5-4q4db"] Nov 04 03:35:30.947303 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:35:30.947268 4405 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1a63263a-ab89-45ed-82da-2a2c1dbe274a-maas-api-tls\") pod \"maas-api-79b4f86cc7-dwtpl\" (UID: \"1a63263a-ab89-45ed-82da-2a2c1dbe274a\") " pod="odh-ai-gateway-infra/maas-api-79b4f86cc7-dwtpl" Nov 04 03:35:30.947303 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:35:30.947305 4405 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/58b86d8b-324e-40b7-8862-203885a84f7a-maas-api-tls\") pod \"maas-api-78d5c4d8b5-4q4db\" (UID: \"58b86d8b-324e-40b7-8862-203885a84f7a\") " pod="odh-ai-gateway-infra/maas-api-78d5c4d8b5-4q4db" Nov 04 03:35:30.947590 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:35:30.947326 4405 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/58b86d8b-324e-40b7-8862-203885a84f7a-openshift-service-ca\") pod \"maas-api-78d5c4d8b5-4q4db\" (UID: \"58b86d8b-324e-40b7-8862-203885a84f7a\") " pod="odh-ai-gateway-infra/maas-api-78d5c4d8b5-4q4db" Nov 04 03:35:30.947590 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:35:30.947347 4405 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1a63263a-ab89-45ed-82da-2a2c1dbe274a-metrics-certs\") pod \"maas-api-79b4f86cc7-dwtpl\" (UID: \"1a63263a-ab89-45ed-82da-2a2c1dbe274a\") " pod="odh-ai-gateway-infra/maas-api-79b4f86cc7-dwtpl" Nov 04 03:35:30.947590 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:35:30.947376 4405 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5mm69\" (UniqueName: \"kubernetes.io/projected/58b86d8b-324e-40b7-8862-203885a84f7a-kube-api-access-5mm69\") pod \"maas-api-78d5c4d8b5-4q4db\" (UID: \"58b86d8b-324e-40b7-8862-203885a84f7a\") " pod="odh-ai-gateway-infra/maas-api-78d5c4d8b5-4q4db" Nov 04 03:35:30.947590 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:35:30.947510 4405 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/58b86d8b-324e-40b7-8862-203885a84f7a-metrics-certs\") pod \"maas-api-78d5c4d8b5-4q4db\" (UID: \"58b86d8b-324e-40b7-8862-203885a84f7a\") " pod="odh-ai-gateway-infra/maas-api-78d5c4d8b5-4q4db" Nov 04 03:35:30.947590 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:35:30.947558 4405 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p87cv\" (UniqueName: \"kubernetes.io/projected/1a63263a-ab89-45ed-82da-2a2c1dbe274a-kube-api-access-p87cv\") pod \"maas-api-79b4f86cc7-dwtpl\" (UID: \"1a63263a-ab89-45ed-82da-2a2c1dbe274a\") " pod="odh-ai-gateway-infra/maas-api-79b4f86cc7-dwtpl" Nov 04 03:35:30.947590 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:35:30.947581 4405 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1a63263a-ab89-45ed-82da-2a2c1dbe274a-openshift-service-ca\") pod \"maas-api-79b4f86cc7-dwtpl\" (UID: \"1a63263a-ab89-45ed-82da-2a2c1dbe274a\") " pod="odh-ai-gateway-infra/maas-api-79b4f86cc7-dwtpl" Nov 04 03:35:30.948400 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:35:30.948256 4405 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/58b86d8b-324e-40b7-8862-203885a84f7a-openshift-service-ca\") pod \"maas-api-78d5c4d8b5-4q4db\" (UID: \"58b86d8b-324e-40b7-8862-203885a84f7a\") " pod="odh-ai-gateway-infra/maas-api-78d5c4d8b5-4q4db" Nov 04 03:35:30.948563 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:35:30.948535 4405 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1a63263a-ab89-45ed-82da-2a2c1dbe274a-openshift-service-ca\") pod \"maas-api-79b4f86cc7-dwtpl\" (UID: \"1a63263a-ab89-45ed-82da-2a2c1dbe274a\") " pod="odh-ai-gateway-infra/maas-api-79b4f86cc7-dwtpl" Nov 04 03:35:30.950276 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:35:30.950252 4405 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1a63263a-ab89-45ed-82da-2a2c1dbe274a-metrics-certs\") pod \"maas-api-79b4f86cc7-dwtpl\" (UID: \"1a63263a-ab89-45ed-82da-2a2c1dbe274a\") " pod="odh-ai-gateway-infra/maas-api-79b4f86cc7-dwtpl" Nov 04 03:35:30.950386 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:35:30.950303 4405 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/58b86d8b-324e-40b7-8862-203885a84f7a-metrics-certs\") pod \"maas-api-78d5c4d8b5-4q4db\" (UID: \"58b86d8b-324e-40b7-8862-203885a84f7a\") " pod="odh-ai-gateway-infra/maas-api-78d5c4d8b5-4q4db" Nov 04 03:35:30.950386 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:35:30.950312 4405 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1a63263a-ab89-45ed-82da-2a2c1dbe274a-maas-api-tls\") pod \"maas-api-79b4f86cc7-dwtpl\" (UID: \"1a63263a-ab89-45ed-82da-2a2c1dbe274a\") " pod="odh-ai-gateway-infra/maas-api-79b4f86cc7-dwtpl" Nov 04 03:35:30.950386 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:35:30.950333 4405 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/58b86d8b-324e-40b7-8862-203885a84f7a-maas-api-tls\") pod \"maas-api-78d5c4d8b5-4q4db\" (UID: \"58b86d8b-324e-40b7-8862-203885a84f7a\") " pod="odh-ai-gateway-infra/maas-api-78d5c4d8b5-4q4db" Nov 04 03:35:31.008646 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:35:31.008553 4405 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5mm69\" (UniqueName: \"kubernetes.io/projected/58b86d8b-324e-40b7-8862-203885a84f7a-kube-api-access-5mm69\") pod \"maas-api-78d5c4d8b5-4q4db\" (UID: \"58b86d8b-324e-40b7-8862-203885a84f7a\") " pod="odh-ai-gateway-infra/maas-api-78d5c4d8b5-4q4db" Nov 04 03:35:31.008646 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:35:31.008566 4405 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p87cv\" (UniqueName: \"kubernetes.io/projected/1a63263a-ab89-45ed-82da-2a2c1dbe274a-kube-api-access-p87cv\") pod \"maas-api-79b4f86cc7-dwtpl\" (UID: \"1a63263a-ab89-45ed-82da-2a2c1dbe274a\") " pod="odh-ai-gateway-infra/maas-api-79b4f86cc7-dwtpl" Nov 04 03:35:31.047581 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:35:31.047544 4405 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-79b4f86cc7-dwtpl" Nov 04 03:35:31.080841 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:35:31.080806 4405 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-78d5c4d8b5-4q4db" Nov 04 03:35:31.471848 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:35:31.471811 4405 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-79b4f86cc7-dwtpl"] Nov 04 03:35:31.474343 ip-10-0-134-125 kubenswrapper[4405]: W1104 03:35:31.474317 4405 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1a63263a_ab89_45ed_82da_2a2c1dbe274a.slice/crio-47d58960756efde1ea4a6df578ccf4bb1c8718c65c158d974f3dcafc982239cc WatchSource:0}: Error finding container 47d58960756efde1ea4a6df578ccf4bb1c8718c65c158d974f3dcafc982239cc: Status 404 returned error can't find the container with id 47d58960756efde1ea4a6df578ccf4bb1c8718c65c158d974f3dcafc982239cc Nov 04 03:35:31.474450 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:35:31.474371 4405 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-78d5c4d8b5-4q4db"] Nov 04 03:35:31.514789 ip-10-0-134-125 kubenswrapper[4405]: W1104 03:35:31.514758 4405 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod58b86d8b_324e_40b7_8862_203885a84f7a.slice/crio-04ebb2b0d18ebe443da4a5f5ea3b0f1881b663ad334de08587661a01ff352916 WatchSource:0}: Error finding container 04ebb2b0d18ebe443da4a5f5ea3b0f1881b663ad334de08587661a01ff352916: Status 404 returned error can't find the container with id 04ebb2b0d18ebe443da4a5f5ea3b0f1881b663ad334de08587661a01ff352916 Nov 04 03:35:31.775075 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:35:31.774984 4405 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-78d5c4d8b5-4q4db" event={"ID":"58b86d8b-324e-40b7-8862-203885a84f7a","Type":"ContainerStarted","Data":"04ebb2b0d18ebe443da4a5f5ea3b0f1881b663ad334de08587661a01ff352916"} Nov 04 03:35:31.776209 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:35:31.776175 4405 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-79b4f86cc7-dwtpl" event={"ID":"1a63263a-ab89-45ed-82da-2a2c1dbe274a","Type":"ContainerStarted","Data":"47d58960756efde1ea4a6df578ccf4bb1c8718c65c158d974f3dcafc982239cc"} Nov 04 03:35:32.783946 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:35:32.783850 4405 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-79b4f86cc7-dwtpl" event={"ID":"1a63263a-ab89-45ed-82da-2a2c1dbe274a","Type":"ContainerStarted","Data":"15a0178b98b5ea0c465710481ab9dc42fa7cf62d66fefe0648056bb207a6161d"} Nov 04 03:35:32.784434 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:35:32.783953 4405 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-79b4f86cc7-dwtpl" Nov 04 03:35:32.785716 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:35:32.785687 4405 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-78d5c4d8b5-4q4db" event={"ID":"58b86d8b-324e-40b7-8862-203885a84f7a","Type":"ContainerStarted","Data":"6e491feb35e11fb619803b706496bce4b9c9b5d4fa900bd4f45d7d18cd368609"} Nov 04 03:35:32.785859 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:35:32.785778 4405 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-78d5c4d8b5-4q4db" Nov 04 03:35:32.901836 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:35:32.901760 4405 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-79b4f86cc7-dwtpl" podStartSLOduration=2.003735907 podStartE2EDuration="2.90174522s" totalImagesPullingTime="898.009313ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 03:35:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 03:35:32.900975544 +0000 UTC m=+3779.331675661" watchObservedRunningTime="2026-11-04 03:35:32.90174522 +0000 UTC m=+3779.332445339" Nov 04 03:35:33.019391 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:35:33.019334 4405 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-78d5c4d8b5-4q4db" podStartSLOduration=2.1581052 podStartE2EDuration="3.019319144s" totalImagesPullingTime="861.213944ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 03:35:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 03:35:33.017942787 +0000 UTC m=+3779.448642911" watchObservedRunningTime="2026-11-04 03:35:33.019319144 +0000 UTC m=+3779.450019263" Nov 04 03:35:36.883368 ip-10-0-134-125 kubenswrapper[4405]: E1104 03:35:36.883325 4405 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5f2vb" containerName="extractor" Nov 04 03:35:38.795965 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:35:38.795936 4405 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-78d5c4d8b5-4q4db" Nov 04 03:35:38.796430 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:35:38.796264 4405 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-79b4f86cc7-dwtpl" Nov 04 03:35:38.967589 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:35:38.967547 4405 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-79b4f86cc7-dwtpl"] Nov 04 03:35:38.967833 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:35:38.967800 4405 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-79b4f86cc7-dwtpl" podUID="1a63263a-ab89-45ed-82da-2a2c1dbe274a" containerName="maas-api" containerID="cri-o://15a0178b98b5ea0c465710481ab9dc42fa7cf62d66fefe0648056bb207a6161d" gracePeriod=30 Nov 04 03:35:39.222160 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:35:39.222133 4405 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-79b4f86cc7-dwtpl" Nov 04 03:35:39.332136 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:35:39.332096 4405 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1a63263a-ab89-45ed-82da-2a2c1dbe274a-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1a63263a-ab89-45ed-82da-2a2c1dbe274a-openshift-service-ca\") pod \"1a63263a-ab89-45ed-82da-2a2c1dbe274a\" (UID: \"1a63263a-ab89-45ed-82da-2a2c1dbe274a\") " Nov 04 03:35:39.332323 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:35:39.332141 4405 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1a63263a-ab89-45ed-82da-2a2c1dbe274a-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1a63263a-ab89-45ed-82da-2a2c1dbe274a-maas-api-tls\") pod \"1a63263a-ab89-45ed-82da-2a2c1dbe274a\" (UID: \"1a63263a-ab89-45ed-82da-2a2c1dbe274a\") " Nov 04 03:35:39.332323 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:35:39.332228 4405 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1a63263a-ab89-45ed-82da-2a2c1dbe274a-metrics-certs\" (UniqueName: \"kubernetes.io/secret/1a63263a-ab89-45ed-82da-2a2c1dbe274a-metrics-certs\") pod \"1a63263a-ab89-45ed-82da-2a2c1dbe274a\" (UID: \"1a63263a-ab89-45ed-82da-2a2c1dbe274a\") " Nov 04 03:35:39.332323 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:35:39.332260 4405 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1a63263a-ab89-45ed-82da-2a2c1dbe274a-kube-api-access-p87cv\" (UniqueName: \"kubernetes.io/projected/1a63263a-ab89-45ed-82da-2a2c1dbe274a-kube-api-access-p87cv\") pod \"1a63263a-ab89-45ed-82da-2a2c1dbe274a\" (UID: \"1a63263a-ab89-45ed-82da-2a2c1dbe274a\") " Nov 04 03:35:39.332559 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:35:39.332536 4405 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1a63263a-ab89-45ed-82da-2a2c1dbe274a-openshift-service-ca" pod "1a63263a-ab89-45ed-82da-2a2c1dbe274a" (UID: "1a63263a-ab89-45ed-82da-2a2c1dbe274a"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 03:35:39.334541 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:35:39.334515 4405 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1a63263a-ab89-45ed-82da-2a2c1dbe274a-maas-api-tls" pod "1a63263a-ab89-45ed-82da-2a2c1dbe274a" (UID: "1a63263a-ab89-45ed-82da-2a2c1dbe274a"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 04 03:35:39.334645 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:35:39.334547 4405 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1a63263a-ab89-45ed-82da-2a2c1dbe274a-metrics-certs" pod "1a63263a-ab89-45ed-82da-2a2c1dbe274a" (UID: "1a63263a-ab89-45ed-82da-2a2c1dbe274a"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 04 03:35:39.334799 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:35:39.334780 4405 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1a63263a-ab89-45ed-82da-2a2c1dbe274a-kube-api-access-p87cv" pod "1a63263a-ab89-45ed-82da-2a2c1dbe274a" (UID: "1a63263a-ab89-45ed-82da-2a2c1dbe274a"). InnerVolumeSpecName "kube-api-access-p87cv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 03:35:39.434067 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:35:39.434033 4405 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1a63263a-ab89-45ed-82da-2a2c1dbe274a-metrics-certs\") on node \"ip-10-0-134-125.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:35:39.434067 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:35:39.434064 4405 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p87cv\" (UniqueName: \"kubernetes.io/projected/1a63263a-ab89-45ed-82da-2a2c1dbe274a-kube-api-access-p87cv\") on node \"ip-10-0-134-125.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:35:39.434067 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:35:39.434074 4405 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1a63263a-ab89-45ed-82da-2a2c1dbe274a-openshift-service-ca\") on node \"ip-10-0-134-125.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:35:39.434328 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:35:39.434084 4405 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1a63263a-ab89-45ed-82da-2a2c1dbe274a-maas-api-tls\") on node \"ip-10-0-134-125.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:35:39.822693 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:35:39.822637 4405 generic.go:356] "Generic (PLEG): container finished" podID="1a63263a-ab89-45ed-82da-2a2c1dbe274a" containerID="15a0178b98b5ea0c465710481ab9dc42fa7cf62d66fefe0648056bb207a6161d" exitCode=0 Nov 04 03:35:39.823132 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:35:39.822703 4405 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-79b4f86cc7-dwtpl" Nov 04 03:35:39.823132 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:35:39.822715 4405 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-79b4f86cc7-dwtpl" event={"ID":"1a63263a-ab89-45ed-82da-2a2c1dbe274a","Type":"ContainerDied","Data":"15a0178b98b5ea0c465710481ab9dc42fa7cf62d66fefe0648056bb207a6161d"} Nov 04 03:35:39.823132 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:35:39.822777 4405 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-79b4f86cc7-dwtpl" event={"ID":"1a63263a-ab89-45ed-82da-2a2c1dbe274a","Type":"ContainerDied","Data":"47d58960756efde1ea4a6df578ccf4bb1c8718c65c158d974f3dcafc982239cc"} Nov 04 03:35:39.823132 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:35:39.822808 4405 scope.go:122] "RemoveContainer" containerID="15a0178b98b5ea0c465710481ab9dc42fa7cf62d66fefe0648056bb207a6161d" Nov 04 03:35:39.832255 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:35:39.832234 4405 scope.go:122] "RemoveContainer" containerID="15a0178b98b5ea0c465710481ab9dc42fa7cf62d66fefe0648056bb207a6161d" Nov 04 03:35:39.832552 ip-10-0-134-125 kubenswrapper[4405]: E1104 03:35:39.832529 4405 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"15a0178b98b5ea0c465710481ab9dc42fa7cf62d66fefe0648056bb207a6161d\": container with ID starting with 15a0178b98b5ea0c465710481ab9dc42fa7cf62d66fefe0648056bb207a6161d not found: ID does not exist" containerID="15a0178b98b5ea0c465710481ab9dc42fa7cf62d66fefe0648056bb207a6161d" Nov 04 03:35:39.832642 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:35:39.832566 4405 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"15a0178b98b5ea0c465710481ab9dc42fa7cf62d66fefe0648056bb207a6161d"} err="failed to get container status \"15a0178b98b5ea0c465710481ab9dc42fa7cf62d66fefe0648056bb207a6161d\": rpc error: code = NotFound desc = could not find container \"15a0178b98b5ea0c465710481ab9dc42fa7cf62d66fefe0648056bb207a6161d\": container with ID starting with 15a0178b98b5ea0c465710481ab9dc42fa7cf62d66fefe0648056bb207a6161d not found: ID does not exist" Nov 04 03:35:39.987119 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:35:39.987024 4405 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-79b4f86cc7-dwtpl"] Nov 04 03:35:39.991203 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:35:39.991171 4405 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-79b4f86cc7-dwtpl"] Nov 04 03:35:41.700403 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:35:41.700367 4405 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1a63263a-ab89-45ed-82da-2a2c1dbe274a" path="/var/lib/kubelet/pods/1a63263a-ab89-45ed-82da-2a2c1dbe274a/volumes" Nov 04 03:35:45.849028 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:35:45.848990 4405 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-dffbe0c3-546cf94567-9wrrk"] Nov 04 03:35:45.849469 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:35:45.849374 4405 cpu_manager.go:414] "containerMap: removing container" podUID="1a63263a-ab89-45ed-82da-2a2c1dbe274a" containerName="maas-api" Nov 04 03:35:45.849469 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:35:45.849390 4405 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1a63263a-ab89-45ed-82da-2a2c1dbe274a" containerName="maas-api" Nov 04 03:35:45.849551 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:35:45.849487 4405 memory_manager.go:365] "RemoveStaleState removing state" podUID="1a63263a-ab89-45ed-82da-2a2c1dbe274a" containerName="maas-api" Nov 04 03:35:45.853998 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:35:45.853972 4405 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-dffbe0c3-546cf94567-9wrrk" Nov 04 03:35:45.907611 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:35:45.907317 4405 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-dffbe0c3\"" Nov 04 03:35:45.956925 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:35:45.956893 4405 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-dffbe0c3-dockercfg-rrq5z\"" Nov 04 03:35:45.959282 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:35:45.959253 4405 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-dffbe0c3-546cf94567-9wrrk"] Nov 04 03:35:46.000111 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:35:46.000062 4405 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9vqx4\" (UniqueName: \"kubernetes.io/projected/5f470d61-8a52-46fa-8999-468ac1f72bb9-kube-api-access-9vqx4\") pod \"payload-pre-processing-e2e-ait-dffbe0c3-546cf94567-9wrrk\" (UID: \"5f470d61-8a52-46fa-8999-468ac1f72bb9\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-dffbe0c3-546cf94567-9wrrk" Nov 04 03:35:46.000318 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:35:46.000210 4405 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5f470d61-8a52-46fa-8999-468ac1f72bb9-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-dffbe0c3-546cf94567-9wrrk\" (UID: \"5f470d61-8a52-46fa-8999-468ac1f72bb9\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-dffbe0c3-546cf94567-9wrrk" Nov 04 03:35:46.008626 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:35:46.008573 4405 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-dffbe0c3-7f8d8457b-mmn44"] Nov 04 03:35:46.012133 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:35:46.012113 4405 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-dffbe0c3-7f8d8457b-mmn44" Nov 04 03:35:46.051151 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:35:46.051113 4405 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-dffbe0c3-7f8d8457b-mmn44"] Nov 04 03:35:46.101527 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:35:46.101436 4405 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5f470d61-8a52-46fa-8999-468ac1f72bb9-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-dffbe0c3-546cf94567-9wrrk\" (UID: \"5f470d61-8a52-46fa-8999-468ac1f72bb9\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-dffbe0c3-546cf94567-9wrrk" Nov 04 03:35:46.101719 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:35:46.101526 4405 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9vqx4\" (UniqueName: \"kubernetes.io/projected/5f470d61-8a52-46fa-8999-468ac1f72bb9-kube-api-access-9vqx4\") pod \"payload-pre-processing-e2e-ait-dffbe0c3-546cf94567-9wrrk\" (UID: \"5f470d61-8a52-46fa-8999-468ac1f72bb9\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-dffbe0c3-546cf94567-9wrrk" Nov 04 03:35:46.102275 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:35:46.102250 4405 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5f470d61-8a52-46fa-8999-468ac1f72bb9-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-dffbe0c3-546cf94567-9wrrk\" (UID: \"5f470d61-8a52-46fa-8999-468ac1f72bb9\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-dffbe0c3-546cf94567-9wrrk" Nov 04 03:35:46.164640 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:35:46.164610 4405 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9vqx4\" (UniqueName: \"kubernetes.io/projected/5f470d61-8a52-46fa-8999-468ac1f72bb9-kube-api-access-9vqx4\") pod \"payload-pre-processing-e2e-ait-dffbe0c3-546cf94567-9wrrk\" (UID: \"5f470d61-8a52-46fa-8999-468ac1f72bb9\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-dffbe0c3-546cf94567-9wrrk" Nov 04 03:35:46.167400 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:35:46.167376 4405 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-dffbe0c3-546cf94567-9wrrk" Nov 04 03:35:46.172536 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:35:46.172512 4405 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-dffbe0c3-945fc4bdb-k7t6m"] Nov 04 03:35:46.176318 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:35:46.176297 4405 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-dffbe0c3-945fc4bdb-k7t6m" Nov 04 03:35:46.202428 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:35:46.202394 4405 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8dzf5\" (UniqueName: \"kubernetes.io/projected/b1463858-87ac-44bb-9efa-511c2300997a-kube-api-access-8dzf5\") pod \"payload-processing-e2e-ait-dffbe0c3-7f8d8457b-mmn44\" (UID: \"b1463858-87ac-44bb-9efa-511c2300997a\") " pod="openshift-ingress/payload-processing-e2e-ait-dffbe0c3-7f8d8457b-mmn44" Nov 04 03:35:46.202604 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:35:46.202504 4405 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b1463858-87ac-44bb-9efa-511c2300997a-plugins-config-volume\") pod \"payload-processing-e2e-ait-dffbe0c3-7f8d8457b-mmn44\" (UID: \"b1463858-87ac-44bb-9efa-511c2300997a\") " pod="openshift-ingress/payload-processing-e2e-ait-dffbe0c3-7f8d8457b-mmn44" Nov 04 03:35:46.232826 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:35:46.232792 4405 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-dffbe0c3-serving-cert\"" Nov 04 03:35:46.237799 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:35:46.235399 4405 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-dffbe0c3-945fc4bdb-k7t6m"] Nov 04 03:35:46.303144 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:35:46.303102 4405 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b1463858-87ac-44bb-9efa-511c2300997a-plugins-config-volume\") pod \"payload-processing-e2e-ait-dffbe0c3-7f8d8457b-mmn44\" (UID: \"b1463858-87ac-44bb-9efa-511c2300997a\") " pod="openshift-ingress/payload-processing-e2e-ait-dffbe0c3-7f8d8457b-mmn44" Nov 04 03:35:46.303144 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:35:46.303145 4405 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c2f8d6a2-4ebd-427b-a670-ca449265aab1-openshift-service-ca\") pod \"maas-api-e2e-ait-dffbe0c3-945fc4bdb-k7t6m\" (UID: \"c2f8d6a2-4ebd-427b-a670-ca449265aab1\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-dffbe0c3-945fc4bdb-k7t6m" Nov 04 03:35:46.303433 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:35:46.303165 4405 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9z79r\" (UniqueName: \"kubernetes.io/projected/c2f8d6a2-4ebd-427b-a670-ca449265aab1-kube-api-access-9z79r\") pod \"maas-api-e2e-ait-dffbe0c3-945fc4bdb-k7t6m\" (UID: \"c2f8d6a2-4ebd-427b-a670-ca449265aab1\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-dffbe0c3-945fc4bdb-k7t6m" Nov 04 03:35:46.303433 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:35:46.303322 4405 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c2f8d6a2-4ebd-427b-a670-ca449265aab1-metrics-certs\") pod \"maas-api-e2e-ait-dffbe0c3-945fc4bdb-k7t6m\" (UID: \"c2f8d6a2-4ebd-427b-a670-ca449265aab1\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-dffbe0c3-945fc4bdb-k7t6m" Nov 04 03:35:46.303554 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:35:46.303439 4405 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8dzf5\" (UniqueName: \"kubernetes.io/projected/b1463858-87ac-44bb-9efa-511c2300997a-kube-api-access-8dzf5\") pod \"payload-processing-e2e-ait-dffbe0c3-7f8d8457b-mmn44\" (UID: \"b1463858-87ac-44bb-9efa-511c2300997a\") " pod="openshift-ingress/payload-processing-e2e-ait-dffbe0c3-7f8d8457b-mmn44" Nov 04 03:35:46.303554 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:35:46.303513 4405 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c2f8d6a2-4ebd-427b-a670-ca449265aab1-maas-api-tls\") pod \"maas-api-e2e-ait-dffbe0c3-945fc4bdb-k7t6m\" (UID: \"c2f8d6a2-4ebd-427b-a670-ca449265aab1\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-dffbe0c3-945fc4bdb-k7t6m" Nov 04 03:35:46.303954 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:35:46.303935 4405 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b1463858-87ac-44bb-9efa-511c2300997a-plugins-config-volume\") pod \"payload-processing-e2e-ait-dffbe0c3-7f8d8457b-mmn44\" (UID: \"b1463858-87ac-44bb-9efa-511c2300997a\") " pod="openshift-ingress/payload-processing-e2e-ait-dffbe0c3-7f8d8457b-mmn44" Nov 04 03:35:46.367124 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:35:46.367045 4405 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8dzf5\" (UniqueName: \"kubernetes.io/projected/b1463858-87ac-44bb-9efa-511c2300997a-kube-api-access-8dzf5\") pod \"payload-processing-e2e-ait-dffbe0c3-7f8d8457b-mmn44\" (UID: \"b1463858-87ac-44bb-9efa-511c2300997a\") " pod="openshift-ingress/payload-processing-e2e-ait-dffbe0c3-7f8d8457b-mmn44" Nov 04 03:35:46.404750 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:35:46.404713 4405 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c2f8d6a2-4ebd-427b-a670-ca449265aab1-maas-api-tls\") pod \"maas-api-e2e-ait-dffbe0c3-945fc4bdb-k7t6m\" (UID: \"c2f8d6a2-4ebd-427b-a670-ca449265aab1\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-dffbe0c3-945fc4bdb-k7t6m" Nov 04 03:35:46.404936 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:35:46.404757 4405 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c2f8d6a2-4ebd-427b-a670-ca449265aab1-openshift-service-ca\") pod \"maas-api-e2e-ait-dffbe0c3-945fc4bdb-k7t6m\" (UID: \"c2f8d6a2-4ebd-427b-a670-ca449265aab1\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-dffbe0c3-945fc4bdb-k7t6m" Nov 04 03:35:46.404936 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:35:46.404778 4405 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9z79r\" (UniqueName: \"kubernetes.io/projected/c2f8d6a2-4ebd-427b-a670-ca449265aab1-kube-api-access-9z79r\") pod \"maas-api-e2e-ait-dffbe0c3-945fc4bdb-k7t6m\" (UID: \"c2f8d6a2-4ebd-427b-a670-ca449265aab1\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-dffbe0c3-945fc4bdb-k7t6m" Nov 04 03:35:46.404936 ip-10-0-134-125 kubenswrapper[4405]: E1104 03:35:46.404867 4405 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ait-dffbe0c3-serving-cert: secret "maas-api-e2e-ait-dffbe0c3-serving-cert" not found Nov 04 03:35:46.405054 ip-10-0-134-125 kubenswrapper[4405]: E1104 03:35:46.404939 4405 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c2f8d6a2-4ebd-427b-a670-ca449265aab1-maas-api-tls podName:c2f8d6a2-4ebd-427b-a670-ca449265aab1 nodeName:}" failed. No retries permitted until 2026-11-04 03:35:46.904917844 +0000 UTC m=+3793.335617956 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/c2f8d6a2-4ebd-427b-a670-ca449265aab1-maas-api-tls") pod "maas-api-e2e-ait-dffbe0c3-945fc4bdb-k7t6m" (UID: "c2f8d6a2-4ebd-427b-a670-ca449265aab1") : secret "maas-api-e2e-ait-dffbe0c3-serving-cert" not found Nov 04 03:35:46.405054 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:35:46.404959 4405 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c2f8d6a2-4ebd-427b-a670-ca449265aab1-metrics-certs\") pod \"maas-api-e2e-ait-dffbe0c3-945fc4bdb-k7t6m\" (UID: \"c2f8d6a2-4ebd-427b-a670-ca449265aab1\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-dffbe0c3-945fc4bdb-k7t6m" Nov 04 03:35:46.405460 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:35:46.405435 4405 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c2f8d6a2-4ebd-427b-a670-ca449265aab1-openshift-service-ca\") pod \"maas-api-e2e-ait-dffbe0c3-945fc4bdb-k7t6m\" (UID: \"c2f8d6a2-4ebd-427b-a670-ca449265aab1\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-dffbe0c3-945fc4bdb-k7t6m" Nov 04 03:35:46.407512 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:35:46.407488 4405 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c2f8d6a2-4ebd-427b-a670-ca449265aab1-metrics-certs\") pod \"maas-api-e2e-ait-dffbe0c3-945fc4bdb-k7t6m\" (UID: \"c2f8d6a2-4ebd-427b-a670-ca449265aab1\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-dffbe0c3-945fc4bdb-k7t6m" Nov 04 03:35:46.467656 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:35:46.467624 4405 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9z79r\" (UniqueName: \"kubernetes.io/projected/c2f8d6a2-4ebd-427b-a670-ca449265aab1-kube-api-access-9z79r\") pod \"maas-api-e2e-ait-dffbe0c3-945fc4bdb-k7t6m\" (UID: \"c2f8d6a2-4ebd-427b-a670-ca449265aab1\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-dffbe0c3-945fc4bdb-k7t6m" Nov 04 03:35:46.542235 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:35:46.542189 4405 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-dffbe0c3-546cf94567-9wrrk"] Nov 04 03:35:46.544555 ip-10-0-134-125 kubenswrapper[4405]: W1104 03:35:46.544517 4405 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5f470d61_8a52_46fa_8999_468ac1f72bb9.slice/crio-4549cf6f1547d5f93ae6acd33af8c6dc036e9028dcb9b7f04c1736be19171f6a WatchSource:0}: Error finding container 4549cf6f1547d5f93ae6acd33af8c6dc036e9028dcb9b7f04c1736be19171f6a: Status 404 returned error can't find the container with id 4549cf6f1547d5f93ae6acd33af8c6dc036e9028dcb9b7f04c1736be19171f6a Nov 04 03:35:46.625541 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:35:46.625464 4405 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-dffbe0c3-7f8d8457b-mmn44" Nov 04 03:35:46.857854 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:35:46.857817 4405 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-dffbe0c3-546cf94567-9wrrk" event={"ID":"5f470d61-8a52-46fa-8999-468ac1f72bb9","Type":"ContainerStarted","Data":"677fb4d50e7f8f8bbabbed608a67303baee48fc59570ed7f86417f2e1a62a1b4"} Nov 04 03:35:46.858305 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:35:46.857877 4405 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-dffbe0c3-546cf94567-9wrrk" event={"ID":"5f470d61-8a52-46fa-8999-468ac1f72bb9","Type":"ContainerStarted","Data":"4549cf6f1547d5f93ae6acd33af8c6dc036e9028dcb9b7f04c1736be19171f6a"} Nov 04 03:35:46.858305 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:35:46.857969 4405 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-dffbe0c3-546cf94567-9wrrk" Nov 04 03:35:46.910852 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:35:46.910753 4405 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c2f8d6a2-4ebd-427b-a670-ca449265aab1-maas-api-tls\") pod \"maas-api-e2e-ait-dffbe0c3-945fc4bdb-k7t6m\" (UID: \"c2f8d6a2-4ebd-427b-a670-ca449265aab1\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-dffbe0c3-945fc4bdb-k7t6m" Nov 04 03:35:46.913529 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:35:46.913504 4405 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c2f8d6a2-4ebd-427b-a670-ca449265aab1-maas-api-tls\") pod \"maas-api-e2e-ait-dffbe0c3-945fc4bdb-k7t6m\" (UID: \"c2f8d6a2-4ebd-427b-a670-ca449265aab1\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-dffbe0c3-945fc4bdb-k7t6m" Nov 04 03:35:46.975112 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:35:46.975060 4405 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ait-dffbe0c3-546cf94567-9wrrk" podStartSLOduration=1.975045464 podStartE2EDuration="1.975045464s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 03:35:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 03:35:46.973688738 +0000 UTC m=+3793.404388852" watchObservedRunningTime="2026-11-04 03:35:46.975045464 +0000 UTC m=+3793.405745583" Nov 04 03:35:46.997879 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:35:46.997858 4405 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-dffbe0c3-7f8d8457b-mmn44"] Nov 04 03:35:47.000003 ip-10-0-134-125 kubenswrapper[4405]: W1104 03:35:46.999978 4405 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb1463858_87ac_44bb_9efa_511c2300997a.slice/crio-1f330bafb657ad586203a6e8c0fea8e6d6eff1349f56ac6fc9237ba34561b21d WatchSource:0}: Error finding container 1f330bafb657ad586203a6e8c0fea8e6d6eff1349f56ac6fc9237ba34561b21d: Status 404 returned error can't find the container with id 1f330bafb657ad586203a6e8c0fea8e6d6eff1349f56ac6fc9237ba34561b21d Nov 04 03:35:47.108826 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:35:47.108779 4405 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-dffbe0c3-945fc4bdb-k7t6m" Nov 04 03:35:47.489706 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:35:47.489659 4405 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-dffbe0c3-945fc4bdb-k7t6m"] Nov 04 03:35:47.491769 ip-10-0-134-125 kubenswrapper[4405]: W1104 03:35:47.491737 4405 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc2f8d6a2_4ebd_427b_a670_ca449265aab1.slice/crio-05fd931caacf6b2227804389d6968a80bf5ea28129c1f457f914f5f7110fe36e WatchSource:0}: Error finding container 05fd931caacf6b2227804389d6968a80bf5ea28129c1f457f914f5f7110fe36e: Status 404 returned error can't find the container with id 05fd931caacf6b2227804389d6968a80bf5ea28129c1f457f914f5f7110fe36e Nov 04 03:35:47.864002 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:35:47.863948 4405 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-dffbe0c3-7f8d8457b-mmn44" event={"ID":"b1463858-87ac-44bb-9efa-511c2300997a","Type":"ContainerStarted","Data":"4808895e819ac3b18afb950ff8c642060a0133fbbbc75d725a780f078d8b5afc"} Nov 04 03:35:47.864510 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:35:47.864019 4405 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-dffbe0c3-7f8d8457b-mmn44" event={"ID":"b1463858-87ac-44bb-9efa-511c2300997a","Type":"ContainerStarted","Data":"1f330bafb657ad586203a6e8c0fea8e6d6eff1349f56ac6fc9237ba34561b21d"} Nov 04 03:35:47.864510 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:35:47.864127 4405 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ait-dffbe0c3-7f8d8457b-mmn44" Nov 04 03:35:47.865256 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:35:47.865224 4405 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-dffbe0c3-945fc4bdb-k7t6m" event={"ID":"c2f8d6a2-4ebd-427b-a670-ca449265aab1","Type":"ContainerStarted","Data":"05fd931caacf6b2227804389d6968a80bf5ea28129c1f457f914f5f7110fe36e"} Nov 04 03:35:47.980811 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:35:47.980750 4405 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ait-dffbe0c3-7f8d8457b-mmn44" podStartSLOduration=2.980736373 podStartE2EDuration="2.980736373s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 03:35:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 03:35:47.979260837 +0000 UTC m=+3794.409960958" watchObservedRunningTime="2026-11-04 03:35:47.980736373 +0000 UTC m=+3794.411436493" Nov 04 03:35:48.871380 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:35:48.871329 4405 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-dffbe0c3-945fc4bdb-k7t6m" event={"ID":"c2f8d6a2-4ebd-427b-a670-ca449265aab1","Type":"ContainerStarted","Data":"bd25226727d5d5bc06aa84e2d940e8eebc48881b4d841c3e0e1bb075c1166844"} Nov 04 03:35:48.871914 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:35:48.871403 4405 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-dffbe0c3-945fc4bdb-k7t6m" Nov 04 03:35:48.988817 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:35:48.988753 4405 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-dffbe0c3-945fc4bdb-k7t6m" podStartSLOduration=2.156926905 podStartE2EDuration="2.988738829s" totalImagesPullingTime="831.811924ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 03:35:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 03:35:48.988039961 +0000 UTC m=+3795.418740081" watchObservedRunningTime="2026-11-04 03:35:48.988738829 +0000 UTC m=+3795.419438949" Nov 04 03:35:54.522275 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:35:54.522243 4405 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-bl5mg"] Nov 04 03:35:54.884219 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:35:54.884180 4405 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-dffbe0c3-945fc4bdb-k7t6m" Nov 04 03:35:57.868708 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:35:57.868657 4405 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-dffbe0c3-546cf94567-9wrrk" Nov 04 03:35:58.874764 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:35:58.874733 4405 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ait-dffbe0c3-7f8d8457b-mmn44" Nov 04 03:36:06.883732 ip-10-0-134-125 kubenswrapper[4405]: E1104 03:36:06.883690 4405 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5f2vb" containerName="extractor" Nov 04 03:36:30.113351 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:36:30.113311 4405 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-dffbe0c3-7f8d8457b-mmn44"] Nov 04 03:36:30.222360 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:36:30.222324 4405 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-dffbe0c3-546cf94567-9wrrk"] Nov 04 03:36:35.114324 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:36:35.114253 4405 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ait-dffbe0c3-7f8d8457b-mmn44" podUID="b1463858-87ac-44bb-9efa-511c2300997a" containerName="payload-processing" containerID="cri-o://4808895e819ac3b18afb950ff8c642060a0133fbbbc75d725a780f078d8b5afc" gracePeriod=25 Nov 04 03:36:35.223703 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:36:35.223594 4405 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ait-dffbe0c3-546cf94567-9wrrk" podUID="5f470d61-8a52-46fa-8999-468ac1f72bb9" containerName="payload-pre-processing" containerID="cri-o://677fb4d50e7f8f8bbabbed608a67303baee48fc59570ed7f86417f2e1a62a1b4" gracePeriod=25 Nov 04 03:36:35.379127 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:36:35.379062 4405 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-dffbe0c3-7f8d8457b-mmn44" Nov 04 03:36:35.473496 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:36:35.473471 4405 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-dffbe0c3-546cf94567-9wrrk" Nov 04 03:36:35.489072 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:36:35.489046 4405 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b1463858-87ac-44bb-9efa-511c2300997a-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b1463858-87ac-44bb-9efa-511c2300997a-plugins-config-volume\") pod \"b1463858-87ac-44bb-9efa-511c2300997a\" (UID: \"b1463858-87ac-44bb-9efa-511c2300997a\") " Nov 04 03:36:35.489241 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:36:35.489089 4405 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b1463858-87ac-44bb-9efa-511c2300997a-kube-api-access-8dzf5\" (UniqueName: \"kubernetes.io/projected/b1463858-87ac-44bb-9efa-511c2300997a-kube-api-access-8dzf5\") pod \"b1463858-87ac-44bb-9efa-511c2300997a\" (UID: \"b1463858-87ac-44bb-9efa-511c2300997a\") " Nov 04 03:36:35.489241 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:36:35.489170 4405 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5f470d61-8a52-46fa-8999-468ac1f72bb9-kube-api-access-9vqx4\" (UniqueName: \"kubernetes.io/projected/5f470d61-8a52-46fa-8999-468ac1f72bb9-kube-api-access-9vqx4\") pod \"5f470d61-8a52-46fa-8999-468ac1f72bb9\" (UID: \"5f470d61-8a52-46fa-8999-468ac1f72bb9\") " Nov 04 03:36:35.489241 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:36:35.489207 4405 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5f470d61-8a52-46fa-8999-468ac1f72bb9-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5f470d61-8a52-46fa-8999-468ac1f72bb9-plugins-config-volume\") pod \"5f470d61-8a52-46fa-8999-468ac1f72bb9\" (UID: \"5f470d61-8a52-46fa-8999-468ac1f72bb9\") " Nov 04 03:36:35.489451 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:36:35.489424 4405 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b1463858-87ac-44bb-9efa-511c2300997a-plugins-config-volume" pod "b1463858-87ac-44bb-9efa-511c2300997a" (UID: "b1463858-87ac-44bb-9efa-511c2300997a"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 03:36:35.489655 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:36:35.489638 4405 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b1463858-87ac-44bb-9efa-511c2300997a-plugins-config-volume\") on node \"ip-10-0-134-125.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:36:35.489746 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:36:35.489714 4405 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5f470d61-8a52-46fa-8999-468ac1f72bb9-plugins-config-volume" pod "5f470d61-8a52-46fa-8999-468ac1f72bb9" (UID: "5f470d61-8a52-46fa-8999-468ac1f72bb9"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 03:36:35.491419 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:36:35.491394 4405 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5f470d61-8a52-46fa-8999-468ac1f72bb9-kube-api-access-9vqx4" pod "5f470d61-8a52-46fa-8999-468ac1f72bb9" (UID: "5f470d61-8a52-46fa-8999-468ac1f72bb9"). InnerVolumeSpecName "kube-api-access-9vqx4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 03:36:35.491543 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:36:35.491507 4405 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b1463858-87ac-44bb-9efa-511c2300997a-kube-api-access-8dzf5" pod "b1463858-87ac-44bb-9efa-511c2300997a" (UID: "b1463858-87ac-44bb-9efa-511c2300997a"). InnerVolumeSpecName "kube-api-access-8dzf5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 03:36:35.591152 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:36:35.591109 4405 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9vqx4\" (UniqueName: \"kubernetes.io/projected/5f470d61-8a52-46fa-8999-468ac1f72bb9-kube-api-access-9vqx4\") on node \"ip-10-0-134-125.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:36:35.591152 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:36:35.591144 4405 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5f470d61-8a52-46fa-8999-468ac1f72bb9-plugins-config-volume\") on node \"ip-10-0-134-125.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:36:35.591152 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:36:35.591153 4405 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8dzf5\" (UniqueName: \"kubernetes.io/projected/b1463858-87ac-44bb-9efa-511c2300997a-kube-api-access-8dzf5\") on node \"ip-10-0-134-125.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:36:35.701138 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:36:35.701051 4405 scope.go:122] "RemoveContainer" containerID="b2b6a6dd444e4e284b324d7726050cb2a15e986dc3800e6e05b24649729a9292" Nov 04 03:36:36.113752 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:36:36.113716 4405 generic.go:356] "Generic (PLEG): container finished" podID="b1463858-87ac-44bb-9efa-511c2300997a" containerID="4808895e819ac3b18afb950ff8c642060a0133fbbbc75d725a780f078d8b5afc" exitCode=0 Nov 04 03:36:36.113953 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:36:36.113763 4405 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-dffbe0c3-7f8d8457b-mmn44" Nov 04 03:36:36.113953 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:36:36.113800 4405 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-dffbe0c3-7f8d8457b-mmn44" event={"ID":"b1463858-87ac-44bb-9efa-511c2300997a","Type":"ContainerDied","Data":"4808895e819ac3b18afb950ff8c642060a0133fbbbc75d725a780f078d8b5afc"} Nov 04 03:36:36.113953 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:36:36.113850 4405 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-dffbe0c3-7f8d8457b-mmn44" event={"ID":"b1463858-87ac-44bb-9efa-511c2300997a","Type":"ContainerDied","Data":"1f330bafb657ad586203a6e8c0fea8e6d6eff1349f56ac6fc9237ba34561b21d"} Nov 04 03:36:36.113953 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:36:36.113884 4405 scope.go:122] "RemoveContainer" containerID="4808895e819ac3b18afb950ff8c642060a0133fbbbc75d725a780f078d8b5afc" Nov 04 03:36:36.115249 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:36:36.115189 4405 generic.go:356] "Generic (PLEG): container finished" podID="5f470d61-8a52-46fa-8999-468ac1f72bb9" containerID="677fb4d50e7f8f8bbabbed608a67303baee48fc59570ed7f86417f2e1a62a1b4" exitCode=0 Nov 04 03:36:36.115249 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:36:36.115239 4405 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-dffbe0c3-546cf94567-9wrrk" Nov 04 03:36:36.115693 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:36:36.115267 4405 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-dffbe0c3-546cf94567-9wrrk" event={"ID":"5f470d61-8a52-46fa-8999-468ac1f72bb9","Type":"ContainerDied","Data":"677fb4d50e7f8f8bbabbed608a67303baee48fc59570ed7f86417f2e1a62a1b4"} Nov 04 03:36:36.115693 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:36:36.115306 4405 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-dffbe0c3-546cf94567-9wrrk" event={"ID":"5f470d61-8a52-46fa-8999-468ac1f72bb9","Type":"ContainerDied","Data":"4549cf6f1547d5f93ae6acd33af8c6dc036e9028dcb9b7f04c1736be19171f6a"} Nov 04 03:36:36.122851 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:36:36.122834 4405 scope.go:122] "RemoveContainer" containerID="4808895e819ac3b18afb950ff8c642060a0133fbbbc75d725a780f078d8b5afc" Nov 04 03:36:36.123149 ip-10-0-134-125 kubenswrapper[4405]: E1104 03:36:36.123132 4405 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4808895e819ac3b18afb950ff8c642060a0133fbbbc75d725a780f078d8b5afc\": container with ID starting with 4808895e819ac3b18afb950ff8c642060a0133fbbbc75d725a780f078d8b5afc not found: ID does not exist" containerID="4808895e819ac3b18afb950ff8c642060a0133fbbbc75d725a780f078d8b5afc" Nov 04 03:36:36.123199 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:36:36.123159 4405 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4808895e819ac3b18afb950ff8c642060a0133fbbbc75d725a780f078d8b5afc"} err="failed to get container status \"4808895e819ac3b18afb950ff8c642060a0133fbbbc75d725a780f078d8b5afc\": rpc error: code = NotFound desc = could not find container \"4808895e819ac3b18afb950ff8c642060a0133fbbbc75d725a780f078d8b5afc\": container with ID starting with 4808895e819ac3b18afb950ff8c642060a0133fbbbc75d725a780f078d8b5afc not found: ID does not exist" Nov 04 03:36:36.123199 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:36:36.123176 4405 scope.go:122] "RemoveContainer" containerID="677fb4d50e7f8f8bbabbed608a67303baee48fc59570ed7f86417f2e1a62a1b4" Nov 04 03:36:36.134182 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:36:36.134165 4405 scope.go:122] "RemoveContainer" containerID="677fb4d50e7f8f8bbabbed608a67303baee48fc59570ed7f86417f2e1a62a1b4" Nov 04 03:36:36.134448 ip-10-0-134-125 kubenswrapper[4405]: E1104 03:36:36.134428 4405 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"677fb4d50e7f8f8bbabbed608a67303baee48fc59570ed7f86417f2e1a62a1b4\": container with ID starting with 677fb4d50e7f8f8bbabbed608a67303baee48fc59570ed7f86417f2e1a62a1b4 not found: ID does not exist" containerID="677fb4d50e7f8f8bbabbed608a67303baee48fc59570ed7f86417f2e1a62a1b4" Nov 04 03:36:36.134515 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:36:36.134480 4405 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"677fb4d50e7f8f8bbabbed608a67303baee48fc59570ed7f86417f2e1a62a1b4"} err="failed to get container status \"677fb4d50e7f8f8bbabbed608a67303baee48fc59570ed7f86417f2e1a62a1b4\": rpc error: code = NotFound desc = could not find container \"677fb4d50e7f8f8bbabbed608a67303baee48fc59570ed7f86417f2e1a62a1b4\": container with ID starting with 677fb4d50e7f8f8bbabbed608a67303baee48fc59570ed7f86417f2e1a62a1b4 not found: ID does not exist" Nov 04 03:36:36.277341 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:36:36.277297 4405 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-dffbe0c3-546cf94567-9wrrk"] Nov 04 03:36:36.328095 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:36:36.328059 4405 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-dffbe0c3-546cf94567-9wrrk"] Nov 04 03:36:36.330547 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:36:36.330511 4405 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-dffbe0c3-945fc4bdb-k7t6m"] Nov 04 03:36:36.330810 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:36:36.330779 4405 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-dffbe0c3-945fc4bdb-k7t6m" podUID="c2f8d6a2-4ebd-427b-a670-ca449265aab1" containerName="maas-api" containerID="cri-o://bd25226727d5d5bc06aa84e2d940e8eebc48881b4d841c3e0e1bb075c1166844" gracePeriod=30 Nov 04 03:36:36.394890 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:36:36.394717 4405 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-dffbe0c3-7f8d8457b-mmn44"] Nov 04 03:36:36.397453 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:36:36.397417 4405 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-dffbe0c3-7f8d8457b-mmn44"] Nov 04 03:36:36.583132 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:36:36.583101 4405 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-dffbe0c3-945fc4bdb-k7t6m" Nov 04 03:36:36.601305 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:36:36.601276 4405 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c2f8d6a2-4ebd-427b-a670-ca449265aab1-kube-api-access-9z79r\" (UniqueName: \"kubernetes.io/projected/c2f8d6a2-4ebd-427b-a670-ca449265aab1-kube-api-access-9z79r\") pod \"c2f8d6a2-4ebd-427b-a670-ca449265aab1\" (UID: \"c2f8d6a2-4ebd-427b-a670-ca449265aab1\") " Nov 04 03:36:36.601431 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:36:36.601319 4405 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c2f8d6a2-4ebd-427b-a670-ca449265aab1-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c2f8d6a2-4ebd-427b-a670-ca449265aab1-maas-api-tls\") pod \"c2f8d6a2-4ebd-427b-a670-ca449265aab1\" (UID: \"c2f8d6a2-4ebd-427b-a670-ca449265aab1\") " Nov 04 03:36:36.601431 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:36:36.601342 4405 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c2f8d6a2-4ebd-427b-a670-ca449265aab1-metrics-certs\" (UniqueName: \"kubernetes.io/secret/c2f8d6a2-4ebd-427b-a670-ca449265aab1-metrics-certs\") pod \"c2f8d6a2-4ebd-427b-a670-ca449265aab1\" (UID: \"c2f8d6a2-4ebd-427b-a670-ca449265aab1\") " Nov 04 03:36:36.601431 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:36:36.601417 4405 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c2f8d6a2-4ebd-427b-a670-ca449265aab1-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c2f8d6a2-4ebd-427b-a670-ca449265aab1-openshift-service-ca\") pod \"c2f8d6a2-4ebd-427b-a670-ca449265aab1\" (UID: \"c2f8d6a2-4ebd-427b-a670-ca449265aab1\") " Nov 04 03:36:36.601881 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:36:36.601849 4405 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c2f8d6a2-4ebd-427b-a670-ca449265aab1-openshift-service-ca" pod "c2f8d6a2-4ebd-427b-a670-ca449265aab1" (UID: "c2f8d6a2-4ebd-427b-a670-ca449265aab1"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 03:36:36.603748 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:36:36.603724 4405 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c2f8d6a2-4ebd-427b-a670-ca449265aab1-metrics-certs" pod "c2f8d6a2-4ebd-427b-a670-ca449265aab1" (UID: "c2f8d6a2-4ebd-427b-a670-ca449265aab1"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 04 03:36:36.603879 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:36:36.603859 4405 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c2f8d6a2-4ebd-427b-a670-ca449265aab1-kube-api-access-9z79r" pod "c2f8d6a2-4ebd-427b-a670-ca449265aab1" (UID: "c2f8d6a2-4ebd-427b-a670-ca449265aab1"). InnerVolumeSpecName "kube-api-access-9z79r". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 03:36:36.604368 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:36:36.604345 4405 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c2f8d6a2-4ebd-427b-a670-ca449265aab1-maas-api-tls" pod "c2f8d6a2-4ebd-427b-a670-ca449265aab1" (UID: "c2f8d6a2-4ebd-427b-a670-ca449265aab1"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 04 03:36:36.702527 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:36:36.702437 4405 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c2f8d6a2-4ebd-427b-a670-ca449265aab1-openshift-service-ca\") on node \"ip-10-0-134-125.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:36:36.702527 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:36:36.702477 4405 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9z79r\" (UniqueName: \"kubernetes.io/projected/c2f8d6a2-4ebd-427b-a670-ca449265aab1-kube-api-access-9z79r\") on node \"ip-10-0-134-125.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:36:36.702527 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:36:36.702493 4405 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c2f8d6a2-4ebd-427b-a670-ca449265aab1-maas-api-tls\") on node \"ip-10-0-134-125.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:36:36.702527 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:36:36.702507 4405 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c2f8d6a2-4ebd-427b-a670-ca449265aab1-metrics-certs\") on node \"ip-10-0-134-125.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:36:36.883205 ip-10-0-134-125 kubenswrapper[4405]: E1104 03:36:36.883166 4405 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5f2vb" containerName="extractor" Nov 04 03:36:37.120745 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:36:37.120710 4405 generic.go:356] "Generic (PLEG): container finished" podID="c2f8d6a2-4ebd-427b-a670-ca449265aab1" containerID="bd25226727d5d5bc06aa84e2d940e8eebc48881b4d841c3e0e1bb075c1166844" exitCode=0 Nov 04 03:36:37.121204 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:36:37.120764 4405 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-dffbe0c3-945fc4bdb-k7t6m" Nov 04 03:36:37.121204 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:36:37.120789 4405 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-dffbe0c3-945fc4bdb-k7t6m" event={"ID":"c2f8d6a2-4ebd-427b-a670-ca449265aab1","Type":"ContainerDied","Data":"bd25226727d5d5bc06aa84e2d940e8eebc48881b4d841c3e0e1bb075c1166844"} Nov 04 03:36:37.121204 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:36:37.120845 4405 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-dffbe0c3-945fc4bdb-k7t6m" event={"ID":"c2f8d6a2-4ebd-427b-a670-ca449265aab1","Type":"ContainerDied","Data":"05fd931caacf6b2227804389d6968a80bf5ea28129c1f457f914f5f7110fe36e"} Nov 04 03:36:37.121204 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:36:37.120876 4405 scope.go:122] "RemoveContainer" containerID="bd25226727d5d5bc06aa84e2d940e8eebc48881b4d841c3e0e1bb075c1166844" Nov 04 03:36:37.133538 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:36:37.133520 4405 scope.go:122] "RemoveContainer" containerID="bd25226727d5d5bc06aa84e2d940e8eebc48881b4d841c3e0e1bb075c1166844" Nov 04 03:36:37.133929 ip-10-0-134-125 kubenswrapper[4405]: E1104 03:36:37.133902 4405 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bd25226727d5d5bc06aa84e2d940e8eebc48881b4d841c3e0e1bb075c1166844\": container with ID starting with bd25226727d5d5bc06aa84e2d940e8eebc48881b4d841c3e0e1bb075c1166844 not found: ID does not exist" containerID="bd25226727d5d5bc06aa84e2d940e8eebc48881b4d841c3e0e1bb075c1166844" Nov 04 03:36:37.133991 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:36:37.133940 4405 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bd25226727d5d5bc06aa84e2d940e8eebc48881b4d841c3e0e1bb075c1166844"} err="failed to get container status \"bd25226727d5d5bc06aa84e2d940e8eebc48881b4d841c3e0e1bb075c1166844\": rpc error: code = NotFound desc = could not find container \"bd25226727d5d5bc06aa84e2d940e8eebc48881b4d841c3e0e1bb075c1166844\": container with ID starting with bd25226727d5d5bc06aa84e2d940e8eebc48881b4d841c3e0e1bb075c1166844 not found: ID does not exist" Nov 04 03:36:37.287344 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:36:37.287300 4405 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-dffbe0c3-945fc4bdb-k7t6m"] Nov 04 03:36:37.291225 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:36:37.291187 4405 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-dffbe0c3-945fc4bdb-k7t6m"] Nov 04 03:36:37.699649 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:36:37.699613 4405 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5f470d61-8a52-46fa-8999-468ac1f72bb9" path="/var/lib/kubelet/pods/5f470d61-8a52-46fa-8999-468ac1f72bb9/volumes" Nov 04 03:36:37.700046 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:36:37.700032 4405 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b1463858-87ac-44bb-9efa-511c2300997a" path="/var/lib/kubelet/pods/b1463858-87ac-44bb-9efa-511c2300997a/volumes" Nov 04 03:36:37.700396 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:36:37.700383 4405 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c2f8d6a2-4ebd-427b-a670-ca449265aab1" path="/var/lib/kubelet/pods/c2f8d6a2-4ebd-427b-a670-ca449265aab1/volumes" Nov 04 03:36:41.368637 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:36:41.368590 4405 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-authcfg-http-cc2474-openshift-default-7f6458856f-qw52z"] Nov 04 03:36:41.369125 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:36:41.369105 4405 cpu_manager.go:414] "containerMap: removing container" podUID="5f470d61-8a52-46fa-8999-468ac1f72bb9" containerName="payload-pre-processing" Nov 04 03:36:41.369218 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:36:41.369126 4405 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5f470d61-8a52-46fa-8999-468ac1f72bb9" containerName="payload-pre-processing" Nov 04 03:36:41.369218 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:36:41.369137 4405 cpu_manager.go:414] "containerMap: removing container" podUID="c2f8d6a2-4ebd-427b-a670-ca449265aab1" containerName="maas-api" Nov 04 03:36:41.369218 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:36:41.369143 4405 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c2f8d6a2-4ebd-427b-a670-ca449265aab1" containerName="maas-api" Nov 04 03:36:41.369218 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:36:41.369168 4405 cpu_manager.go:414] "containerMap: removing container" podUID="b1463858-87ac-44bb-9efa-511c2300997a" containerName="payload-processing" Nov 04 03:36:41.369218 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:36:41.369173 4405 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b1463858-87ac-44bb-9efa-511c2300997a" containerName="payload-processing" Nov 04 03:36:41.369414 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:36:41.369230 4405 memory_manager.go:365] "RemoveStaleState removing state" podUID="b1463858-87ac-44bb-9efa-511c2300997a" containerName="payload-processing" Nov 04 03:36:41.369414 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:36:41.369243 4405 memory_manager.go:365] "RemoveStaleState removing state" podUID="5f470d61-8a52-46fa-8999-468ac1f72bb9" containerName="payload-pre-processing" Nov 04 03:36:41.369414 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:36:41.369249 4405 memory_manager.go:365] "RemoveStaleState removing state" podUID="c2f8d6a2-4ebd-427b-a670-ca449265aab1" containerName="maas-api" Nov 04 03:36:41.372346 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:36:41.372326 4405 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-authcfg-http-cc2474-openshift-default-7f6458856f-qw52z" Nov 04 03:36:41.424951 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:36:41.424917 4405 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-authcfg-http-cc2474-openshift-default-dockercfg-rl5v7\"" Nov 04 03:36:41.449648 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:36:41.449613 4405 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rhjg9\" (UniqueName: \"kubernetes.io/projected/ab0796a3-e6f1-441a-b0f5-4d91c32c92d5-kube-api-access-rhjg9\") pod \"e2e-authcfg-http-cc2474-openshift-default-7f6458856f-qw52z\" (UID: \"ab0796a3-e6f1-441a-b0f5-4d91c32c92d5\") " pod="openshift-ingress/e2e-authcfg-http-cc2474-openshift-default-7f6458856f-qw52z" Nov 04 03:36:41.449648 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:36:41.449652 4405 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ab0796a3-e6f1-441a-b0f5-4d91c32c92d5-istio-token\") pod \"e2e-authcfg-http-cc2474-openshift-default-7f6458856f-qw52z\" (UID: \"ab0796a3-e6f1-441a-b0f5-4d91c32c92d5\") " pod="openshift-ingress/e2e-authcfg-http-cc2474-openshift-default-7f6458856f-qw52z" Nov 04 03:36:41.449897 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:36:41.449834 4405 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ab0796a3-e6f1-441a-b0f5-4d91c32c92d5-istio-data\") pod \"e2e-authcfg-http-cc2474-openshift-default-7f6458856f-qw52z\" (UID: \"ab0796a3-e6f1-441a-b0f5-4d91c32c92d5\") " pod="openshift-ingress/e2e-authcfg-http-cc2474-openshift-default-7f6458856f-qw52z" Nov 04 03:36:41.449943 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:36:41.449909 4405 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ab0796a3-e6f1-441a-b0f5-4d91c32c92d5-credential-socket\") pod \"e2e-authcfg-http-cc2474-openshift-default-7f6458856f-qw52z\" (UID: \"ab0796a3-e6f1-441a-b0f5-4d91c32c92d5\") " pod="openshift-ingress/e2e-authcfg-http-cc2474-openshift-default-7f6458856f-qw52z" Nov 04 03:36:41.449982 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:36:41.449966 4405 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ab0796a3-e6f1-441a-b0f5-4d91c32c92d5-istiod-ca-cert\") pod \"e2e-authcfg-http-cc2474-openshift-default-7f6458856f-qw52z\" (UID: \"ab0796a3-e6f1-441a-b0f5-4d91c32c92d5\") " pod="openshift-ingress/e2e-authcfg-http-cc2474-openshift-default-7f6458856f-qw52z" Nov 04 03:36:41.450023 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:36:41.450002 4405 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ab0796a3-e6f1-441a-b0f5-4d91c32c92d5-workload-socket\") pod \"e2e-authcfg-http-cc2474-openshift-default-7f6458856f-qw52z\" (UID: \"ab0796a3-e6f1-441a-b0f5-4d91c32c92d5\") " pod="openshift-ingress/e2e-authcfg-http-cc2474-openshift-default-7f6458856f-qw52z" Nov 04 03:36:41.450067 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:36:41.450030 4405 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ab0796a3-e6f1-441a-b0f5-4d91c32c92d5-istio-envoy\") pod \"e2e-authcfg-http-cc2474-openshift-default-7f6458856f-qw52z\" (UID: \"ab0796a3-e6f1-441a-b0f5-4d91c32c92d5\") " pod="openshift-ingress/e2e-authcfg-http-cc2474-openshift-default-7f6458856f-qw52z" Nov 04 03:36:41.450153 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:36:41.450133 4405 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ab0796a3-e6f1-441a-b0f5-4d91c32c92d5-istio-podinfo\") pod \"e2e-authcfg-http-cc2474-openshift-default-7f6458856f-qw52z\" (UID: \"ab0796a3-e6f1-441a-b0f5-4d91c32c92d5\") " pod="openshift-ingress/e2e-authcfg-http-cc2474-openshift-default-7f6458856f-qw52z" Nov 04 03:36:41.450194 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:36:41.450173 4405 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ab0796a3-e6f1-441a-b0f5-4d91c32c92d5-workload-certs\") pod \"e2e-authcfg-http-cc2474-openshift-default-7f6458856f-qw52z\" (UID: \"ab0796a3-e6f1-441a-b0f5-4d91c32c92d5\") " pod="openshift-ingress/e2e-authcfg-http-cc2474-openshift-default-7f6458856f-qw52z" Nov 04 03:36:41.477555 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:36:41.477115 4405 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-authcfg-http-cc2474-openshift-default-7f6458856f-qw52z"] Nov 04 03:36:41.551489 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:36:41.551443 4405 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ab0796a3-e6f1-441a-b0f5-4d91c32c92d5-istio-data\") pod \"e2e-authcfg-http-cc2474-openshift-default-7f6458856f-qw52z\" (UID: \"ab0796a3-e6f1-441a-b0f5-4d91c32c92d5\") " pod="openshift-ingress/e2e-authcfg-http-cc2474-openshift-default-7f6458856f-qw52z" Nov 04 03:36:41.551727 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:36:41.551537 4405 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ab0796a3-e6f1-441a-b0f5-4d91c32c92d5-credential-socket\") pod \"e2e-authcfg-http-cc2474-openshift-default-7f6458856f-qw52z\" (UID: \"ab0796a3-e6f1-441a-b0f5-4d91c32c92d5\") " pod="openshift-ingress/e2e-authcfg-http-cc2474-openshift-default-7f6458856f-qw52z" Nov 04 03:36:41.551727 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:36:41.551586 4405 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ab0796a3-e6f1-441a-b0f5-4d91c32c92d5-istiod-ca-cert\") pod \"e2e-authcfg-http-cc2474-openshift-default-7f6458856f-qw52z\" (UID: \"ab0796a3-e6f1-441a-b0f5-4d91c32c92d5\") " pod="openshift-ingress/e2e-authcfg-http-cc2474-openshift-default-7f6458856f-qw52z" Nov 04 03:36:41.551727 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:36:41.551620 4405 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ab0796a3-e6f1-441a-b0f5-4d91c32c92d5-workload-socket\") pod \"e2e-authcfg-http-cc2474-openshift-default-7f6458856f-qw52z\" (UID: \"ab0796a3-e6f1-441a-b0f5-4d91c32c92d5\") " pod="openshift-ingress/e2e-authcfg-http-cc2474-openshift-default-7f6458856f-qw52z" Nov 04 03:36:41.551727 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:36:41.551648 4405 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ab0796a3-e6f1-441a-b0f5-4d91c32c92d5-istio-envoy\") pod \"e2e-authcfg-http-cc2474-openshift-default-7f6458856f-qw52z\" (UID: \"ab0796a3-e6f1-441a-b0f5-4d91c32c92d5\") " pod="openshift-ingress/e2e-authcfg-http-cc2474-openshift-default-7f6458856f-qw52z" Nov 04 03:36:41.551924 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:36:41.551744 4405 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ab0796a3-e6f1-441a-b0f5-4d91c32c92d5-istio-podinfo\") pod \"e2e-authcfg-http-cc2474-openshift-default-7f6458856f-qw52z\" (UID: \"ab0796a3-e6f1-441a-b0f5-4d91c32c92d5\") " pod="openshift-ingress/e2e-authcfg-http-cc2474-openshift-default-7f6458856f-qw52z" Nov 04 03:36:41.551924 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:36:41.551779 4405 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ab0796a3-e6f1-441a-b0f5-4d91c32c92d5-workload-certs\") pod \"e2e-authcfg-http-cc2474-openshift-default-7f6458856f-qw52z\" (UID: \"ab0796a3-e6f1-441a-b0f5-4d91c32c92d5\") " pod="openshift-ingress/e2e-authcfg-http-cc2474-openshift-default-7f6458856f-qw52z" Nov 04 03:36:41.551924 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:36:41.551833 4405 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rhjg9\" (UniqueName: \"kubernetes.io/projected/ab0796a3-e6f1-441a-b0f5-4d91c32c92d5-kube-api-access-rhjg9\") pod \"e2e-authcfg-http-cc2474-openshift-default-7f6458856f-qw52z\" (UID: \"ab0796a3-e6f1-441a-b0f5-4d91c32c92d5\") " pod="openshift-ingress/e2e-authcfg-http-cc2474-openshift-default-7f6458856f-qw52z" Nov 04 03:36:41.551924 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:36:41.551866 4405 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ab0796a3-e6f1-441a-b0f5-4d91c32c92d5-istio-token\") pod \"e2e-authcfg-http-cc2474-openshift-default-7f6458856f-qw52z\" (UID: \"ab0796a3-e6f1-441a-b0f5-4d91c32c92d5\") " pod="openshift-ingress/e2e-authcfg-http-cc2474-openshift-default-7f6458856f-qw52z" Nov 04 03:36:41.552770 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:36:41.552738 4405 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ab0796a3-e6f1-441a-b0f5-4d91c32c92d5-istio-data\") pod \"e2e-authcfg-http-cc2474-openshift-default-7f6458856f-qw52z\" (UID: \"ab0796a3-e6f1-441a-b0f5-4d91c32c92d5\") " pod="openshift-ingress/e2e-authcfg-http-cc2474-openshift-default-7f6458856f-qw52z" Nov 04 03:36:41.553059 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:36:41.553036 4405 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ab0796a3-e6f1-441a-b0f5-4d91c32c92d5-credential-socket\") pod \"e2e-authcfg-http-cc2474-openshift-default-7f6458856f-qw52z\" (UID: \"ab0796a3-e6f1-441a-b0f5-4d91c32c92d5\") " pod="openshift-ingress/e2e-authcfg-http-cc2474-openshift-default-7f6458856f-qw52z" Nov 04 03:36:41.553896 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:36:41.553870 4405 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ab0796a3-e6f1-441a-b0f5-4d91c32c92d5-istiod-ca-cert\") pod \"e2e-authcfg-http-cc2474-openshift-default-7f6458856f-qw52z\" (UID: \"ab0796a3-e6f1-441a-b0f5-4d91c32c92d5\") " pod="openshift-ingress/e2e-authcfg-http-cc2474-openshift-default-7f6458856f-qw52z" Nov 04 03:36:41.554191 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:36:41.554169 4405 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ab0796a3-e6f1-441a-b0f5-4d91c32c92d5-workload-socket\") pod \"e2e-authcfg-http-cc2474-openshift-default-7f6458856f-qw52z\" (UID: \"ab0796a3-e6f1-441a-b0f5-4d91c32c92d5\") " pod="openshift-ingress/e2e-authcfg-http-cc2474-openshift-default-7f6458856f-qw52z" Nov 04 03:36:41.554816 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:36:41.554789 4405 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ab0796a3-e6f1-441a-b0f5-4d91c32c92d5-workload-certs\") pod \"e2e-authcfg-http-cc2474-openshift-default-7f6458856f-qw52z\" (UID: \"ab0796a3-e6f1-441a-b0f5-4d91c32c92d5\") " pod="openshift-ingress/e2e-authcfg-http-cc2474-openshift-default-7f6458856f-qw52z" Nov 04 03:36:41.556964 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:36:41.556929 4405 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ab0796a3-e6f1-441a-b0f5-4d91c32c92d5-istio-envoy\") pod \"e2e-authcfg-http-cc2474-openshift-default-7f6458856f-qw52z\" (UID: \"ab0796a3-e6f1-441a-b0f5-4d91c32c92d5\") " pod="openshift-ingress/e2e-authcfg-http-cc2474-openshift-default-7f6458856f-qw52z" Nov 04 03:36:41.557644 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:36:41.557619 4405 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ab0796a3-e6f1-441a-b0f5-4d91c32c92d5-istio-podinfo\") pod \"e2e-authcfg-http-cc2474-openshift-default-7f6458856f-qw52z\" (UID: \"ab0796a3-e6f1-441a-b0f5-4d91c32c92d5\") " pod="openshift-ingress/e2e-authcfg-http-cc2474-openshift-default-7f6458856f-qw52z" Nov 04 03:36:41.614830 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:36:41.614796 4405 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ab0796a3-e6f1-441a-b0f5-4d91c32c92d5-istio-token\") pod \"e2e-authcfg-http-cc2474-openshift-default-7f6458856f-qw52z\" (UID: \"ab0796a3-e6f1-441a-b0f5-4d91c32c92d5\") " pod="openshift-ingress/e2e-authcfg-http-cc2474-openshift-default-7f6458856f-qw52z" Nov 04 03:36:41.615135 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:36:41.615113 4405 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rhjg9\" (UniqueName: \"kubernetes.io/projected/ab0796a3-e6f1-441a-b0f5-4d91c32c92d5-kube-api-access-rhjg9\") pod \"e2e-authcfg-http-cc2474-openshift-default-7f6458856f-qw52z\" (UID: \"ab0796a3-e6f1-441a-b0f5-4d91c32c92d5\") " pod="openshift-ingress/e2e-authcfg-http-cc2474-openshift-default-7f6458856f-qw52z" Nov 04 03:36:41.685700 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:36:41.685580 4405 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-authcfg-http-cc2474-openshift-default-7f6458856f-qw52z" Nov 04 03:36:42.123394 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:36:42.123364 4405 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-authcfg-http-cc2474-openshift-default-7f6458856f-qw52z"] Nov 04 03:36:42.125372 ip-10-0-134-125 kubenswrapper[4405]: W1104 03:36:42.125335 4405 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podab0796a3_e6f1_441a_b0f5_4d91c32c92d5.slice/crio-a2b7956bf689f0d58810a3beb046be46a332c613fa832f9225e24f9a5854f0df WatchSource:0}: Error finding container a2b7956bf689f0d58810a3beb046be46a332c613fa832f9225e24f9a5854f0df: Status 404 returned error can't find the container with id a2b7956bf689f0d58810a3beb046be46a332c613fa832f9225e24f9a5854f0df Nov 04 03:36:42.127561 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:36:42.127531 4405 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 04 03:36:42.127655 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:36:42.127595 4405 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 04 03:36:42.127655 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:36:42.127627 4405 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 04 03:36:42.150416 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:36:42.150379 4405 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-authcfg-http-cc2474-openshift-default-7f6458856f-qw52z" event={"ID":"ab0796a3-e6f1-441a-b0f5-4d91c32c92d5","Type":"ContainerStarted","Data":"a2b7956bf689f0d58810a3beb046be46a332c613fa832f9225e24f9a5854f0df"} Nov 04 03:36:42.733840 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:36:42.733802 4405 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-bl5mg"] Nov 04 03:36:43.156887 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:36:43.156845 4405 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-authcfg-http-cc2474-openshift-default-7f6458856f-qw52z" event={"ID":"ab0796a3-e6f1-441a-b0f5-4d91c32c92d5","Type":"ContainerStarted","Data":"614f2922503f50e5fc5d852c86079bb3d487191f836187e8c2dbf9b3dd75ea0f"} Nov 04 03:36:43.327582 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:36:43.327518 4405 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-authcfg-http-cc2474-openshift-default-7f6458856f-qw52z" podStartSLOduration=2.327501538 podStartE2EDuration="2.327501538s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 03:36:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 03:36:43.326360736 +0000 UTC m=+3849.757060856" watchObservedRunningTime="2026-11-04 03:36:43.327501538 +0000 UTC m=+3849.758201683" Nov 04 03:36:43.686501 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:36:43.686461 4405 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-authcfg-http-cc2474-openshift-default-7f6458856f-qw52z" Nov 04 03:36:43.692926 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:36:43.692899 4405 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-authcfg-http-cc2474-openshift-default-7f6458856f-qw52z" Nov 04 03:36:44.162345 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:36:44.162314 4405 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-authcfg-http-cc2474-openshift-default-7f6458856f-qw52z" Nov 04 03:36:44.163590 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:36:44.163574 4405 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-authcfg-http-cc2474-openshift-default-7f6458856f-qw52z" Nov 04 03:36:50.111420 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:36:50.111385 4405 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-authcfg-http-cc2474-7df96f57b7-dp897"] Nov 04 03:36:50.115139 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:36:50.115118 4405 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-cc2474-7df96f57b7-dp897" Nov 04 03:36:50.167372 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:36:50.167339 4405 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-authcfg-http-cc2474\"" Nov 04 03:36:50.217127 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:36:50.217085 4405 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-authcfg-http-cc2474-dockercfg-sdwp2\"" Nov 04 03:36:50.218957 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:36:50.218936 4405 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-authcfg-http-cc2474-7df96f57b7-dp897"] Nov 04 03:36:50.246143 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:36:50.246097 4405 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jwfxc\" (UniqueName: \"kubernetes.io/projected/d319203f-26fb-438b-8466-b895b6c6d2d4-kube-api-access-jwfxc\") pod \"payload-pre-processing-e2e-authcfg-http-cc2474-7df96f57b7-dp897\" (UID: \"d319203f-26fb-438b-8466-b895b6c6d2d4\") " pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-cc2474-7df96f57b7-dp897" Nov 04 03:36:50.246300 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:36:50.246192 4405 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d319203f-26fb-438b-8466-b895b6c6d2d4-plugins-config-volume\") pod \"payload-pre-processing-e2e-authcfg-http-cc2474-7df96f57b7-dp897\" (UID: \"d319203f-26fb-438b-8466-b895b6c6d2d4\") " pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-cc2474-7df96f57b7-dp897" Nov 04 03:36:50.347566 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:36:50.347530 4405 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jwfxc\" (UniqueName: \"kubernetes.io/projected/d319203f-26fb-438b-8466-b895b6c6d2d4-kube-api-access-jwfxc\") pod \"payload-pre-processing-e2e-authcfg-http-cc2474-7df96f57b7-dp897\" (UID: \"d319203f-26fb-438b-8466-b895b6c6d2d4\") " pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-cc2474-7df96f57b7-dp897" Nov 04 03:36:50.347826 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:36:50.347623 4405 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d319203f-26fb-438b-8466-b895b6c6d2d4-plugins-config-volume\") pod \"payload-pre-processing-e2e-authcfg-http-cc2474-7df96f57b7-dp897\" (UID: \"d319203f-26fb-438b-8466-b895b6c6d2d4\") " pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-cc2474-7df96f57b7-dp897" Nov 04 03:36:50.348333 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:36:50.348311 4405 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d319203f-26fb-438b-8466-b895b6c6d2d4-plugins-config-volume\") pod \"payload-pre-processing-e2e-authcfg-http-cc2474-7df96f57b7-dp897\" (UID: \"d319203f-26fb-438b-8466-b895b6c6d2d4\") " pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-cc2474-7df96f57b7-dp897" Nov 04 03:36:50.410576 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:36:50.410501 4405 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jwfxc\" (UniqueName: \"kubernetes.io/projected/d319203f-26fb-438b-8466-b895b6c6d2d4-kube-api-access-jwfxc\") pod \"payload-pre-processing-e2e-authcfg-http-cc2474-7df96f57b7-dp897\" (UID: \"d319203f-26fb-438b-8466-b895b6c6d2d4\") " pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-cc2474-7df96f57b7-dp897" Nov 04 03:36:50.429948 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:36:50.429908 4405 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-cc2474-7df96f57b7-dp897" Nov 04 03:36:50.809093 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:36:50.809066 4405 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-authcfg-http-cc2474-7df96f57b7-dp897"] Nov 04 03:36:50.811333 ip-10-0-134-125 kubenswrapper[4405]: W1104 03:36:50.811304 4405 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd319203f_26fb_438b_8466_b895b6c6d2d4.slice/crio-7b1a57ea0e64cfb7da459f26ecac8b1b163f769d31c18d7783c9f3956d24c63b WatchSource:0}: Error finding container 7b1a57ea0e64cfb7da459f26ecac8b1b163f769d31c18d7783c9f3956d24c63b: Status 404 returned error can't find the container with id 7b1a57ea0e64cfb7da459f26ecac8b1b163f769d31c18d7783c9f3956d24c63b Nov 04 03:36:51.205926 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:36:51.205889 4405 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-cc2474-7df96f57b7-dp897" event={"ID":"d319203f-26fb-438b-8466-b895b6c6d2d4","Type":"ContainerStarted","Data":"a7c50e588efd91d5672bf780542d79d209125e4a92d0bcbe0b77dc49d558f52e"} Nov 04 03:36:51.206385 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:36:51.205942 4405 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-cc2474-7df96f57b7-dp897" event={"ID":"d319203f-26fb-438b-8466-b895b6c6d2d4","Type":"ContainerStarted","Data":"7b1a57ea0e64cfb7da459f26ecac8b1b163f769d31c18d7783c9f3956d24c63b"} Nov 04 03:36:51.206385 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:36:51.206066 4405 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-cc2474-7df96f57b7-dp897" Nov 04 03:36:51.322597 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:36:51.322533 4405 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-cc2474-7df96f57b7-dp897" podStartSLOduration=1.322516477 podStartE2EDuration="1.322516477s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 03:36:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 03:36:51.321483532 +0000 UTC m=+3857.752183654" watchObservedRunningTime="2026-11-04 03:36:51.322516477 +0000 UTC m=+3857.753216596" Nov 04 03:36:53.060194 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:36:53.060158 4405 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-authcfg-http-cc2474-55d557dfc-x2lf6"] Nov 04 03:36:53.064211 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:36:53.064191 4405 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-cc2474-55d557dfc-x2lf6" Nov 04 03:36:53.116313 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:36:53.116279 4405 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-authcfg-http-cc2474-serving-cert\"" Nov 04 03:36:53.123352 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:36:53.123321 4405 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-authcfg-http-cc2474-55d557dfc-x2lf6"] Nov 04 03:36:53.174297 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:36:53.174253 4405 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7f32fb6c-d557-461b-b22f-90a4d3ea46e3-maas-api-tls\") pod \"maas-api-e2e-authcfg-http-cc2474-55d557dfc-x2lf6\" (UID: \"7f32fb6c-d557-461b-b22f-90a4d3ea46e3\") " pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-cc2474-55d557dfc-x2lf6" Nov 04 03:36:53.174297 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:36:53.174303 4405 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7f32fb6c-d557-461b-b22f-90a4d3ea46e3-openshift-service-ca\") pod \"maas-api-e2e-authcfg-http-cc2474-55d557dfc-x2lf6\" (UID: \"7f32fb6c-d557-461b-b22f-90a4d3ea46e3\") " pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-cc2474-55d557dfc-x2lf6" Nov 04 03:36:53.174541 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:36:53.174392 4405 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7f32fb6c-d557-461b-b22f-90a4d3ea46e3-metrics-certs\") pod \"maas-api-e2e-authcfg-http-cc2474-55d557dfc-x2lf6\" (UID: \"7f32fb6c-d557-461b-b22f-90a4d3ea46e3\") " pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-cc2474-55d557dfc-x2lf6" Nov 04 03:36:53.174541 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:36:53.174497 4405 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2q96g\" (UniqueName: \"kubernetes.io/projected/7f32fb6c-d557-461b-b22f-90a4d3ea46e3-kube-api-access-2q96g\") pod \"maas-api-e2e-authcfg-http-cc2474-55d557dfc-x2lf6\" (UID: \"7f32fb6c-d557-461b-b22f-90a4d3ea46e3\") " pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-cc2474-55d557dfc-x2lf6" Nov 04 03:36:53.275459 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:36:53.275420 4405 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7f32fb6c-d557-461b-b22f-90a4d3ea46e3-metrics-certs\") pod \"maas-api-e2e-authcfg-http-cc2474-55d557dfc-x2lf6\" (UID: \"7f32fb6c-d557-461b-b22f-90a4d3ea46e3\") " pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-cc2474-55d557dfc-x2lf6" Nov 04 03:36:53.275730 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:36:53.275514 4405 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2q96g\" (UniqueName: \"kubernetes.io/projected/7f32fb6c-d557-461b-b22f-90a4d3ea46e3-kube-api-access-2q96g\") pod \"maas-api-e2e-authcfg-http-cc2474-55d557dfc-x2lf6\" (UID: \"7f32fb6c-d557-461b-b22f-90a4d3ea46e3\") " pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-cc2474-55d557dfc-x2lf6" Nov 04 03:36:53.275730 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:36:53.275583 4405 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7f32fb6c-d557-461b-b22f-90a4d3ea46e3-maas-api-tls\") pod \"maas-api-e2e-authcfg-http-cc2474-55d557dfc-x2lf6\" (UID: \"7f32fb6c-d557-461b-b22f-90a4d3ea46e3\") " pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-cc2474-55d557dfc-x2lf6" Nov 04 03:36:53.275730 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:36:53.275605 4405 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7f32fb6c-d557-461b-b22f-90a4d3ea46e3-openshift-service-ca\") pod \"maas-api-e2e-authcfg-http-cc2474-55d557dfc-x2lf6\" (UID: \"7f32fb6c-d557-461b-b22f-90a4d3ea46e3\") " pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-cc2474-55d557dfc-x2lf6" Nov 04 03:36:53.275921 ip-10-0-134-125 kubenswrapper[4405]: E1104 03:36:53.275759 4405 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-authcfg-http-cc2474-serving-cert: secret "maas-api-e2e-authcfg-http-cc2474-serving-cert" not found Nov 04 03:36:53.275921 ip-10-0-134-125 kubenswrapper[4405]: E1104 03:36:53.275849 4405 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7f32fb6c-d557-461b-b22f-90a4d3ea46e3-maas-api-tls podName:7f32fb6c-d557-461b-b22f-90a4d3ea46e3 nodeName:}" failed. No retries permitted until 2026-11-04 03:36:53.775827055 +0000 UTC m=+3860.206527166 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/7f32fb6c-d557-461b-b22f-90a4d3ea46e3-maas-api-tls") pod "maas-api-e2e-authcfg-http-cc2474-55d557dfc-x2lf6" (UID: "7f32fb6c-d557-461b-b22f-90a4d3ea46e3") : secret "maas-api-e2e-authcfg-http-cc2474-serving-cert" not found Nov 04 03:36:53.276303 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:36:53.276277 4405 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7f32fb6c-d557-461b-b22f-90a4d3ea46e3-openshift-service-ca\") pod \"maas-api-e2e-authcfg-http-cc2474-55d557dfc-x2lf6\" (UID: \"7f32fb6c-d557-461b-b22f-90a4d3ea46e3\") " pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-cc2474-55d557dfc-x2lf6" Nov 04 03:36:53.278231 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:36:53.278211 4405 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7f32fb6c-d557-461b-b22f-90a4d3ea46e3-metrics-certs\") pod \"maas-api-e2e-authcfg-http-cc2474-55d557dfc-x2lf6\" (UID: \"7f32fb6c-d557-461b-b22f-90a4d3ea46e3\") " pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-cc2474-55d557dfc-x2lf6" Nov 04 03:36:53.337144 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:36:53.337116 4405 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2q96g\" (UniqueName: \"kubernetes.io/projected/7f32fb6c-d557-461b-b22f-90a4d3ea46e3-kube-api-access-2q96g\") pod \"maas-api-e2e-authcfg-http-cc2474-55d557dfc-x2lf6\" (UID: \"7f32fb6c-d557-461b-b22f-90a4d3ea46e3\") " pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-cc2474-55d557dfc-x2lf6" Nov 04 03:36:53.781961 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:36:53.781861 4405 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7f32fb6c-d557-461b-b22f-90a4d3ea46e3-maas-api-tls\") pod \"maas-api-e2e-authcfg-http-cc2474-55d557dfc-x2lf6\" (UID: \"7f32fb6c-d557-461b-b22f-90a4d3ea46e3\") " pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-cc2474-55d557dfc-x2lf6" Nov 04 03:36:53.784463 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:36:53.784433 4405 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7f32fb6c-d557-461b-b22f-90a4d3ea46e3-maas-api-tls\") pod \"maas-api-e2e-authcfg-http-cc2474-55d557dfc-x2lf6\" (UID: \"7f32fb6c-d557-461b-b22f-90a4d3ea46e3\") " pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-cc2474-55d557dfc-x2lf6" Nov 04 03:36:53.978240 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:36:53.978201 4405 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-cc2474-55d557dfc-x2lf6" Nov 04 03:36:54.353014 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:36:54.352989 4405 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-authcfg-http-cc2474-55d557dfc-x2lf6"] Nov 04 03:36:54.355136 ip-10-0-134-125 kubenswrapper[4405]: W1104 03:36:54.355096 4405 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7f32fb6c_d557_461b_b22f_90a4d3ea46e3.slice/crio-5c349f0a2bf61b5d8425bae40772b0784118c8f5ba632d7d34952c7841bf8a18 WatchSource:0}: Error finding container 5c349f0a2bf61b5d8425bae40772b0784118c8f5ba632d7d34952c7841bf8a18: Status 404 returned error can't find the container with id 5c349f0a2bf61b5d8425bae40772b0784118c8f5ba632d7d34952c7841bf8a18 Nov 04 03:36:55.228386 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:36:55.228344 4405 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-cc2474-55d557dfc-x2lf6" event={"ID":"7f32fb6c-d557-461b-b22f-90a4d3ea46e3","Type":"ContainerStarted","Data":"5c349f0a2bf61b5d8425bae40772b0784118c8f5ba632d7d34952c7841bf8a18"} Nov 04 03:36:56.234983 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:36:56.234943 4405 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-cc2474-55d557dfc-x2lf6" event={"ID":"7f32fb6c-d557-461b-b22f-90a4d3ea46e3","Type":"ContainerStarted","Data":"1494eaaa6233695c3c4fc861f6f541a24c7351e6b559e13b4f1599300be2e73a"} Nov 04 03:36:56.235424 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:36:56.235066 4405 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-cc2474-55d557dfc-x2lf6" Nov 04 03:36:56.352475 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:36:56.352412 4405 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-cc2474-55d557dfc-x2lf6" podStartSLOduration=2.500919277 podStartE2EDuration="3.352397638s" totalImagesPullingTime="851.478361ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 03:36:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 03:36:56.352004899 +0000 UTC m=+3862.782705021" watchObservedRunningTime="2026-11-04 03:36:56.352397638 +0000 UTC m=+3862.783097757" Nov 04 03:37:01.356009 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:37:01.355962 4405 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-bl5mg"] Nov 04 03:37:02.219633 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:37:02.219599 4405 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-cc2474-7df96f57b7-dp897" Nov 04 03:37:02.245173 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:37:02.245143 4405 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-cc2474-55d557dfc-x2lf6" Nov 04 03:37:06.884586 ip-10-0-134-125 kubenswrapper[4405]: E1104 03:37:06.884539 4405 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5f2vb" containerName="extractor" Nov 04 03:37:11.312108 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:37:11.312067 4405 generic.go:356] "Generic (PLEG): container finished" podID="ab0796a3-e6f1-441a-b0f5-4d91c32c92d5" containerID="614f2922503f50e5fc5d852c86079bb3d487191f836187e8c2dbf9b3dd75ea0f" exitCode=0 Nov 04 03:37:11.312633 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:37:11.312144 4405 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-authcfg-http-cc2474-openshift-default-7f6458856f-qw52z" event={"ID":"ab0796a3-e6f1-441a-b0f5-4d91c32c92d5","Type":"ContainerDied","Data":"614f2922503f50e5fc5d852c86079bb3d487191f836187e8c2dbf9b3dd75ea0f"} Nov 04 03:37:11.312633 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:37:11.312532 4405 scope.go:122] "RemoveContainer" containerID="614f2922503f50e5fc5d852c86079bb3d487191f836187e8c2dbf9b3dd75ea0f" Nov 04 03:37:11.313732 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:37:11.313707 4405 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 04 03:37:11.313795 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:37:11.313772 4405 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 04 03:37:11.313835 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:37:11.313804 4405 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 04 03:37:12.323497 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:37:12.323455 4405 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-authcfg-http-cc2474-openshift-default-7f6458856f-qw52z" event={"ID":"ab0796a3-e6f1-441a-b0f5-4d91c32c92d5","Type":"ContainerStarted","Data":"c65274aa868f4739b8ea499a31c0f8822cc70b23d51d3f8ec15b2478f97c0b7d"} Nov 04 03:37:12.686777 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:37:12.686729 4405 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-authcfg-http-cc2474-openshift-default-7f6458856f-qw52z" Nov 04 03:37:12.688130 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:37:12.688103 4405 patch_prober.go:29] interesting pod/e2e-authcfg-http-cc2474-openshift-default-7f6458856f-qw52z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.176:15021/healthz/ready\": dial tcp 10.134.0.176:15021: connect: connection refused" start-of-body= Nov 04 03:37:12.688274 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:37:12.688153 4405 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-authcfg-http-cc2474-openshift-default-7f6458856f-qw52z" podUID="ab0796a3-e6f1-441a-b0f5-4d91c32c92d5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.176:15021/healthz/ready\": dial tcp 10.134.0.176:15021: connect: connection refused" Nov 04 03:37:13.686160 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:37:13.686109 4405 patch_prober.go:29] interesting pod/e2e-authcfg-http-cc2474-openshift-default-7f6458856f-qw52z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.176:15021/healthz/ready\": dial tcp 10.134.0.176:15021: connect: connection refused" start-of-body= Nov 04 03:37:13.686573 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:37:13.686181 4405 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-authcfg-http-cc2474-openshift-default-7f6458856f-qw52z" podUID="ab0796a3-e6f1-441a-b0f5-4d91c32c92d5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.176:15021/healthz/ready\": dial tcp 10.134.0.176:15021: connect: connection refused" Nov 04 03:37:14.162752 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:37:14.162708 4405 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-authcfg-http-cc2474-openshift-default-7f6458856f-qw52z" Nov 04 03:37:14.686956 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:37:14.686917 4405 patch_prober.go:29] interesting pod/e2e-authcfg-http-cc2474-openshift-default-7f6458856f-qw52z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.176:15021/healthz/ready\": dial tcp 10.134.0.176:15021: connect: connection refused" start-of-body= Nov 04 03:37:14.687379 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:37:14.686980 4405 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-authcfg-http-cc2474-openshift-default-7f6458856f-qw52z" podUID="ab0796a3-e6f1-441a-b0f5-4d91c32c92d5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.176:15021/healthz/ready\": dial tcp 10.134.0.176:15021: connect: connection refused" Nov 04 03:37:15.686181 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:37:15.686136 4405 patch_prober.go:29] interesting pod/e2e-authcfg-http-cc2474-openshift-default-7f6458856f-qw52z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.176:15021/healthz/ready\": dial tcp 10.134.0.176:15021: connect: connection refused" start-of-body= Nov 04 03:37:15.686379 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:37:15.686201 4405 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-authcfg-http-cc2474-openshift-default-7f6458856f-qw52z" podUID="ab0796a3-e6f1-441a-b0f5-4d91c32c92d5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.176:15021/healthz/ready\": dial tcp 10.134.0.176:15021: connect: connection refused" Nov 04 03:37:16.686925 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:37:16.686884 4405 patch_prober.go:29] interesting pod/e2e-authcfg-http-cc2474-openshift-default-7f6458856f-qw52z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.176:15021/healthz/ready\": dial tcp 10.134.0.176:15021: connect: connection refused" start-of-body= Nov 04 03:37:16.687348 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:37:16.686945 4405 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-authcfg-http-cc2474-openshift-default-7f6458856f-qw52z" podUID="ab0796a3-e6f1-441a-b0f5-4d91c32c92d5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.176:15021/healthz/ready\": dial tcp 10.134.0.176:15021: connect: connection refused" Nov 04 03:37:17.686868 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:37:17.686827 4405 patch_prober.go:29] interesting pod/e2e-authcfg-http-cc2474-openshift-default-7f6458856f-qw52z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.176:15021/healthz/ready\": dial tcp 10.134.0.176:15021: connect: connection refused" start-of-body= Nov 04 03:37:17.687069 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:37:17.686900 4405 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-authcfg-http-cc2474-openshift-default-7f6458856f-qw52z" podUID="ab0796a3-e6f1-441a-b0f5-4d91c32c92d5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.176:15021/healthz/ready\": dial tcp 10.134.0.176:15021: connect: connection refused" Nov 04 03:37:18.691486 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:37:18.691451 4405 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-authcfg-http-cc2474-openshift-default-7f6458856f-qw52z" Nov 04 03:37:18.692544 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:37:18.692526 4405 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-authcfg-http-cc2474-openshift-default-7f6458856f-qw52z" Nov 04 03:37:34.274163 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:37:34.274131 4405 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-vwcx8_ffbb2d83-5a93-4a5b-bce3-e40afb8f58ea/ovn-acl-logging/0.log" Nov 04 03:37:34.291481 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:37:34.291455 4405 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-vwcx8_ffbb2d83-5a93-4a5b-bce3-e40afb8f58ea/ovn-acl-logging/0.log" Nov 04 03:37:36.885281 ip-10-0-134-125 kubenswrapper[4405]: E1104 03:37:36.885241 4405 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5f2vb" containerName="extractor" Nov 04 03:38:06.884891 ip-10-0-134-125 kubenswrapper[4405]: E1104 03:38:06.884804 4405 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5f2vb" containerName="extractor" Nov 04 03:38:36.884581 ip-10-0-134-125 kubenswrapper[4405]: E1104 03:38:36.884542 4405 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5f2vb" containerName="extractor" Nov 04 03:38:38.401500 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:38:38.401465 4405 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-authcfg-http-cc2474-7df96f57b7-dp897"] Nov 04 03:38:41.095899 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:38:41.095864 4405 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-authcfg-http-cc2474-55d557dfc-x2lf6"] Nov 04 03:38:41.096423 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:38:41.096087 4405 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-cc2474-55d557dfc-x2lf6" podUID="7f32fb6c-d557-461b-b22f-90a4d3ea46e3" containerName="maas-api" containerID="cri-o://1494eaaa6233695c3c4fc861f6f541a24c7351e6b559e13b4f1599300be2e73a" gracePeriod=30 Nov 04 03:38:41.358849 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:38:41.358780 4405 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-cc2474-55d557dfc-x2lf6" Nov 04 03:38:41.401188 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:38:41.401148 4405 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7f32fb6c-d557-461b-b22f-90a4d3ea46e3-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7f32fb6c-d557-461b-b22f-90a4d3ea46e3-openshift-service-ca\") pod \"7f32fb6c-d557-461b-b22f-90a4d3ea46e3\" (UID: \"7f32fb6c-d557-461b-b22f-90a4d3ea46e3\") " Nov 04 03:38:41.401326 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:38:41.401204 4405 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7f32fb6c-d557-461b-b22f-90a4d3ea46e3-metrics-certs\" (UniqueName: \"kubernetes.io/secret/7f32fb6c-d557-461b-b22f-90a4d3ea46e3-metrics-certs\") pod \"7f32fb6c-d557-461b-b22f-90a4d3ea46e3\" (UID: \"7f32fb6c-d557-461b-b22f-90a4d3ea46e3\") " Nov 04 03:38:41.401326 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:38:41.401287 4405 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7f32fb6c-d557-461b-b22f-90a4d3ea46e3-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7f32fb6c-d557-461b-b22f-90a4d3ea46e3-maas-api-tls\") pod \"7f32fb6c-d557-461b-b22f-90a4d3ea46e3\" (UID: \"7f32fb6c-d557-461b-b22f-90a4d3ea46e3\") " Nov 04 03:38:41.401401 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:38:41.401328 4405 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7f32fb6c-d557-461b-b22f-90a4d3ea46e3-kube-api-access-2q96g\" (UniqueName: \"kubernetes.io/projected/7f32fb6c-d557-461b-b22f-90a4d3ea46e3-kube-api-access-2q96g\") pod \"7f32fb6c-d557-461b-b22f-90a4d3ea46e3\" (UID: \"7f32fb6c-d557-461b-b22f-90a4d3ea46e3\") " Nov 04 03:38:41.401539 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:38:41.401515 4405 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7f32fb6c-d557-461b-b22f-90a4d3ea46e3-openshift-service-ca" pod "7f32fb6c-d557-461b-b22f-90a4d3ea46e3" (UID: "7f32fb6c-d557-461b-b22f-90a4d3ea46e3"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 03:38:41.401684 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:38:41.401650 4405 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7f32fb6c-d557-461b-b22f-90a4d3ea46e3-openshift-service-ca\") on node \"ip-10-0-134-125.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:38:41.403604 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:38:41.403572 4405 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7f32fb6c-d557-461b-b22f-90a4d3ea46e3-maas-api-tls" pod "7f32fb6c-d557-461b-b22f-90a4d3ea46e3" (UID: "7f32fb6c-d557-461b-b22f-90a4d3ea46e3"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 04 03:38:41.403784 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:38:41.403696 4405 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7f32fb6c-d557-461b-b22f-90a4d3ea46e3-kube-api-access-2q96g" pod "7f32fb6c-d557-461b-b22f-90a4d3ea46e3" (UID: "7f32fb6c-d557-461b-b22f-90a4d3ea46e3"). InnerVolumeSpecName "kube-api-access-2q96g". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 03:38:41.403784 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:38:41.403728 4405 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7f32fb6c-d557-461b-b22f-90a4d3ea46e3-metrics-certs" pod "7f32fb6c-d557-461b-b22f-90a4d3ea46e3" (UID: "7f32fb6c-d557-461b-b22f-90a4d3ea46e3"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 04 03:38:41.502414 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:38:41.502383 4405 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7f32fb6c-d557-461b-b22f-90a4d3ea46e3-metrics-certs\") on node \"ip-10-0-134-125.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:38:41.502414 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:38:41.502412 4405 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7f32fb6c-d557-461b-b22f-90a4d3ea46e3-maas-api-tls\") on node \"ip-10-0-134-125.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:38:41.502637 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:38:41.502423 4405 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2q96g\" (UniqueName: \"kubernetes.io/projected/7f32fb6c-d557-461b-b22f-90a4d3ea46e3-kube-api-access-2q96g\") on node \"ip-10-0-134-125.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:38:41.765868 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:38:41.765829 4405 generic.go:356] "Generic (PLEG): container finished" podID="7f32fb6c-d557-461b-b22f-90a4d3ea46e3" containerID="1494eaaa6233695c3c4fc861f6f541a24c7351e6b559e13b4f1599300be2e73a" exitCode=0 Nov 04 03:38:41.766071 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:38:41.765882 4405 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-cc2474-55d557dfc-x2lf6" Nov 04 03:38:41.766071 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:38:41.765909 4405 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-cc2474-55d557dfc-x2lf6" event={"ID":"7f32fb6c-d557-461b-b22f-90a4d3ea46e3","Type":"ContainerDied","Data":"1494eaaa6233695c3c4fc861f6f541a24c7351e6b559e13b4f1599300be2e73a"} Nov 04 03:38:41.766071 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:38:41.765965 4405 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-cc2474-55d557dfc-x2lf6" event={"ID":"7f32fb6c-d557-461b-b22f-90a4d3ea46e3","Type":"ContainerDied","Data":"5c349f0a2bf61b5d8425bae40772b0784118c8f5ba632d7d34952c7841bf8a18"} Nov 04 03:38:41.766071 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:38:41.765987 4405 scope.go:122] "RemoveContainer" containerID="1494eaaa6233695c3c4fc861f6f541a24c7351e6b559e13b4f1599300be2e73a" Nov 04 03:38:41.775298 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:38:41.775280 4405 scope.go:122] "RemoveContainer" containerID="1494eaaa6233695c3c4fc861f6f541a24c7351e6b559e13b4f1599300be2e73a" Nov 04 03:38:41.775590 ip-10-0-134-125 kubenswrapper[4405]: E1104 03:38:41.775566 4405 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1494eaaa6233695c3c4fc861f6f541a24c7351e6b559e13b4f1599300be2e73a\": container with ID starting with 1494eaaa6233695c3c4fc861f6f541a24c7351e6b559e13b4f1599300be2e73a not found: ID does not exist" containerID="1494eaaa6233695c3c4fc861f6f541a24c7351e6b559e13b4f1599300be2e73a" Nov 04 03:38:41.775693 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:38:41.775599 4405 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1494eaaa6233695c3c4fc861f6f541a24c7351e6b559e13b4f1599300be2e73a"} err="failed to get container status \"1494eaaa6233695c3c4fc861f6f541a24c7351e6b559e13b4f1599300be2e73a\": rpc error: code = NotFound desc = could not find container \"1494eaaa6233695c3c4fc861f6f541a24c7351e6b559e13b4f1599300be2e73a\": container with ID starting with 1494eaaa6233695c3c4fc861f6f541a24c7351e6b559e13b4f1599300be2e73a not found: ID does not exist" Nov 04 03:38:41.929071 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:38:41.929031 4405 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-authcfg-http-cc2474-55d557dfc-x2lf6"] Nov 04 03:38:41.978828 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:38:41.978739 4405 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-authcfg-http-cc2474-55d557dfc-x2lf6"] Nov 04 03:38:43.402720 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:38:43.402639 4405 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-cc2474-7df96f57b7-dp897" podUID="d319203f-26fb-438b-8466-b895b6c6d2d4" containerName="payload-pre-processing" containerID="cri-o://a7c50e588efd91d5672bf780542d79d209125e4a92d0bcbe0b77dc49d558f52e" gracePeriod=25 Nov 04 03:38:43.653106 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:38:43.653034 4405 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-cc2474-7df96f57b7-dp897" Nov 04 03:38:43.700233 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:38:43.700201 4405 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7f32fb6c-d557-461b-b22f-90a4d3ea46e3" path="/var/lib/kubelet/pods/7f32fb6c-d557-461b-b22f-90a4d3ea46e3/volumes" Nov 04 03:38:43.728146 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:38:43.728111 4405 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d319203f-26fb-438b-8466-b895b6c6d2d4-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d319203f-26fb-438b-8466-b895b6c6d2d4-plugins-config-volume\") pod \"d319203f-26fb-438b-8466-b895b6c6d2d4\" (UID: \"d319203f-26fb-438b-8466-b895b6c6d2d4\") " Nov 04 03:38:43.728325 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:38:43.728158 4405 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d319203f-26fb-438b-8466-b895b6c6d2d4-kube-api-access-jwfxc\" (UniqueName: \"kubernetes.io/projected/d319203f-26fb-438b-8466-b895b6c6d2d4-kube-api-access-jwfxc\") pod \"d319203f-26fb-438b-8466-b895b6c6d2d4\" (UID: \"d319203f-26fb-438b-8466-b895b6c6d2d4\") " Nov 04 03:38:43.728506 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:38:43.728482 4405 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d319203f-26fb-438b-8466-b895b6c6d2d4-plugins-config-volume" pod "d319203f-26fb-438b-8466-b895b6c6d2d4" (UID: "d319203f-26fb-438b-8466-b895b6c6d2d4"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 03:38:43.730488 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:38:43.730464 4405 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d319203f-26fb-438b-8466-b895b6c6d2d4-kube-api-access-jwfxc" pod "d319203f-26fb-438b-8466-b895b6c6d2d4" (UID: "d319203f-26fb-438b-8466-b895b6c6d2d4"). InnerVolumeSpecName "kube-api-access-jwfxc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 03:38:43.795799 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:38:43.795760 4405 generic.go:356] "Generic (PLEG): container finished" podID="d319203f-26fb-438b-8466-b895b6c6d2d4" containerID="a7c50e588efd91d5672bf780542d79d209125e4a92d0bcbe0b77dc49d558f52e" exitCode=0 Nov 04 03:38:43.795953 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:38:43.795873 4405 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-cc2474-7df96f57b7-dp897" event={"ID":"d319203f-26fb-438b-8466-b895b6c6d2d4","Type":"ContainerDied","Data":"a7c50e588efd91d5672bf780542d79d209125e4a92d0bcbe0b77dc49d558f52e"} Nov 04 03:38:43.795953 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:38:43.795893 4405 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-cc2474-7df96f57b7-dp897" Nov 04 03:38:43.796049 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:38:43.795961 4405 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-cc2474-7df96f57b7-dp897" event={"ID":"d319203f-26fb-438b-8466-b895b6c6d2d4","Type":"ContainerDied","Data":"7b1a57ea0e64cfb7da459f26ecac8b1b163f769d31c18d7783c9f3956d24c63b"} Nov 04 03:38:43.796049 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:38:43.795993 4405 scope.go:122] "RemoveContainer" containerID="a7c50e588efd91d5672bf780542d79d209125e4a92d0bcbe0b77dc49d558f52e" Nov 04 03:38:43.807123 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:38:43.807100 4405 scope.go:122] "RemoveContainer" containerID="a7c50e588efd91d5672bf780542d79d209125e4a92d0bcbe0b77dc49d558f52e" Nov 04 03:38:43.807478 ip-10-0-134-125 kubenswrapper[4405]: E1104 03:38:43.807446 4405 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a7c50e588efd91d5672bf780542d79d209125e4a92d0bcbe0b77dc49d558f52e\": container with ID starting with a7c50e588efd91d5672bf780542d79d209125e4a92d0bcbe0b77dc49d558f52e not found: ID does not exist" containerID="a7c50e588efd91d5672bf780542d79d209125e4a92d0bcbe0b77dc49d558f52e" Nov 04 03:38:43.807617 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:38:43.807482 4405 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a7c50e588efd91d5672bf780542d79d209125e4a92d0bcbe0b77dc49d558f52e"} err="failed to get container status \"a7c50e588efd91d5672bf780542d79d209125e4a92d0bcbe0b77dc49d558f52e\": rpc error: code = NotFound desc = could not find container \"a7c50e588efd91d5672bf780542d79d209125e4a92d0bcbe0b77dc49d558f52e\": container with ID starting with a7c50e588efd91d5672bf780542d79d209125e4a92d0bcbe0b77dc49d558f52e not found: ID does not exist" Nov 04 03:38:43.829237 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:38:43.829194 4405 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d319203f-26fb-438b-8466-b895b6c6d2d4-plugins-config-volume\") on node \"ip-10-0-134-125.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:38:43.829237 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:38:43.829240 4405 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jwfxc\" (UniqueName: \"kubernetes.io/projected/d319203f-26fb-438b-8466-b895b6c6d2d4-kube-api-access-jwfxc\") on node \"ip-10-0-134-125.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:38:43.948530 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:38:43.948442 4405 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-authcfg-http-cc2474-7df96f57b7-dp897"] Nov 04 03:38:43.953103 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:38:43.953072 4405 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-authcfg-http-cc2474-7df96f57b7-dp897"] Nov 04 03:38:45.679468 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:38:45.679434 4405 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-authcfg-http-cc2474-openshift-default-7f6458856f-qw52z"] Nov 04 03:38:45.679952 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:38:45.679656 4405 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-authcfg-http-cc2474-openshift-default-7f6458856f-qw52z" podUID="ab0796a3-e6f1-441a-b0f5-4d91c32c92d5" containerName="istio-proxy" containerID="cri-o://c65274aa868f4739b8ea499a31c0f8822cc70b23d51d3f8ec15b2478f97c0b7d" gracePeriod=30 Nov 04 03:38:45.700060 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:38:45.700027 4405 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d319203f-26fb-438b-8466-b895b6c6d2d4" path="/var/lib/kubelet/pods/d319203f-26fb-438b-8466-b895b6c6d2d4/volumes" Nov 04 03:38:47.627057 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:38:47.627024 4405 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-bl5mg"] Nov 04 03:38:48.695096 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:38:48.695043 4405 patch_prober.go:29] interesting pod/e2e-authcfg-http-cc2474-openshift-default-7f6458856f-qw52z container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 04 03:38:48.695539 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:38:48.695118 4405 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-authcfg-http-cc2474-openshift-default-7f6458856f-qw52z" podUID="ab0796a3-e6f1-441a-b0f5-4d91c32c92d5" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 04 03:38:50.833502 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:38:50.833466 4405 generic.go:356] "Generic (PLEG): container finished" podID="ab0796a3-e6f1-441a-b0f5-4d91c32c92d5" containerID="c65274aa868f4739b8ea499a31c0f8822cc70b23d51d3f8ec15b2478f97c0b7d" exitCode=0 Nov 04 03:38:50.833984 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:38:50.833540 4405 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-authcfg-http-cc2474-openshift-default-7f6458856f-qw52z" event={"ID":"ab0796a3-e6f1-441a-b0f5-4d91c32c92d5","Type":"ContainerDied","Data":"c65274aa868f4739b8ea499a31c0f8822cc70b23d51d3f8ec15b2478f97c0b7d"} Nov 04 03:38:50.833984 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:38:50.833613 4405 scope.go:122] "RemoveContainer" containerID="614f2922503f50e5fc5d852c86079bb3d487191f836187e8c2dbf9b3dd75ea0f" Nov 04 03:38:50.940389 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:38:50.940363 4405 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-authcfg-http-cc2474-openshift-default-7f6458856f-qw52z" Nov 04 03:38:51.098724 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:38:51.098691 4405 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ab0796a3-e6f1-441a-b0f5-4d91c32c92d5-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ab0796a3-e6f1-441a-b0f5-4d91c32c92d5-credential-socket\") pod \"ab0796a3-e6f1-441a-b0f5-4d91c32c92d5\" (UID: \"ab0796a3-e6f1-441a-b0f5-4d91c32c92d5\") " Nov 04 03:38:51.098925 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:38:51.098782 4405 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ab0796a3-e6f1-441a-b0f5-4d91c32c92d5-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ab0796a3-e6f1-441a-b0f5-4d91c32c92d5-workload-socket\") pod \"ab0796a3-e6f1-441a-b0f5-4d91c32c92d5\" (UID: \"ab0796a3-e6f1-441a-b0f5-4d91c32c92d5\") " Nov 04 03:38:51.098925 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:38:51.098849 4405 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ab0796a3-e6f1-441a-b0f5-4d91c32c92d5-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ab0796a3-e6f1-441a-b0f5-4d91c32c92d5-istiod-ca-cert\") pod \"ab0796a3-e6f1-441a-b0f5-4d91c32c92d5\" (UID: \"ab0796a3-e6f1-441a-b0f5-4d91c32c92d5\") " Nov 04 03:38:51.098925 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:38:51.098882 4405 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/ab0796a3-e6f1-441a-b0f5-4d91c32c92d5-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ab0796a3-e6f1-441a-b0f5-4d91c32c92d5-istio-podinfo\") pod \"ab0796a3-e6f1-441a-b0f5-4d91c32c92d5\" (UID: \"ab0796a3-e6f1-441a-b0f5-4d91c32c92d5\") " Nov 04 03:38:51.099188 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:38:51.098942 4405 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ab0796a3-e6f1-441a-b0f5-4d91c32c92d5-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ab0796a3-e6f1-441a-b0f5-4d91c32c92d5-istio-envoy\") pod \"ab0796a3-e6f1-441a-b0f5-4d91c32c92d5\" (UID: \"ab0796a3-e6f1-441a-b0f5-4d91c32c92d5\") " Nov 04 03:38:51.099188 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:38:51.099005 4405 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ab0796a3-e6f1-441a-b0f5-4d91c32c92d5-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ab0796a3-e6f1-441a-b0f5-4d91c32c92d5-workload-certs\") pod \"ab0796a3-e6f1-441a-b0f5-4d91c32c92d5\" (UID: \"ab0796a3-e6f1-441a-b0f5-4d91c32c92d5\") " Nov 04 03:38:51.099188 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:38:51.099023 4405 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ab0796a3-e6f1-441a-b0f5-4d91c32c92d5-credential-socket" pod "ab0796a3-e6f1-441a-b0f5-4d91c32c92d5" (UID: "ab0796a3-e6f1-441a-b0f5-4d91c32c92d5"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 03:38:51.099188 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:38:51.099043 4405 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ab0796a3-e6f1-441a-b0f5-4d91c32c92d5-istio-token\" (UniqueName: \"kubernetes.io/projected/ab0796a3-e6f1-441a-b0f5-4d91c32c92d5-istio-token\") pod \"ab0796a3-e6f1-441a-b0f5-4d91c32c92d5\" (UID: \"ab0796a3-e6f1-441a-b0f5-4d91c32c92d5\") " Nov 04 03:38:51.099188 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:38:51.099091 4405 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ab0796a3-e6f1-441a-b0f5-4d91c32c92d5-kube-api-access-rhjg9\" (UniqueName: \"kubernetes.io/projected/ab0796a3-e6f1-441a-b0f5-4d91c32c92d5-kube-api-access-rhjg9\") pod \"ab0796a3-e6f1-441a-b0f5-4d91c32c92d5\" (UID: \"ab0796a3-e6f1-441a-b0f5-4d91c32c92d5\") " Nov 04 03:38:51.099188 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:38:51.099100 4405 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ab0796a3-e6f1-441a-b0f5-4d91c32c92d5-workload-socket" pod "ab0796a3-e6f1-441a-b0f5-4d91c32c92d5" (UID: "ab0796a3-e6f1-441a-b0f5-4d91c32c92d5"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 03:38:51.099188 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:38:51.099144 4405 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ab0796a3-e6f1-441a-b0f5-4d91c32c92d5-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ab0796a3-e6f1-441a-b0f5-4d91c32c92d5-istio-data\") pod \"ab0796a3-e6f1-441a-b0f5-4d91c32c92d5\" (UID: \"ab0796a3-e6f1-441a-b0f5-4d91c32c92d5\") " Nov 04 03:38:51.099550 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:38:51.099216 4405 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ab0796a3-e6f1-441a-b0f5-4d91c32c92d5-istiod-ca-cert" pod "ab0796a3-e6f1-441a-b0f5-4d91c32c92d5" (UID: "ab0796a3-e6f1-441a-b0f5-4d91c32c92d5"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 03:38:51.099550 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:38:51.099343 4405 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ab0796a3-e6f1-441a-b0f5-4d91c32c92d5-workload-certs" pod "ab0796a3-e6f1-441a-b0f5-4d91c32c92d5" (UID: "ab0796a3-e6f1-441a-b0f5-4d91c32c92d5"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 03:38:51.099550 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:38:51.099541 4405 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ab0796a3-e6f1-441a-b0f5-4d91c32c92d5-credential-socket\") on node \"ip-10-0-134-125.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:38:51.099732 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:38:51.099555 4405 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ab0796a3-e6f1-441a-b0f5-4d91c32c92d5-istio-data" pod "ab0796a3-e6f1-441a-b0f5-4d91c32c92d5" (UID: "ab0796a3-e6f1-441a-b0f5-4d91c32c92d5"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 03:38:51.099732 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:38:51.099564 4405 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ab0796a3-e6f1-441a-b0f5-4d91c32c92d5-workload-socket\") on node \"ip-10-0-134-125.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:38:51.099732 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:38:51.099596 4405 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ab0796a3-e6f1-441a-b0f5-4d91c32c92d5-istiod-ca-cert\") on node \"ip-10-0-134-125.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:38:51.099732 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:38:51.099616 4405 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ab0796a3-e6f1-441a-b0f5-4d91c32c92d5-workload-certs\") on node \"ip-10-0-134-125.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:38:51.101470 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:38:51.101434 4405 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ab0796a3-e6f1-441a-b0f5-4d91c32c92d5-kube-api-access-rhjg9" pod "ab0796a3-e6f1-441a-b0f5-4d91c32c92d5" (UID: "ab0796a3-e6f1-441a-b0f5-4d91c32c92d5"). InnerVolumeSpecName "kube-api-access-rhjg9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 03:38:51.101586 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:38:51.101440 4405 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ab0796a3-e6f1-441a-b0f5-4d91c32c92d5-istio-token" pod "ab0796a3-e6f1-441a-b0f5-4d91c32c92d5" (UID: "ab0796a3-e6f1-441a-b0f5-4d91c32c92d5"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 03:38:51.101727 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:38:51.101704 4405 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ab0796a3-e6f1-441a-b0f5-4d91c32c92d5-istio-envoy" pod "ab0796a3-e6f1-441a-b0f5-4d91c32c92d5" (UID: "ab0796a3-e6f1-441a-b0f5-4d91c32c92d5"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 03:38:51.101811 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:38:51.101717 4405 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ab0796a3-e6f1-441a-b0f5-4d91c32c92d5-istio-podinfo" pod "ab0796a3-e6f1-441a-b0f5-4d91c32c92d5" (UID: "ab0796a3-e6f1-441a-b0f5-4d91c32c92d5"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 04 03:38:51.200244 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:38:51.200203 4405 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ab0796a3-e6f1-441a-b0f5-4d91c32c92d5-istio-podinfo\") on node \"ip-10-0-134-125.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:38:51.200244 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:38:51.200244 4405 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ab0796a3-e6f1-441a-b0f5-4d91c32c92d5-istio-envoy\") on node \"ip-10-0-134-125.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:38:51.200458 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:38:51.200258 4405 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ab0796a3-e6f1-441a-b0f5-4d91c32c92d5-istio-token\") on node \"ip-10-0-134-125.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:38:51.200458 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:38:51.200267 4405 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rhjg9\" (UniqueName: \"kubernetes.io/projected/ab0796a3-e6f1-441a-b0f5-4d91c32c92d5-kube-api-access-rhjg9\") on node \"ip-10-0-134-125.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:38:51.200458 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:38:51.200279 4405 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ab0796a3-e6f1-441a-b0f5-4d91c32c92d5-istio-data\") on node \"ip-10-0-134-125.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:38:51.839025 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:38:51.838994 4405 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-authcfg-http-cc2474-openshift-default-7f6458856f-qw52z" Nov 04 03:38:51.839025 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:38:51.839005 4405 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-authcfg-http-cc2474-openshift-default-7f6458856f-qw52z" event={"ID":"ab0796a3-e6f1-441a-b0f5-4d91c32c92d5","Type":"ContainerDied","Data":"a2b7956bf689f0d58810a3beb046be46a332c613fa832f9225e24f9a5854f0df"} Nov 04 03:38:51.839544 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:38:51.839068 4405 scope.go:122] "RemoveContainer" containerID="c65274aa868f4739b8ea499a31c0f8822cc70b23d51d3f8ec15b2478f97c0b7d" Nov 04 03:38:52.006054 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:38:52.005965 4405 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-authcfg-http-cc2474-openshift-default-7f6458856f-qw52z"] Nov 04 03:38:52.055786 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:38:52.055746 4405 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-authcfg-http-cc2474-openshift-default-7f6458856f-qw52z"] Nov 04 03:38:53.700715 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:38:53.700650 4405 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ab0796a3-e6f1-441a-b0f5-4d91c32c92d5" path="/var/lib/kubelet/pods/ab0796a3-e6f1-441a-b0f5-4d91c32c92d5/volumes" Nov 04 03:38:59.246415 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:38:59.246379 4405 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-bl5mg"] Nov 04 03:39:06.935705 ip-10-0-134-125 kubenswrapper[4405]: E1104 03:39:06.934727 4405 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5f2vb" containerName="extractor" Nov 04 03:39:09.457071 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:39:09.457031 4405 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-bl5mg"] Nov 04 03:39:18.160585 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:39:18.160545 4405 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-bl5mg"] Nov 04 03:39:30.004832 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:39:30.004779 4405 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76597d6855-xjqqw container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.138:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 04 03:39:30.005696 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:39:30.005433 4405 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-xjqqw" podUID="c97cb26c-6fc7-45db-9984-aa459ddb54df" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.138:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 04 03:39:33.048650 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:39:33.048602 4405 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-rpzrs"] Nov 04 03:39:33.098894 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:39:33.098862 4405 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-rpzrs"] Nov 04 03:39:33.699954 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:39:33.699923 4405 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ca8a537a-6cd6-4543-b2fa-b8ad5f9fc659" path="/var/lib/kubelet/pods/ca8a537a-6cd6-4543-b2fa-b8ad5f9fc659/volumes" Nov 04 03:39:36.882443 ip-10-0-134-125 kubenswrapper[4405]: E1104 03:39:36.882403 4405 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5f2vb" containerName="extractor" Nov 04 03:39:56.139701 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:39:56.139647 4405 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-bl5mg"] Nov 04 03:39:57.058694 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:39:57.057885 4405 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-jsfp9"] Nov 04 03:39:57.101694 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:39:57.099533 4405 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-jsfp9"] Nov 04 03:39:57.719694 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:39:57.719496 4405 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="577a3035-6086-41dd-aa8e-1b0713d4a664" path="/var/lib/kubelet/pods/577a3035-6086-41dd-aa8e-1b0713d4a664/volumes" Nov 04 03:39:57.965905 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:39:57.965861 4405 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76597d6855-dqn2b container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.139:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 04 03:39:57.966213 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:39:57.966179 4405 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-dqn2b" podUID="415bafea-577e-4fba-925e-eb2c492e3cca" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.139:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 04 03:40:06.883739 ip-10-0-134-125 kubenswrapper[4405]: E1104 03:40:06.883703 4405 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5f2vb" containerName="extractor" Nov 04 03:40:10.429925 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:40:10.429885 4405 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_usage-logs-tenancy-proxy-9c5d49bcd-vj798_214526eb-504c-474f-8fbb-fcfe9438586a/proxy/0.log" Nov 04 03:40:11.527482 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:40:11.527446 4405 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-78d5c4d8b5-4q4db_58b86d8b-324e-40b7-8862-203885a84f7a/maas-api/0.log" Nov 04 03:40:12.581252 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:40:12.581220 4405 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-c01e047c-sgxrj_0577ac72-e357-4c18-ad55-6f1566b4b6ca/revoke-keys/0.log" Nov 04 03:40:15.499024 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:40:15.498986 4405 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-7b7bb4bc77-rm4jt_f33c29b3-72db-4504-b751-2a2666001277/manager/0.log" Nov 04 03:40:16.293810 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:40:16.293780 4405 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-678d87766f-jcwkh_f647d4a1-e49d-4562-8440-6fccab60fc85/manager/0.log" Nov 04 03:40:16.996953 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:40:16.996924 4405 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-7c9b87d574-bl5mg_464f01e2-36a3-4d4b-b92c-a15b621b76a9/limitador/0.log" Nov 04 03:40:19.252228 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:40:19.252199 4405 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-54497bf445-qvs4m_8f77d20d-3db8-47dc-a6ee-dec3f1b68432/kube-auth-proxy/0.log" Nov 04 03:40:19.717279 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:40:19.717232 4405 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-76597d6855-7c95j_76df1601-21fb-40ab-a5ee-433f367453c7/istio-proxy/0.log" Nov 04 03:40:20.241279 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:40:20.241252 4405 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-76597d6855-dqn2b_415bafea-577e-4fba-925e-eb2c492e3cca/istio-proxy/0.log" Nov 04 03:40:21.047694 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:40:21.047619 4405 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c01e047c-sgxrj"] Nov 04 03:40:21.097152 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:40:21.097115 4405 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c01e047c-sgxrj"] Nov 04 03:40:21.699625 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:40:21.699589 4405 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0577ac72-e357-4c18-ad55-6f1566b4b6ca" path="/var/lib/kubelet/pods/0577ac72-e357-4c18-ad55-6f1566b4b6ca/volumes" Nov 04 03:40:21.750053 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:40:21.750010 4405 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-76597d6855-xjqqw_c97cb26c-6fc7-45db-9984-aa459ddb54df/istio-proxy/0.log" Nov 04 03:40:24.339646 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:40:24.339609 4405 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-embedding-simulated-kserve-67bf594cd6-gnjj2_22d477a6-9ab8-4330-ae26-813caff99cb3/main/0.log" Nov 04 03:40:24.610016 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:40:24.609926 4405 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-trlp-test-simulated-kserve-69b588897d-fg8cm_f490d2f8-2111-42e4-b438-a1180fcacb45/main/0.log" Nov 04 03:40:25.154345 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:40:25.154315 4405 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_echo-simulator-kserve-64b6f8f96c-2lg6v_fe1fd7ca-7aad-4761-abe0-78bbd438ffad/main/0.log" Nov 04 03:40:25.709336 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:40:25.709303 4405 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_premium-simulated-simulated-premium-kserve-5f5c4b459f-ljhhw_f66b1861-d87b-4d57-ab27-f1a24544a534/main/0.log" Nov 04 03:40:35.772260 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:40:35.772227 4405 scope.go:122] "RemoveContainer" containerID="fc144a1b5bd37b8ca1ea5af55f088d3eae73e2a891e364574efb134ad5eda4a0" Nov 04 03:40:36.883521 ip-10-0-134-125 kubenswrapper[4405]: E1104 03:40:36.883482 4405 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5f2vb" containerName="extractor" Nov 04 03:40:38.685686 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:40:38.685645 4405 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_usage-logs-tenancy-proxy-9c5d49bcd-vj798_214526eb-504c-474f-8fbb-fcfe9438586a/proxy/0.log" Nov 04 03:40:39.314890 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:40:39.314849 4405 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-78d5c4d8b5-4q4db_58b86d8b-324e-40b7-8862-203885a84f7a/maas-api/0.log" Nov 04 03:40:42.165121 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:40:42.165092 4405 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-7b7bb4bc77-rm4jt_f33c29b3-72db-4504-b751-2a2666001277/manager/0.log" Nov 04 03:40:43.005542 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:40:43.005509 4405 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-678d87766f-jcwkh_f647d4a1-e49d-4562-8440-6fccab60fc85/manager/0.log" Nov 04 03:40:43.429848 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:40:43.429814 4405 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-7c9b87d574-bl5mg_464f01e2-36a3-4d4b-b92c-a15b621b76a9/limitador/0.log" Nov 04 03:40:45.160114 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:40:45.160006 4405 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-54497bf445-qvs4m_8f77d20d-3db8-47dc-a6ee-dec3f1b68432/kube-auth-proxy/0.log" Nov 04 03:40:45.629782 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:40:45.629748 4405 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-76597d6855-7c95j_76df1601-21fb-40ab-a5ee-433f367453c7/istio-proxy/0.log" Nov 04 03:40:46.136551 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:40:46.136498 4405 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-76597d6855-dqn2b_415bafea-577e-4fba-925e-eb2c492e3cca/istio-proxy/0.log" Nov 04 03:40:47.830397 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:40:47.830364 4405 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-76597d6855-xjqqw_c97cb26c-6fc7-45db-9984-aa459ddb54df/istio-proxy/0.log" Nov 04 03:40:50.057931 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:40:50.057898 4405 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-embedding-simulated-kserve-67bf594cd6-gnjj2_22d477a6-9ab8-4330-ae26-813caff99cb3/main/0.log" Nov 04 03:40:50.392754 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:40:50.392726 4405 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-trlp-test-simulated-kserve-69b588897d-fg8cm_f490d2f8-2111-42e4-b438-a1180fcacb45/main/0.log" Nov 04 03:40:50.956299 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:40:50.956268 4405 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_echo-simulator-kserve-64b6f8f96c-2lg6v_fe1fd7ca-7aad-4761-abe0-78bbd438ffad/main/0.log" Nov 04 03:40:51.524711 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:40:51.524657 4405 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_premium-simulated-simulated-premium-kserve-5f5c4b459f-ljhhw_f66b1861-d87b-4d57-ab27-f1a24544a534/main/0.log" Nov 04 03:41:06.883347 ip-10-0-134-125 kubenswrapper[4405]: E1104 03:41:06.883298 4405 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5f2vb" containerName="extractor" Nov 04 03:41:35.792015 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:41:35.791982 4405 scope.go:122] "RemoveContainer" containerID="922e2385b3d60b6317c286c6b9ad327747441dab8a6386f2319bc402757318c7" Nov 04 03:41:35.800229 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:41:35.800207 4405 scope.go:122] "RemoveContainer" containerID="0cc3862ede90a79322ba01c8524ffaa2825bd808738bbcdd13506c30028cf40b" Nov 04 03:41:36.882714 ip-10-0-134-125 kubenswrapper[4405]: E1104 03:41:36.882655 4405 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5f2vb" containerName="extractor" Nov 04 03:42:06.808296 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:42:06.808266 4405 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-krvqk_3a7ab1db-ad88-40d9-a7e9-b511bbd4584c/global-pull-secret-syncer/0.log" Nov 04 03:42:06.884612 ip-10-0-134-125 kubenswrapper[4405]: E1104 03:42:06.884566 4405 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5f2vb" containerName="extractor" Nov 04 03:42:09.731318 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:42:09.731285 4405 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-wcp7z_5b77e30b-cb0c-49c1-80b9-330a138dbfb7/konnectivity-agent/0.log" Nov 04 03:42:10.975597 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:42:10.975461 4405 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-134-125.us-west-1.compute.internal_ee7b213d76bf8a5295d150c84b46451e/haproxy/0.log" Nov 04 03:42:15.878130 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:42:15.878041 4405 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-7b7bb4bc77-rm4jt_f33c29b3-72db-4504-b751-2a2666001277/manager/0.log" Nov 04 03:42:17.089206 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:42:17.089166 4405 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-must-gather-wjk9m/perf-node-gather-daemonset-6mmnd"] Nov 04 03:42:17.089743 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:42:17.089720 4405 cpu_manager.go:414] "containerMap: removing container" podUID="d319203f-26fb-438b-8466-b895b6c6d2d4" containerName="payload-pre-processing" Nov 04 03:42:17.089743 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:42:17.089740 4405 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d319203f-26fb-438b-8466-b895b6c6d2d4" containerName="payload-pre-processing" Nov 04 03:42:17.089931 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:42:17.089780 4405 cpu_manager.go:414] "containerMap: removing container" podUID="ab0796a3-e6f1-441a-b0f5-4d91c32c92d5" containerName="istio-proxy" Nov 04 03:42:17.089931 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:42:17.089789 4405 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ab0796a3-e6f1-441a-b0f5-4d91c32c92d5" containerName="istio-proxy" Nov 04 03:42:17.089931 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:42:17.089808 4405 cpu_manager.go:414] "containerMap: removing container" podUID="7f32fb6c-d557-461b-b22f-90a4d3ea46e3" containerName="maas-api" Nov 04 03:42:17.089931 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:42:17.089815 4405 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7f32fb6c-d557-461b-b22f-90a4d3ea46e3" containerName="maas-api" Nov 04 03:42:17.089931 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:42:17.089831 4405 cpu_manager.go:414] "containerMap: removing container" podUID="ab0796a3-e6f1-441a-b0f5-4d91c32c92d5" containerName="istio-proxy" Nov 04 03:42:17.089931 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:42:17.089839 4405 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ab0796a3-e6f1-441a-b0f5-4d91c32c92d5" containerName="istio-proxy" Nov 04 03:42:17.089931 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:42:17.089909 4405 memory_manager.go:365] "RemoveStaleState removing state" podUID="ab0796a3-e6f1-441a-b0f5-4d91c32c92d5" containerName="istio-proxy" Nov 04 03:42:17.089931 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:42:17.089921 4405 memory_manager.go:365] "RemoveStaleState removing state" podUID="7f32fb6c-d557-461b-b22f-90a4d3ea46e3" containerName="maas-api" Nov 04 03:42:17.089931 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:42:17.089930 4405 memory_manager.go:365] "RemoveStaleState removing state" podUID="ab0796a3-e6f1-441a-b0f5-4d91c32c92d5" containerName="istio-proxy" Nov 04 03:42:17.090243 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:42:17.089942 4405 memory_manager.go:365] "RemoveStaleState removing state" podUID="d319203f-26fb-438b-8466-b895b6c6d2d4" containerName="payload-pre-processing" Nov 04 03:42:17.093375 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:42:17.093355 4405 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-wjk9m/perf-node-gather-daemonset-6mmnd" Nov 04 03:42:17.146681 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:42:17.146639 4405 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-wjk9m\"/\"kube-root-ca.crt\"" Nov 04 03:42:17.151116 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:42:17.151093 4405 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-wjk9m\"/\"openshift-service-ca.crt\"" Nov 04 03:42:17.173848 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:42:17.173813 4405 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qd2zj\" (UniqueName: \"kubernetes.io/projected/d8b781f9-790d-434c-a645-c3d6567050b8-kube-api-access-qd2zj\") pod \"perf-node-gather-daemonset-6mmnd\" (UID: \"d8b781f9-790d-434c-a645-c3d6567050b8\") " pod="openshift-must-gather-wjk9m/perf-node-gather-daemonset-6mmnd" Nov 04 03:42:17.173973 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:42:17.173915 4405 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/d8b781f9-790d-434c-a645-c3d6567050b8-podres\") pod \"perf-node-gather-daemonset-6mmnd\" (UID: \"d8b781f9-790d-434c-a645-c3d6567050b8\") " pod="openshift-must-gather-wjk9m/perf-node-gather-daemonset-6mmnd" Nov 04 03:42:17.173973 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:42:17.173939 4405 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/d8b781f9-790d-434c-a645-c3d6567050b8-sys\") pod \"perf-node-gather-daemonset-6mmnd\" (UID: \"d8b781f9-790d-434c-a645-c3d6567050b8\") " pod="openshift-must-gather-wjk9m/perf-node-gather-daemonset-6mmnd" Nov 04 03:42:17.174054 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:42:17.174005 4405 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/d8b781f9-790d-434c-a645-c3d6567050b8-lib-modules\") pod \"perf-node-gather-daemonset-6mmnd\" (UID: \"d8b781f9-790d-434c-a645-c3d6567050b8\") " pod="openshift-must-gather-wjk9m/perf-node-gather-daemonset-6mmnd" Nov 04 03:42:17.174090 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:42:17.174055 4405 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/d8b781f9-790d-434c-a645-c3d6567050b8-proc\") pod \"perf-node-gather-daemonset-6mmnd\" (UID: \"d8b781f9-790d-434c-a645-c3d6567050b8\") " pod="openshift-must-gather-wjk9m/perf-node-gather-daemonset-6mmnd" Nov 04 03:42:17.212197 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:42:17.212033 4405 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-wjk9m/perf-node-gather-daemonset-6mmnd"] Nov 04 03:42:17.220818 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:42:17.220779 4405 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-wjk9m\"/\"default-dockercfg-fdwhv\"" Nov 04 03:42:17.231282 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:42:17.231248 4405 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-678d87766f-jcwkh_f647d4a1-e49d-4562-8440-6fccab60fc85/manager/0.log" Nov 04 03:42:17.275050 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:42:17.275011 4405 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qd2zj\" (UniqueName: \"kubernetes.io/projected/d8b781f9-790d-434c-a645-c3d6567050b8-kube-api-access-qd2zj\") pod \"perf-node-gather-daemonset-6mmnd\" (UID: \"d8b781f9-790d-434c-a645-c3d6567050b8\") " pod="openshift-must-gather-wjk9m/perf-node-gather-daemonset-6mmnd" Nov 04 03:42:17.275269 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:42:17.275066 4405 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/d8b781f9-790d-434c-a645-c3d6567050b8-podres\") pod \"perf-node-gather-daemonset-6mmnd\" (UID: \"d8b781f9-790d-434c-a645-c3d6567050b8\") " pod="openshift-must-gather-wjk9m/perf-node-gather-daemonset-6mmnd" Nov 04 03:42:17.275269 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:42:17.275093 4405 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/d8b781f9-790d-434c-a645-c3d6567050b8-sys\") pod \"perf-node-gather-daemonset-6mmnd\" (UID: \"d8b781f9-790d-434c-a645-c3d6567050b8\") " pod="openshift-must-gather-wjk9m/perf-node-gather-daemonset-6mmnd" Nov 04 03:42:17.275269 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:42:17.275127 4405 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/d8b781f9-790d-434c-a645-c3d6567050b8-lib-modules\") pod \"perf-node-gather-daemonset-6mmnd\" (UID: \"d8b781f9-790d-434c-a645-c3d6567050b8\") " pod="openshift-must-gather-wjk9m/perf-node-gather-daemonset-6mmnd" Nov 04 03:42:17.275269 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:42:17.275182 4405 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/d8b781f9-790d-434c-a645-c3d6567050b8-proc\") pod \"perf-node-gather-daemonset-6mmnd\" (UID: \"d8b781f9-790d-434c-a645-c3d6567050b8\") " pod="openshift-must-gather-wjk9m/perf-node-gather-daemonset-6mmnd" Nov 04 03:42:17.275269 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:42:17.275230 4405 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/d8b781f9-790d-434c-a645-c3d6567050b8-podres\") pod \"perf-node-gather-daemonset-6mmnd\" (UID: \"d8b781f9-790d-434c-a645-c3d6567050b8\") " pod="openshift-must-gather-wjk9m/perf-node-gather-daemonset-6mmnd" Nov 04 03:42:17.275269 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:42:17.275229 4405 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/d8b781f9-790d-434c-a645-c3d6567050b8-sys\") pod \"perf-node-gather-daemonset-6mmnd\" (UID: \"d8b781f9-790d-434c-a645-c3d6567050b8\") " pod="openshift-must-gather-wjk9m/perf-node-gather-daemonset-6mmnd" Nov 04 03:42:17.275538 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:42:17.275267 4405 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/d8b781f9-790d-434c-a645-c3d6567050b8-lib-modules\") pod \"perf-node-gather-daemonset-6mmnd\" (UID: \"d8b781f9-790d-434c-a645-c3d6567050b8\") " pod="openshift-must-gather-wjk9m/perf-node-gather-daemonset-6mmnd" Nov 04 03:42:17.275538 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:42:17.275273 4405 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/d8b781f9-790d-434c-a645-c3d6567050b8-proc\") pod \"perf-node-gather-daemonset-6mmnd\" (UID: \"d8b781f9-790d-434c-a645-c3d6567050b8\") " pod="openshift-must-gather-wjk9m/perf-node-gather-daemonset-6mmnd" Nov 04 03:42:17.340923 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:42:17.340848 4405 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qd2zj\" (UniqueName: \"kubernetes.io/projected/d8b781f9-790d-434c-a645-c3d6567050b8-kube-api-access-qd2zj\") pod \"perf-node-gather-daemonset-6mmnd\" (UID: \"d8b781f9-790d-434c-a645-c3d6567050b8\") " pod="openshift-must-gather-wjk9m/perf-node-gather-daemonset-6mmnd" Nov 04 03:42:17.407804 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:42:17.407773 4405 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-wjk9m/perf-node-gather-daemonset-6mmnd" Nov 04 03:42:17.784090 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:42:17.784062 4405 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-wjk9m/perf-node-gather-daemonset-6mmnd"] Nov 04 03:42:17.786615 ip-10-0-134-125 kubenswrapper[4405]: W1104 03:42:17.786587 4405 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-podd8b781f9_790d_434c_a645_c3d6567050b8.slice/crio-dd9c4def32e3894ad4876a980c2ce570a30efddabbea1342dcce7812e8146bb7 WatchSource:0}: Error finding container dd9c4def32e3894ad4876a980c2ce570a30efddabbea1342dcce7812e8146bb7: Status 404 returned error can't find the container with id dd9c4def32e3894ad4876a980c2ce570a30efddabbea1342dcce7812e8146bb7 Nov 04 03:42:17.816084 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:42:17.816057 4405 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-7c9b87d574-bl5mg_464f01e2-36a3-4d4b-b92c-a15b621b76a9/limitador/0.log" Nov 04 03:42:17.855840 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:42:17.855799 4405 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-wjk9m/perf-node-gather-daemonset-6mmnd" event={"ID":"d8b781f9-790d-434c-a645-c3d6567050b8","Type":"ContainerStarted","Data":"dd9c4def32e3894ad4876a980c2ce570a30efddabbea1342dcce7812e8146bb7"} Nov 04 03:42:18.862009 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:42:18.861975 4405 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-wjk9m/perf-node-gather-daemonset-6mmnd" event={"ID":"d8b781f9-790d-434c-a645-c3d6567050b8","Type":"ContainerStarted","Data":"a55292c1d6ce42890ef3555d30b20ac2d910015b78ee940979d131df32f9f799"} Nov 04 03:42:18.862429 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:42:18.862101 4405 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-wjk9m/perf-node-gather-daemonset-6mmnd" Nov 04 03:42:18.977964 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:42:18.977909 4405 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-must-gather-wjk9m/perf-node-gather-daemonset-6mmnd" podStartSLOduration=1.9778941799999998 podStartE2EDuration="1.97789418s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-04 03:42:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 03:42:18.977003291 +0000 UTC m=+4185.407703411" watchObservedRunningTime="2026-11-04 03:42:18.97789418 +0000 UTC m=+4185.408594300" Nov 04 03:42:24.877694 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:42:24.877647 4405 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-wjk9m/perf-node-gather-daemonset-6mmnd" Nov 04 03:42:26.450720 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:42:26.450690 4405 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-296tb_2226b455-2db8-4e6e-875b-9f3f05947bf9/node-exporter/0.log" Nov 04 03:42:26.871456 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:42:26.871427 4405 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-296tb_2226b455-2db8-4e6e-875b-9f3f05947bf9/kube-rbac-proxy/0.log" Nov 04 03:42:27.294178 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:42:27.294095 4405 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-296tb_2226b455-2db8-4e6e-875b-9f3f05947bf9/init-textfile/0.log" Nov 04 03:42:31.553702 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:42:31.553654 4405 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_a3c43b2d-590a-4490-b05b-7b30b58592bf/prometheus/0.log" Nov 04 03:42:31.975433 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:42:31.975400 4405 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_a3c43b2d-590a-4490-b05b-7b30b58592bf/config-reloader/0.log" Nov 04 03:42:32.397849 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:42:32.397819 4405 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_a3c43b2d-590a-4490-b05b-7b30b58592bf/thanos-sidecar/0.log" Nov 04 03:42:32.821270 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:42:32.821193 4405 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_a3c43b2d-590a-4490-b05b-7b30b58592bf/kube-rbac-proxy-web/0.log" Nov 04 03:42:33.243885 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:42:33.243852 4405 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_a3c43b2d-590a-4490-b05b-7b30b58592bf/kube-rbac-proxy/0.log" Nov 04 03:42:33.663830 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:42:33.663801 4405 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_a3c43b2d-590a-4490-b05b-7b30b58592bf/kube-rbac-proxy-thanos/0.log" Nov 04 03:42:34.087795 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:42:34.087764 4405 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_a3c43b2d-590a-4490-b05b-7b30b58592bf/init-config-reloader/0.log" Nov 04 03:42:34.302139 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:42:34.302115 4405 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-vwcx8_ffbb2d83-5a93-4a5b-bce3-e40afb8f58ea/ovn-acl-logging/0.log" Nov 04 03:42:34.319235 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:42:34.319206 4405 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-vwcx8_ffbb2d83-5a93-4a5b-bce3-e40afb8f58ea/ovn-acl-logging/0.log" Nov 04 03:42:35.371239 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:42:35.371210 4405 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-admission-webhook-6489d8bf89-9ckn2_514441fd-195f-4e0a-8313-4d1047fe6b02/prometheus-operator-admission-webhook/0.log" Nov 04 03:42:36.883131 ip-10-0-134-125 kubenswrapper[4405]: E1104 03:42:36.883087 4405 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5f2vb" containerName="extractor" Nov 04 03:42:40.875835 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:42:40.875803 4405 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-54497bf445-qvs4m_8f77d20d-3db8-47dc-a6ee-dec3f1b68432/kube-auth-proxy/0.log" Nov 04 03:42:41.334314 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:42:41.334280 4405 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-76597d6855-7c95j_76df1601-21fb-40ab-a5ee-433f367453c7/istio-proxy/0.log" Nov 04 03:42:41.775686 ip-10-0-134-125 kubenswrapper[4405]: I1104 03:42:41.775573 4405 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-76597d6855-dqn2b_415bafea-577e-4fba-925e-eb2c492e3cca/istio-proxy/0.log"